Visual programming for deep learning

Abstract

Various implementations of the present disclosure relate to visual programming for deep learning. In a computer-implemented method, a visual representation of an artificial neural network is presented, the visual representation comprising graphical elements representing layers of the artificial neural network; in response to receiving a drag-and-drop operation for the graphic elements, modifying an intermediate representation of the artificial neural network, the intermediate representation being independent of a deep learning framework, the drag-and-drop operation modifying a connection relationship between the graphic elements; and modifying a code of the artificial neural network for the target deep learning framework based on the intermediate representation of the artificial neural network.

Description

[0001] This application is a divisional application of the invention patent application with the application date of June 28, 2019, the Chinese national application number 201910578856.X, and the invention name “Visual Programming for Deep Learning”. Background Art

[0002] Due to the widespread application of deep learning, more and more developers are learning and using deep learning programming. However, in the current deep learning programming process, developers need to write a lot of code to preprocess data and build deep learning models. Beginners are usually overwhelmed by a lot of underlying code and find it difficult to quickly understand and master deep learning programming. Summary of the invention

[0003] In a computer-implemented method, a visual representation of an artificial neural network is presented, the visual representation including graphic elements representing layers of the artificial neural network; in response to receiving a drag-and-drop operation for the graphic elements, an intermediate representation of the artificial neural network is modified, the intermediate representation being independent of a deep learning framework, the drag-and-drop operation modifying a connection relationship between the graphic elements; and based on the intermediate representation of the artificial neural network, a code of the artificial neural network for a target deep learning framework is modified.

[0004] The invention summary is provided to selectively introduce the inventive concept in a simplified form, which will be further described in the detailed description below. The invention summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 A block diagram showing a computing device capable of implementing various implementations of the present disclosure is shown;

[0006] Figure 2 A schematic diagram showing a visual programming architecture according to some implementations of the present disclosure is shown;

[0007] Figure 3 A schematic diagram showing a visual representation of an artificial neural network according to some implementations of the present disclosure; and

[0008] Figure 4 A flowchart of a method for visual programming according to some implementations of the present disclosure is shown.

[0009] In these drawings, the same or similar reference symbols are used to designate the same or similar elements. DETAILED DESCRIPTION

[0010] The present disclosure will now be discussed with reference to several example implementations. It should be understood that these implementations are discussed only to enable those skilled in the art to better understand and thus implement the present disclosure, and do not imply any limitation on the scope of the present subject matter.

[0011] As used herein, the term "including" and variations thereof are to be interpreted as open-ended terms meaning "including but not limited to". The term "based on" is to be interpreted as "based at least in part on". The terms "an implementation" and "an implementation" are to be interpreted as "at least one implementation". The term "another implementation" is to be interpreted as "at least one other implementation". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0012] The basic principles and several example implementations of the present disclosure are explained below with reference to the accompanying drawings. Figure 1 1 is a block diagram of a computing device 100 capable of implementing multiple implementations of the present disclosure. It should be understood that Figure 1 The computing device 100 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described in the present disclosure. Figure 1 As shown, the computing device 100 may be implemented by a general computing device or a special computing device. The components of the computing device 100 may include, but are not limited to, one or more processors or processing units 110, a memory 120, a storage device 130, one or more communication units 140, one or more input devices 150, and one or more output devices 160.

[0013] In some implementations, the computing device 100 can be implemented as various user terminals or service terminals with computing capabilities. The service terminal can be a server, a large computing device, etc. provided by various service providers. The user terminal is such as any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a site, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. It is also foreseeable that the computing device 100 can support any type of interface for the user (such as a "wearable" circuit, etc.).

[0014] The processing unit 110 may be a real or virtual processor and may be capable of performing various processes according to a program stored in the memory 120. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the computing device 100. The processing unit 110 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller. In addition, the processing unit 110 may also be implemented as a graphics processing unit (GPU) to perform computationally intensive processes such as parallel operations.

[0015] The computing device 100 typically includes a plurality of computer storage media. Such media may be any available media accessible to the computing device 100, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 120 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The memory 120 may include visual programming modules 122, which are configured to perform the functions of the various implementations described herein. The visual programming modules 122 may be accessed and run by the processing unit 110 to implement the corresponding functions.

[0016] Storage device 130 may be a removable or non-removable medium and may include machine-readable media that can be used to store information and / or data and can be accessed within computing device 100. Computing device 100 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 1 As shown in , a disk drive for reading or writing from a removable, nonvolatile disk and an optical drive for reading or writing from a removable, nonvolatile optical disk can be provided. In these cases, each drive can be connected to a bus (not shown) by one or more data media interfaces.

[0017] The communication unit 140 enables communication with another computing device via a communication medium. Additionally, the functions of the components of the computing device 100 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 100 can operate in a networked environment using a logical connection with one or more other servers, a personal computer (PC), or another general network node.

[0018] Input device 150 may be one or more various input devices, such as a mouse, keyboard, tracking ball, voice input device, etc. Output device 160 may be one or more output devices, such as a display, a speaker, a printer, etc. Computing device 100 may also communicate with one or more external devices (not shown) through communication unit 140 as needed, such as storage devices, display devices, etc., communicate with one or more devices that allow a user to interact with computing device 100, or communicate with any device that allows computing device 100 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0019] In some implementations, in addition to being integrated on a single device, some or all of the various components of the computing device 100 can also be set in the form of a cloud computing architecture. In a cloud computing architecture, these components can be remotely arranged and can work together to implement the functions described in the present disclosure. In some implementations, cloud computing provides computing, software, data access and storage services, which do not require end users to know the physical location or configuration of the system or hardware that provides these services. In various implementations, cloud computing uses appropriate protocols to provide services through a wide area network (such as the Internet). For example, a cloud computing provider provides applications through a wide area network, and they can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data can be stored on a server at a remote location. The computing resources in a cloud computing environment can be merged at a remote data center location or they can be dispersed. Cloud computing infrastructure can provide services through a shared data center, even if they appear as a single access point for users. Therefore, the components and functions described herein can be provided from a service provider at a remote location using a cloud computing architecture. Alternatively, they can be provided from a conventional server, or they can be installed directly or otherwise on a client device.

[0020] The computing device 100 can be used to implement the visual programming scheme according to multiple implementations of the present disclosure. For example, the computing device 100 can display a visual representation of an artificial neural network on a display, such as the various layers and the connections between the layers. The computing device 100 can receive user input through the input device 150, such as a drag-and-drop operation, etc. For example, a user can drag and drop a layer onto the canvas. The visual programming module 122 can modify the presented visual representation based on the user input and further update the code associated with the artificial neural network. These codes can be provided to the output device 160 to be provided to the user as output 180, etc. For example, these codes can be displayed on the display simultaneously with the visual representation to be presented to the user.

[0021] Figure 2 A schematic diagram of an architecture 200 for visual programming according to some implementations of the present disclosure is shown. For example, the architecture 200 may be composed of Figure 1 The illustrated computing device 100, and in particular the visual programming module 122, is implemented. However, it should be appreciated that the architecture 200 may also be implemented on any other suitable device.

[0022] like Figure 2 As shown, the visual model editor 204 can load the visual model 202, which can be a visual model edited and saved by the developer using the visual model editor 204, or a visual model shared by other developers. The visual model 202 can be saved in the form of a description file, or in other appropriate forms. Alternatively, the developer can also use the visual model editor 204 to edit the visual model from scratch.

[0023] For example, the visual model editor 204 may include a canvas on which developers can edit the visual model of deep learning by dragging and dropping. Drag and drop is a common operation mode in a graphical user interface, and users can use the drag and drop operation to easily edit the connection relationship between graphic elements. For example, a user can drag and drop a graphic element to the input of another graphic element, and a user can also move a graphic element away from the input of a graphic element. Figure 3 A schematic diagram of a visual representation 300 of an artificial neural network according to some implementations of the present disclosure is shown. The visual representation 300 can be presented on a canvas of the visual model editor 204. The visual representation 300 includes graphical elements representing various layers of the artificial neural network, such as graphical user interface elements that can interact with developers. For example, the developer can click on the graphical interface element to modify the parameters of each layer, such as the shape (also known as the dimension of the data), the kernel size, the activation function, etc. Here, the parameter represents one or more modifiable properties of the corresponding layer.

[0024] like Figure 3 As shown, graphic element 302 represents an input layer, graphic element 304 represents a two-dimensional convolutional layer, graphic element 306 represents a two-dimensional maximum pooling layer, graphic element 308 represents a flattening layer, and graphic element 310 represents a fully connected layer. At each graphic element, the shape of the corresponding layer is also shown. For example, the original data shape of the input layer is [28, 28, 1], that is, a tensor of 28x28x 1.

[0025] It should be understood that the term "layer" here can include not only the usual processing layers of artificial neural networks (e.g., convolutional layers, fully connected layers, pooling layers), but also structured input / output, e.g. Figure 3302 in the input layer. In addition, the processing layer of the artificial neural network usually includes a set of multiple operators. For example, a fully connected layer may include a combination of two operators: matrix multiplication and addition. Therefore, the term "layer" here can also include more basic computing components than the usual processing layer, that is, operators, such as matrix multiplication.

[0026] The visualization 300 includes not only the individual graphic elements but also the connection relationships between these graphic elements, thereby forming a visualization of the corresponding model as a whole.

[0027] In a graphical manner, the developer can easily modify the parameters of each layer and connect between the layers. For example, when the developer edits the visual representation 300, the visual model editor 204 can determine the shape of each layer. Based on these shape parameters, it can be verified whether the shapes of the layers of the artificial neural network match. For example, if the shapes of two layers do not match, an error indication can be displayed in the visual representation 300. For example, the visual programming tool can verify the artificial neural network model immediately after the shape of any layer is changed and / or the connection relationship between any graphical elements is changed, in order to find problems with the model as soon as possible.

[0028] In some implementations, a user can drag and drop a first graphic element to a second graphic element to connect the first graphic element to the second graphic element. The output of the first graphic element is provided to the input of the second graphic element, so the data dimension of the output of the first graphic element should match the data dimension of the input of the second graphic element, otherwise the artificial neural network will be wrong. In addition, the visual editor 204 can automatically connect the graphic elements according to the shapes of the various graphic elements. For example, when a user adds a new graphic element, a graphic element that matches the shape of the graphic element can be automatically connected to the graphic element.

[0029] For example, in Figure 3 In the example shown, if the user modifies the data dimension of the input layer represented by the graphic element 302, the visual model editor 204 can compare the data dimension of the input layer with the data dimension of the input of the convolutional layer represented by the graphic element 304 to verify whether the shapes of the two layers match.

[0030] In some implementations, if the user modifies the data dimension of the input layer represented by the graphic element 302, the visual model editor 204 may adjust the data dimension of the input of the convolution layer represented by the graphic element 304 to the data dimension of the input layer represented by the graphic element 302. In addition, the visual model editor 204 may also verify whether the data dimension of the output of the convolution layer represented by the graphic element 304 matches.

[0031] In some implementations, the visual model editor 204 may provide a collection of predefined artificial neural network layers. These construction layers may be organized into multiple toolbox panels, such as input / output, convolutional neural network (CNN), recurrent neural network (RNN), fully connected layers, tensor operations, etc. Developers may select the desired construction layers from these toolbox panels as needed to build an artificial neural network model. For example, the interface of the visual model editor 204 may also include a search option that supports searching for a specific layer, and layers matching the search may be selected and dragged onto the canvas.

[0032] In some implementations, developers can save artificial neural network models or visualization models for subsequent use. These models can be saved in the form of description files or in any other suitable form. The saved visualization models can also be shared with other developers for use.

[0033] In some implementations, the interface of the visual model editor 204 may also include a data interface for providing a preview of a data set. For example, some public data, such as a public data set, may be provided, and a data import interface may also be provided for developers to upload local data sets for training. After the developer selects a specific data set, the visual model editor 204 may load the data set and display the data set and / or related information. For example, the total sample size of the data set, the sample size of the training set, and the sample size of the test set may all be displayed on the interface. For example, a unified user interface may be provided for public data and local data, and users may quickly import their own data. Users may preview uploaded data and train the model and use the trained model for inference.

[0034] like Figure 2 As shown, when the developer modifies the visual representation of the artificial neural network, the intermediate model generator 208 can instantly convert the visual representation into an intermediate representation, which is independent of the deep learning framework. The intermediate representation can correspond to the visual representation 300 and indicate the parameters and name of each graphic element. For example, the intermediate representation can include standardized descriptions such as parameters and names of predefined graphic elements. The description is independent of the deep learning framework. Generally speaking, each deep learning framework has its own specific specifications for the layers of the artificial neural network. Under different deep learning frameworks, the specifications of the specific layers of the artificial neural network and their specific parameters and names are also different.

[0035] The intermediate representation can be embodied in the form of an intermediate computational graph, and each node can correspond to a graphical element in the visual representation. The intermediate computational graph can describe the artificial neural network as a directed graph and is independent of the deep learning framework. There may not be a one-to-one correspondence between nodes and layers of the deep learning framework. For example, a node can correspond to multiple layers of a deep learning framework, or multiple nodes can correspond to a layer in a deep learning framework.

[0036] In some implementations, the edited or modified artificial neural network model may also be verified by the intermediate model generator 208, and related errors may be indicated at corresponding graphical elements by the visual model editor 204. Figure 3 In the example shown, if the user modifies the data dimension of the input layer represented by the graphic element 302, the intermediate model generator 208 can obtain the data dimension of the input of the convolution layer represented by the graphic element 304, and compare the data dimension of the input layer with the data dimension of the convolution layer to verify whether their shapes match.

[0037] In some other implementations, if the user modifies the data dimension of the input layer represented by the graphic element 302, the intermediate model generator 208 may also adjust the data dimension of the input of the convolution layer represented by the graphic element 304 to the data dimension of the input layer accordingly. Then, the intermediate model generator 208 may also verify whether the data dimension of the output of the convolution layer matches.

[0038] The deep learning framework may provide interfaces in multiple programming languages ​​to support programs in different programming languages. In some implementations, the intermediate representation may also be independent of the programming language supported by the deep learning framework.

[0039] The intermediate representation may then be provided to a target code generator 210. For each supported deep learning framework, the target code generator 210 is capable of generating training and / or inference programs and codes, such as TensorFlow code 212, PyTorch code 214, or CNTK code 216, etc. For example, a developer may select a target deep learning framework, and the target code generator 210 may generate code for the target deep learning framework based on predetermined rules. For example, the target code generator 210 may convert each node of the intermediate representation into a predefined code corresponding to the node under the target deep learning framework.

[0040] In some implementations, the developer can change the target deep learning framework to another target deep learning framework. Then, the target code generator 210 can determine the code for the other target deep learning framework based on the intermediate representation of the artificial neural network. In this way, it is very easy to switch between multiple deep learning frameworks.

[0041] In some implementations, the visualization model and the corresponding code can be displayed in parallel on the interface. For example, the canvas and the code portion can be displayed simultaneously on the interface. The code can also include code stubs or editable areas to allow users to customize their own metrics for training artificial neural networks to replace common metrics such as loss and accuracy.

[0042] like Figure 2 As shown, the job submitter 220 can receive the code 212, 214 or 216, and the input data 218, and submit the generated program to the local host 222, the cloud 224 or the cloud 226 for training or inference. The cloud 224 and 226 can deploy a deep learning cloud platform (also known as a machine learning cloud platform) for executing these codes or programs. In addition, the program can also be submitted to a remote workstation for execution.

[0043] During the training process, a metric graph of the artificial neural network training can also be drawn in real time on the interface, that is, a graph showing the changes in the value of the metric during the training process. In this way, developers can easily observe the training process and debug the artificial neural network and / or its training process. By viewing the execution progress and running indicators or metrics in real time, users can easily decide whether to continue running or terminate early.

[0044] For example, the job submitter 220 can provide a one-click job submission feature and can be integrated with an integrated development environment for AI. Therefore, the training code can be easily run in the cloud or locally.

[0045] The visual programming tool provided by multiple implementations of the present disclosure can integrate data preprocessing, deep learning model design and job submission / management as a complete pipeline. The visual programming tool provides model construction, training and inference functions through a simple user interface, and allows users to drag and drop deep learning layers to design models in a simple and visual way. In this way, the development process of deep learning can be greatly simplified.

[0046] The tool can automatically generate training code so that users can train models based on the visual model representation. The tool can automatically synchronize between the visual model representation and the generated code. When the user modifies the visual model representation or the code, the code or the visual model representation is automatically updated.

[0047] Figure 4 A flowchart of a method 400 for visual programming of deep learning according to some implementations of the present disclosure is shown. The method 400 may be implemented as follows: Figure 1 The visual programming module 122 shown in FIG. Figure 2The architecture 200 shown is implemented.

[0048] At 402, a visual representation of an artificial neural network is presented, e.g., Figure 3 Visual representation 300 is shown. Visual representation 300 includes graphical elements, such as graphical elements 302-310, representing layers of an artificial neural network.

[0049] At 404, when a user such as a developer performs an editing operation on the visualization 300, for example, when performing a drag-and-drop operation on a graphic element, the intermediate representation of the artificial neural network may be modified to indicate the connection relationship between the graphic elements modified by the drag-and-drop operation. The intermediate representation of the artificial neural network is independent of the deep learning framework.

[0050] At 406 , based on the intermediate representation of the artificial neural network, the code of the artificial neural network for the target deep learning framework is modified.

[0051] In some implementations, method 400 further includes modifying an intermediate representation of the artificial neural network in response to an editing operation on the code; and adjusting a visual representation of the artificial neural network based on the modified intermediate representation.

[0052] In some implementations, method 400 further includes, in response to receiving a drag-and-drop operation for the graphical element network, verifying whether the shapes of the layers of the artificial neural network match.

[0053] In some implementations, method 400 also includes, in response to receiving a search operation associated with a keyword, presenting a graphical element representing at least one candidate layer corresponding to the keyword; and in response to receiving a selection of the graphical element for at least one candidate layer, adding the selected graphical element to the visual representation.

[0054] In some implementations, method 400 further includes presenting a code stub for customizing the metric of the artificial neural network; and customizing the metric of the artificial neural network in response to an edit operation on the code stub.

[0055] In some implementations, the drag-and-drop operation is performed to at least one of: add new graphical elements representing layers of an artificial neural network to the visual representation; delete graphical elements representing layers of an artificial neural network from the visual representation; modify connection relationships between graphical elements representing layers of an artificial neural network; and modify parameters of graphical elements representing layers of an artificial neural network.

[0056] In some implementations, method 400 further includes, in response to receiving an indication to modify the target deep learning framework to another target deep learning framework, determining a code of the artificial neural network for another target deep learning framework based on the intermediate representation of the artificial neural network.

[0057] Some example implementations of the present disclosure are listed below.

[0058] In a first aspect, the present disclosure provides a computer-implemented method. The method includes: presenting a visual representation of an artificial neural network, the visual representation including a graphic element representing a layer of the artificial neural network; in response to receiving a drag-and-drop operation on the graphic element, modifying an intermediate representation of the artificial neural network, the intermediate representation being independent of a deep learning framework, the drag-and-drop operation modifying a connection relationship between the graphic elements; and based on the intermediate representation of the artificial neural network, modifying a code of the artificial neural network for a target deep learning framework.

[0059] In some implementations, the method further includes: modifying an intermediate representation of the artificial neural network in response to an editing operation on the code; and adjusting a visual representation of the artificial neural network based on the modified intermediate representation.

[0060] In some implementations, the method further includes, in response to receiving a drag and drop operation for the graphical element, verifying a dimension of data associated with the layer of the artificial neural network.

[0061] In some implementations, the method further includes: in response to receiving a search operation associated with a keyword, presenting a graphical element representing at least one candidate layer corresponding to the keyword; and in response to receiving a selection of a graphical element for at least one candidate layer, adding the selected graphical element to the visual representation.

[0062] In some implementations, the method further includes: presenting a code stub for customizing a metric of the artificial neural network; and customizing the metric of the artificial neural network in response to an edit operation on the code stub.

[0063] In some implementations, the method further includes modifying the intermediate representation of the artificial neural network in response to at least one of: adding a new graphical element representing a layer of the artificial neural network to the visual representation; deleting a graphical element representing a layer of the artificial neural network from the visual representation; and modifying a parameter of a graphical element representing a layer of the artificial neural network.

[0064] In some implementations, the method further includes: in response to receiving an indication to modify the target deep learning framework to another target deep learning framework, determining a code of the artificial neural network for another target deep learning framework based on the intermediate representation of the artificial neural network.

[0065] In a second aspect, the present disclosure provides a device. The device includes: a processing unit; and a memory, coupled to the processing unit and containing instructions stored thereon, the instructions causing the device to perform the following actions when executed by the processing unit: presenting a visual representation of an artificial neural network, the visual representation including a graphic element representing a layer of the artificial neural network; in response to receiving a drag-and-drop operation for the graphic element, modifying an intermediate representation of the artificial neural network, the intermediate representation being independent of a deep learning framework, the drag-and-drop operation modifying a connection relationship between the graphic elements; and modifying a code of the artificial neural network for a target deep learning framework based on the intermediate representation of the artificial neural network.

[0066] In some implementations, the actions further include: modifying an intermediate representation of the artificial neural network in response to an editing operation on the code; and adjusting a visual representation of the artificial neural network based on the modified intermediate representation.

[0067] In some implementations, the actions further include: in response to receiving a drag and drop operation for the person graphical element, verifying a dimension of a layer associated with a layer of the artificial neural network.

[0068] In some implementations, the action also includes: in response to receiving a search operation associated with a keyword, presenting a graphical element representing at least one candidate layer corresponding to the keyword; and in response to receiving a selection of a graphical element for at least one candidate layer, adding the selected graphical element to the visual representation.

[0069] In some implementations, the actions further include: presenting a code stub for customizing a metric of the artificial neural network; and customizing the metric of the artificial neural network in response to an edit operation on the code stub.

[0070] In some implementations, the action also includes: modifying the intermediate representation of the artificial neural network in response to at least one of the following operations: adding new graphic elements representing layers of the artificial neural network to the visual representation; deleting graphic elements representing layers of the artificial neural network from the visual representation; modifying connection relationships between graphic elements representing layers of the artificial neural network; and modifying parameters of graphic elements representing layers of the artificial neural network.

[0071] In some implementations, the actions further include: in response to receiving an indication to modify the target deep learning framework to another target deep learning framework, determining a code of the artificial neural network for another target deep learning framework based on the intermediate representation of the artificial neural network.

[0072] In a third aspect, the present disclosure provides a computer program product, which is tangibly stored in a non-transitory computer storage medium and includes computer executable instructions, which when executed by a device cause the device to perform the method in the first aspect of the present disclosure.

[0073] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a device, cause the device to perform the method in the first aspect of the present disclosure.

[0074] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0075] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0076] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0077] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a separate implementation can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination.

[0078] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A computer-implemented method comprising: presenting a visual representation of an artificial neural network, wherein the visual representation of the artificial neural network includes graphical elements representing layers of the artificial neural network; receiving a drag-and-drop operation for the graphic elements, the drag-and-drop operation being configured to modify a connection relationship between the graphic elements, including automatically connecting a first graphic element to a second graphic element of the graphic elements; In response to receiving the drag-and-drop operation, altering an intermediate representation of the artificial neural network, the intermediate representation of the artificial neural network being independent of a deep learning framework; as well as Based on the intermediate representation of the artificial neural network, the code of the artificial neural network for the target deep learning framework is modified.

2. The computer-implemented method of claim 1 , further comprising: In response to an edit operation on a code of the artificial neural network for a target deep learning framework, modifying an intermediate representation of the artificial neural network; as well as The visual representation of the artificial neural network is adjusted based on the intermediate representation of the artificial neural network.

3. The computer-implemented method of claim 1 , further comprising: In response to receiving a drag and drop operation for the graphical element, a dimension of data associated with a layer of the artificial neural network is verified.

4. The computer-implemented method of claim 1 , further comprising: In response to receiving a search operation associated with a keyword, presenting a graphical element representing at least one candidate layer corresponding to the keyword; as well as In response to receiving a selection of a graphical element for the at least one candidate layer, adding the selected graphical element for the at least one candidate layer to the visual representation of the artificial neural network.

5. The computer-implemented method of claim 1 , further comprising: Rendering a code stub for customizing metrics of the artificial neural network; as well as In response to an edit operation on the code stub, a metric of the artificial neural network is customized.

6. The computer-implemented method of claim 1 , further comprising: Modifying an intermediate representation of the artificial neural network in response to at least one of the following: adding a new graphical element representing a layer of an artificial neural network to the visual representation of the artificial neural network; Deleting graphical elements representing layers of an artificial neural network from the visual representation of the artificial neural network; and Modify the parameters of a graphical element representing a layer of an artificial neural network.

7. A device comprising: Processing circuit; as well as The memory includes instructions that, when executed by the processing circuit, cause the processing circuit to perform operations including: presenting a visual representation of an artificial neural network, wherein the visual representation of the artificial neural network includes graphical elements representing layers of the artificial neural network; receiving a drag-and-drop operation for the graphic elements, the drag-and-drop operation being configured to modify a connection relationship between the graphic elements, including automatically connecting a first graphic element to a second graphic element of the graphic elements; In response to receiving the drag-and-drop operation, altering an intermediate representation of the artificial neural network, the intermediate representation of the artificial neural network being independent of a deep learning framework; and Based on the intermediate representation of the artificial neural network, the code of the artificial neural network for the target deep learning framework is modified.

8. The apparatus of claim 7, wherein the instructions further cause the processing circuit to perform operations comprising: In response to an edit operation on a code of the artificial neural network for a target deep learning framework, modifying an intermediate representation of the artificial neural network; as well as The visual representation of the artificial neural network is adjusted based on the intermediate representation of the artificial neural network.

9. The apparatus of claim 7, wherein the instructions further cause the processing circuit to perform operations comprising: In response to receiving a drag and drop operation for the graphical element, a dimension of data associated with a layer of the artificial neural network is verified.

10. The apparatus of claim 7, wherein the instructions further cause the processing circuit to perform operations comprising: In response to receiving a search operation associated with a keyword, presenting a graphical element representing at least one candidate layer corresponding to the keyword; as well as In response to receiving a selection of a graphical element for the at least one candidate layer, adding the selected graphical element of the at least one candidate layer to the visual representation of the artificial neural network.

11. The apparatus of claim 7, wherein the instructions further cause the processing circuit to perform operations comprising: Rendering a code stub for customizing metrics of the artificial neural network; as well as In response to an edit operation on the code stub, a metric of the artificial neural network is customized.

12. The apparatus of claim 7, wherein the instructions further cause the processing circuit to perform operations comprising: Modifying an intermediate representation of the artificial neural network in response to at least one of the following: adding a new graphical element representing a layer of an artificial neural network to the visual representation of the artificial neural network; Deleting graphical elements representing layers of an artificial neural network from the visual representation of the artificial neural network; and Modify the parameters of a graphical element representing a layer of an artificial neural network.

13. The apparatus of claim 7, wherein the instructions further cause the processing circuit to perform operations comprising: In response to receiving an instruction to change the target deep learning framework to another target deep learning framework, a code of the artificial neural network for the other target deep learning framework is determined based on the intermediate representation of the artificial neural network.

14. A non-transitory machine-readable medium comprising instructions that, when executed by a processing circuit, cause the processing circuit to perform operations comprising: presenting a visual representation of an artificial neural network, wherein the visual representation of the artificial neural network includes graphical elements representing layers of the artificial neural network; receiving a drag-and-drop operation for the graphic elements, the drag-and-drop operation being configured to modify a connection relationship between the graphic elements, including automatically connecting a first graphic element to a second graphic element of the graphic elements; In response to receiving the drag-and-drop operation, altering an intermediate representation of the artificial neural network, the intermediate representation of the artificial neural network being independent of a deep learning framework; as well as Based on the intermediate representation of the artificial neural network, the code of the artificial neural network for the target deep learning framework is modified.

15. The non-transitory machine-readable medium of claim 14, wherein the instructions further cause the processing circuit to perform operations comprising: In response to an edit operation on a code of the artificial neural network for a target deep learning framework, modifying an intermediate representation of the artificial neural network; as well as The visual representation of the artificial neural network is adjusted based on the intermediate representation of the artificial neural network.

16. The non-transitory machine-readable medium of claim 14, wherein the instructions further cause the processing circuit to perform operations comprising: In response to receiving a drag and drop operation for the graphical element, a dimension of data associated with a layer of the artificial neural network is verified.

17. The non-transitory machine-readable medium of claim 14, wherein the instructions further cause the processing circuit to perform operations comprising: In response to receiving a search operation associated with a keyword, presenting a graphical element representing at least one candidate layer corresponding to the keyword; as well as In response to receiving a selection of a graphical element for the at least one candidate layer, adding the selected graphical element for the at least one candidate layer to the visual representation of the artificial neural network.

18. The non-transitory machine-readable medium of claim 14, wherein the instructions further cause the processing circuit to perform operations comprising: Rendering a code stub for customizing metrics of the artificial neural network; as well as In response to an edit operation on the code stub, a metric of the artificial neural network is customized.

19. The non-transitory machine-readable medium of claim 14, wherein the instructions further cause the processing circuit to perform operations comprising: Modifying an intermediate representation of the artificial neural network in response to at least one of the following: adding a new graphical element representing a layer of an artificial neural network to the visual representation of the artificial neural network; Deleting graphical elements representing layers of an artificial neural network from the visual representation of the artificial neural network; and Modify the parameters of a graphical element representing a layer of an artificial neural network.

20. The non-transitory machine-readable medium of claim 14, wherein the instructions further cause the processing circuit to perform operations comprising: In response to receiving an instruction to change the target deep learning framework to another target deep learning framework, a code of the artificial neural network for the other target deep learning framework is determined based on the intermediate representation of the artificial neural network.

Images