Method and apparatus for visualizing neural network model generation and training

Abstract

The application provides a method and device for visualizing generation and training of a neural network model, the method comprising: obtaining a visual model file of a neural network model generated by a visual editor; converting the visual model file into an XML text file; the XML text file being a model file conforming to a preset unified modeling standard; and generating a target neural network model based on the XML text file. The method and device for visualizing generation and training of a neural network model provided by the application can generate a visual model meeting user requirements through a visual model construction interactive framework, and then convert the visual model into a model file conforming to a unified modeling standard, so that cross-platform convenient training of the model can be realized using the model file, the complexity of the model generation and training process is reduced, and the model generation process is more intuitive.

Description

Technical Field

[0001] This application relates to the field of deep learning, and in particular to a method and apparatus for visualizing neural network model generation and training. Background Technology

[0002] Artificial intelligence (AI) is a branch of computer science that attempts to understand the nature of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. Since its inception, AI has seen its theories and technologies mature, and its application areas continue to expand.

[0003] With the rapid development of artificial intelligence, the in-depth practical application of AI in key fields faces a series of challenges, including complex models, numerous tools, large data volumes, and long computation times. These challenges not only increase the difficulty of combining intelligent needs with intelligent generation in key fields but also raise the difficulty and threshold for AI research and application in these areas. Furthermore, the traditional code-based neural network modeling process requires users to have a deep understanding of algorithm principles and in-depth business research. Users need to fully understand the specialized construction knowledge of neural networks based on different frameworks such as TensorFlow and PyTorch, and abstractly construct models using code supported by different deep learning frameworks. This results in a high application threshold for deep learning modeling and training, and the relatively high coupling between neural network modeling and training techniques and computer implementation techniques leads to complex and unintuitive model generation processes and high training thresholds. Summary of the Invention

[0004] This application provides a method and apparatus for generating and training a visual neural network model, in order to solve the technical problems of complex and unintuitive model generation processes and high training thresholds in related technologies.

[0005] In a first aspect, embodiments of this application provide a method for generating and training a visual neural network model, including:

[0006] Obtain the visualization model file of the neural network model generated by the visualization editor;

[0007] The visualization model file is converted into an XML text file; the XML text file is a model file that conforms to a preset unified modeling standard.

[0008] Based on the XML text file, a target neural network model is generated; the target neural network model is a neural network model that has completed model training under a specified deep learning framework based on a preset unified modeling standard.

[0009] In some embodiments, converting the visualization model file into an XML text file includes:

[0010] A first model document tree is generated based on the visualized model file; the first model document tree is a visualized model document tree.

[0011] The first model document tree is converted into a second model document tree; the second model document tree is a model document tree that conforms to a preset unified modeling standard;

[0012] The second model document tree text is serialized to obtain the XML text file.

[0013] In some embodiments, converting the first model document tree into a second model document tree includes:

[0014] Traverse the element nodes in the first model document tree;

[0015] Determine the element nodes and element connections in the first model document tree;

[0016] Convert the element nodes in the first model document tree into element nodes in the second model document tree;

[0017] The element connection relationships in the first model document tree are converted into element connection relationships in the second model document tree.

[0018] In some embodiments, converting element nodes in the first model document tree into element nodes in the second model document tree includes:

[0019] When the node number is a first preset value, the attribute information of the current node number in the first model document tree is read, and the root node of the second model document tree is generated based on the attribute information of the current node number.

[0020] In some embodiments, converting element nodes in the first model document tree into element nodes in the second model document tree includes:

[0021] If the node number is greater than a first preset value, read the node type information of the current node number in the first model document tree, and generate the corresponding node of the second model document tree based on the node type information of the current node number.

[0022] In some embodiments, converting the element connection relationships in the first model document tree into element connection relationships in the second model document tree includes:

[0023] Nodes in the first model document tree whose edge attribute is a second preset value are identified as connection nodes;

[0024] Determine the element connection relationships of the connected nodes;

[0025] Based on the element connection relationship of the connected nodes, the parent-child relationship of the nodes in the second model document tree corresponding to the connected nodes is determined.

[0026] In some embodiments, converting the element connection relationships in the first model document tree into element connection relationships in the second model document tree includes:

[0027] Based on the parent attribute information of the element nodes in the first model document tree, the parent-child relationship of the nodes in the second model document tree is adjusted.

[0028] In some embodiments, generating a target neural network model based on the XML text file includes:

[0029] Based on the preset unified modeling standard, the XML text file is parsed to generate model training information under the specified deep learning framework;

[0030] Based on the model training information, the training of a neural network model under the specified deep learning framework is completed, and the target neural network model is generated.

[0031] Secondly, embodiments of this application also provide an apparatus for visualizing neural network model generation and training, comprising:

[0032] The first acquisition module acquires the visualization model file of the neural network model generated by the visualization editor;

[0033] The first conversion module is used to convert the visualization model file into an XML text file; the XML text file is a model file that conforms to a preset unified modeling standard.

[0034] The first generation module is used to generate a target neural network model based on the XML text file; the target neural network model is a neural network model that has been trained under a specified deep learning framework based on a preset unified modeling standard.

[0035] In some embodiments, the first conversion module includes a first generation submodule, a first conversion submodule, and a first execution submodule, wherein:

[0036] The first generation submodule is used to generate a first model document tree based on the visualized model file; the first model document tree is a visualized model document tree;

[0037] The first conversion submodule is used to convert the first model document tree into a second model document tree; the second model document tree is a model document tree that conforms to a preset unified modeling standard;

[0038] The first execution submodule is used to serialize the second model document tree text to obtain the XML text file.

[0039] In some embodiments, the first conversion submodule includes a first traversal unit, a first determination unit, a first conversion unit, and a second conversion unit, wherein:

[0040] The first traversal unit is used to traverse the element nodes in the first model document tree;

[0041] The first determining unit is used to determine the element nodes and element connections in the first model document tree;

[0042] The first conversion unit is used to convert element nodes in the first model document tree into element nodes in the second model document tree;

[0043] The second conversion unit is used to convert the element connection relationship in the first model document tree into the element connection relationship in the second model document tree.

[0044] In some embodiments, the first conversion unit includes:

[0045] The first reading subunit is used to read the attribute information of the current node number in the first model document tree when the node number is a first preset value;

[0046] The first generation subunit is used to generate the root node of the second model document tree based on the attribute information of the current node number.

[0047] In some embodiments, the first conversion unit includes:

[0048] The second reading subunit is used to read the node type information of the current node number in the first model document tree when the node number is greater than the first preset value;

[0049] The second generation subunit is used to generate the corresponding node of the second model document tree based on the node type information of the current node number.

[0050] In some embodiments, the second conversion unit includes:

[0051] The first determining subunit is used to determine that nodes in the first model document tree whose edge attribute is a second preset value are connecting nodes.

[0052] The second determining subunit is used to determine the element connection relationship of the connecting nodes;

[0053] The third determining subunit is used to determine the parent-child relationship of the nodes in the second model document tree corresponding to the connecting nodes based on the element connection relationship of the connecting nodes.

[0054] In some embodiments, the second conversion unit includes:

[0055] The first adjustment subunit is used to adjust the parent-child relationship of nodes in the second model document tree based on the parent attribute information of element nodes in the first model document tree.

[0056] In some embodiments, the first generation module includes a second generation submodule and a third generation submodule, wherein:

[0057] The second generation submodule is used to parse the XML text file based on the preset unified modeling standard and generate model training information under the specified deep learning framework;

[0058] The third generation submodule is used to train a neural network model under a specified deep learning framework based on the model training information, and generate the target neural network model.

[0059] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for generating and training a visual neural network model as described above.

[0060] Fourthly, embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method for generating and training a visual neural network model as described above.

[0061] Fifthly, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the method for generating and training a visual neural network model as described above.

[0062] The method and apparatus for generating and training a visual neural network model provided in this application can generate a visual model that meets user needs by constructing an interactive framework through a visual model. The visual model is then converted into a model file that conforms to a unified modeling standard. Using this model file, cross-platform training of the model can be achieved, reducing the complexity of the model generation and training process and making the model generation process more intuitive. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart illustrating the method for generating and training a visual neural network model provided in an embodiment of this application;

[0065] Figure 2 This is a logical flowchart illustrating the principle of generating and training a visualized neural network model provided in the embodiments of this application;

[0066] Figure 3 This is a schematic diagram of the structure of a device for generating and training a visual neural network model according to an embodiment of this application;

[0067] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0068] For users who focus on application business, being able to skip the tedious process of learning related code programming and directly analyze their conceived neural network models in a more intuitive, concise, and efficient way through visual modeling is a very important user need. This lowers the application threshold for users to conduct deep learning modeling and training and helps users in specific fields quickly customize intelligent models.

[0069] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0070] Figure 1 This is a flowchart illustrating the method for generating and training a visual neural network model provided in an embodiment of this application, as shown below. Figure 1 As shown in the embodiments of this application, a method for generating and training a visual neural network model is provided, including:

[0071] Step 101: Obtain the visualization model file of the neural network model generated by the visualization editor.

[0072] Specifically, the method for generating and training visual neural network models provided in the application embodiments offers users a system entry point for quickly creating visual modeling tasks. This entry point can complete system environment construction tasks such as loading the visual operator structure system and initializing the operation area of ​​the visual model building interactive framework. It supports users in creating entirely new visual models on demand or importing existing visual model files for iteration. If a user imports a visual model file, the system can parse its content into a drag-and-drop display format within the visual model building interactive framework.

[0073] Within the visual model building framework, users can construct supervised learning intelligent models on demand through a user-friendly interface. They can customize their models based on the interfaces of the underlying open-source libraries in the visual editor by using visual drag-and-drop operators and appropriately granular scaling. If the model generation process is not yet complete, the current visual model file can be saved, and users can import it later to continue the model generation process as needed.

[0074] Step 102: Convert the visualization model file into an XML text file; the XML text file is a model file that conforms to a preset unified modeling standard.

[0075] Specifically, once the visualization model generation process is complete, the generated visualization model file can be converted into a model file conforming to a preset unified modeling standard. This unified modeling standard assists and constrains the generation of cross-platform neural network models, specifically, it assists and constrains the generation of trainable neural network models. Furthermore, under the joint modeling system of visualization and standardization, the joint modeling process of converting the visualization model file into an XML text file can reduce the influence of specific deep learning framework syntax and specific network frameworks.

[0076] This unified modeling standard can be used to validate the generated XML text. Specifically, it checks whether the XML text meets the encoding requirements and whether it complies with the relevant restrictions of the unified modeling standard.

[0077] Unified modeling standards include, but are not limited to, one or more of the following: network organizational structure specifications, element specifications, element length specifications, element type specifications, etc., and the embodiments of this invention do not specifically limit these.

[0078] Step 103: Generate a target neural network model based on the XML text file; the target neural network model is a neural network model that has completed model training under a specified deep learning framework based on a preset unified modeling standard.

[0079] Specifically, based on a unified modeling standard, XML text files can be converted into neural network models adaptable to a specified platform, and then training can be performed. These XML text files enable convenient cross-platform training. Furthermore, the standardized model file (XML model file) obtained from the visualization model already specifies core training elements such as dataset, hyperparameters, model structure, and deep learning framework. Using this XML as input, under the visualization and standardization joint modeling system, the file can be parsed to complete the training of the neural network model under the specified deep learning framework, generating the required neural network model.

[0080] For example, XML text files can be converted into corresponding neural network models under the PyTorch deep learning framework according to the generation requirements, and then distributed parallel training can be performed.

[0081] For example, XML text files can be converted into corresponding neural network models under the TensorFlow deep learning framework according to the generation requirements, and then distributed parallel training can be performed.

[0082] The method for generating and training a visual neural network model provided in this application constructs an interactive framework through a visual model, which can generate a visual model that meets user needs. The visual model is then converted into a model file that conforms to a unified modeling standard. Using this model file, cross-platform training of the model can be achieved, reducing the complexity of the model generation and training process and making the model generation process more intuitive.

[0083] In some embodiments, converting the visualization model file into an XML text file includes:

[0084] A first model document tree is generated based on the visualized model file; the first model document tree is a visualized model document tree.

[0085] The first model document tree is converted into a second model document tree; the second model document tree is a model document tree that conforms to a preset unified modeling standard;

[0086] The second model document tree text is serialized to obtain the XML text file.

[0087] Specifically, the visual model document tree can be obtained based on the generated visual model file and the visual editor. By calling the getGraphXml interface of the visual editor, the XMLDOC Document instance corresponding to the visual model drawing can be obtained, thus obtaining the visual model document tree. The visual model document tree is then converted into a model document tree conforming to a preset unified modeling standard, including but not limited to the conversion of all element nodes and element connection relationships according to the unified modeling standard. Finally, the model document tree conforming to the preset unified modeling standard is serialized into text and saved as an XML text file, i.e., a standardized model XML file.

[0088] The method for generating and training a visual neural network model provided in this application converts the visual model into an XML text file according to a unified modeling standard, which enables convenient cross-platform training of the model, reduces the complexity of the model generation and training process, and makes the model generation process more intuitive.

[0089] In some embodiments, converting the first model document tree into a second model document tree includes:

[0090] Traverse the element nodes in the first model document tree;

[0091] Determine the element nodes and element connection relationships in the first model document tree;

[0092] Convert the element nodes in the first model document tree into element nodes in the second model document tree;

[0093] The element connection relationships in the first model document tree are converted into element connection relationships in the second model document tree.

[0094] Specifically, the transformation of the model document tree requires traversing all element nodes in the visual model document tree, determining all element nodes and element connection relationships in the visual model document tree, and converting these element nodes and element connection relationships into element nodes and element connection relationships of the model document tree that conform to the preset unified modeling standard.

[0095] Furthermore, object instances in the model document of the unified modeling standard can be initialized first. Then, all element nodes in the visualization model document tree are traversed using a multi-way tree preorder traversal algorithm. Virtual nodes with node number (id) 0 can be ignored during traversal.

[0096] The method for generating and training a visual neural network model provided in this application improves the stability of model conversion and thus the accuracy of the generated XML text file by traversing the visual model document tree and converting the visual model into an XML text file according to a unified modeling standard.

[0097] In some embodiments, converting element nodes in the first model document tree into element nodes in the second model document tree includes:

[0098] When the node number is a first preset value, the attribute information of the current node number in the first model document tree is read, and the root node of the second model document tree is generated based on the attribute information of the current node number.

[0099] Specifically, when traversing the visualization model document tree, if the accessed node number is the first preset value, only its attribute information is read, and an element node with corresponding attribute information is constructed based on this attribute information. Then, it is inserted into the model document tree of the unified modeling standard to form the root node of the model document.

[0100] For example, if the accessed node number is 1, the attribute information of the node with the number 1 is read, and an element node with corresponding attribute information is constructed based on this attribute information. Then, it is inserted into the model document tree of the unified modeling standard to form the root node of the model document.

[0101] The method for generating and training a visual neural network model provided in this application improves the stability of model conversion and thus the accuracy of the generated XML text file by traversing the visual model document tree and converting the visual model into an XML text file according to a unified modeling standard.

[0102] In some embodiments, converting element nodes in the first model document tree into element nodes in the second model document tree includes:

[0103] If the node number is greater than a first preset value, read the node type information of the current node number in the first model document tree, and generate the corresponding node of the second model document tree based on the node type information of the current node number.

[0104] Specifically, when traversing the visualization model document tree, if the accessed node number is greater than a first preset value, the node type information of the current node number in the visualization model document tree is read, and then a corresponding model node is created and inserted into the model document tree of the unified modeling standard. The created model nodes can include dataset nodes, parameter nodes, block nodes, layer nodes, vertex nodes, and their associated attribute information.

[0105] For example, if the accessed node number is greater than 1, the type information of the node with a node number greater than 1 in the visualization model document tree is read, and then the corresponding model node is created and inserted into the model document tree of the unified modeling standard.

[0106] The method for generating and training a visual neural network model provided in this application improves the stability of model conversion and thus the accuracy of the generated XML text file by traversing the visual model document tree and converting the visual model into an XML text file according to a unified modeling standard.

[0107] In some embodiments, converting the element connection relationships in the first model document tree into element connection relationships in the second model document tree includes:

[0108] Nodes in the first model document tree whose edge attribute is a second preset value are identified as connection nodes;

[0109] Determine the element connection relationships of the connected nodes;

[0110] Based on the element connection relationship of the connected nodes, the parent-child relationship of the nodes in the second model document tree corresponding to the connected nodes is determined.

[0111] Specifically, if an element node in the visualization model document tree has an edge attribute with a second preset value, it is determined to be a connection node. Then, the element connection relationship of this connection node is calculated based on its source and target attributes. Finally, based on the element connection relationship of this connection node, the parent attribute information of the corresponding node is set in the unified modeling standard model document tree. The second preset value can be 1.

[0112] The method for generating and training a visual neural network model provided in this application improves the stability of model conversion and thus the accuracy of the generated XML text file by traversing the visual model document tree and converting the visual model into an XML text file according to a unified modeling standard.

[0113] In some embodiments, converting the element connection relationships in the first model document tree into element connection relationships in the second model document tree includes:

[0114] Based on the parent attribute information of the element nodes in the first model document tree, the parent-child relationship of the nodes in the second model document tree is adjusted.

[0115] Specifically, all element nodes in the visualization model document tree are traversed to determine the parent attribute information of the traversed element nodes. Based on the parent attribute information of these element nodes, the parent-child relationship of the nodes in the unified modeling standard model document tree is adjusted to form the correct tree structure.

[0116] The method for generating and training a visual neural network model provided in this application improves the stability of model conversion and thus the accuracy of the generated XML text file by traversing the visual model document tree and converting the visual model into an XML text file according to a unified modeling standard.

[0117] In some embodiments, generating a target neural network model based on the XML text file includes:

[0118] Based on the preset unified modeling standard, the XML text file is parsed to generate model training information under the specified deep learning framework;

[0119] Based on the model training information, the training of a neural network model under the specified deep learning framework is completed, and the target neural network model is generated.

[0120] Specifically, based on a unified modeling standard, XML text files can be converted into neural network models adaptable to a specified platform, and then training can be performed. These XML text files enable convenient cross-platform training. Furthermore, the standardized model file (XML model file) obtained from the visualization model already specifies core training elements such as dataset, hyperparameters, model structure, and deep learning framework. Using this XML as input, under the visualization and standardization joint modeling system, the file can be parsed to complete the training of the neural network model under the specified deep learning framework, generating the required neural network model.

[0121] The method for generating and training a visual neural network model provided in this application converts the visual model into a model file that conforms to a unified modeling standard. Using this model file, cross-platform training of the model can be achieved, reducing the complexity of the model generation and training process.

[0122] The methods described in the above embodiments will be further illustrated below with specific examples.

[0123] Figure 2 This is a logical flowchart illustrating the principles of generating and training a visualized neural network model provided in this application embodiment, such as... Figure 2 As shown in the embodiment of this application, a system entry point is provided for users to quickly create visual modeling tasks, which is the beginning of the execution flow of this invention. This entry point can complete system environment construction such as loading the visual operator structure system and initializing the operation area of ​​the visual model building interactive framework. It supports users to create brand new visual models on demand or import existing visual model files for iteration. If the user imports a visual model file, the system can parse its content into a form that supports drag-and-drop operations in the visual model building interactive framework.

[0124] Within the visual model building interactive framework, users can construct supervised learning intelligent models on demand through a user-friendly interface. Specifically, users can use the visual model building interactive framework, along with built-in visual operators and user-defined operator templates, to construct supervised learning intelligent models as needed through visual drag-and-drop operators and appropriately granular scaling interactions.

[0125] If the user model building process is not yet complete, the current visualization model file can be saved, and users can import it as needed to continue the modeling process. If the user model building process is complete, while saving the visualization model file for subsequent iterations, based on the visualization and standardization joint modeling system planned in the application implementation, it supports the generation of a model file oriented towards a unified modeling standard. This file decouples the user model and the deep learning framework and can be used for subsequent training tasks. Users can set and submit standardized model training tasks based on the generated unified modeling standard model file. Relying on the visualization and standardization joint modeling training method, users can focus on the training dataset, training hyperparameters, and intelligent model application customization. The system can automatically allocate computing resources to execute training tasks based on the existing distributed parallel training framework.

[0126] Finally, users can determine whether the expected requirements are met based on the training results and logs during or after training, and iterate the training of the neural network model as needed. If the training results of the neural network model do not meet the user's needs, the corresponding visual model file associated with the unified modeling standard model file can be imported into the system, and the interactive framework built through the visual model can be used to iterate multiple times to meet the user's needs. If the training results meet the expected requirements, the training results are saved as a model file of the required deep learning framework type. At this point, the visual supervised learning distributed parallel training task is completed.

[0127] This invention provides a method for generating and training visual neural network models. It establishes a visual model building and interaction framework for users, supporting visual drag-and-drop operators and appropriately granular scaling interactions to achieve joint modeling responses based on visualization and standardization. It generates model files with unified modeling standards and supports the scalable representation capabilities of application-domain customized algorithms from the bottom up. Combined with the invented cross-platform neural network standardization definition technology, it supports distributed parallel training of unified modeling standard model files. Therefore, it reduces the tight coupling between neural network modeling technology and computer implementation technology, minimizes the R&D impact of differences in the internal structure of intelligent application models and cross-deep learning frameworks, and lowers the professional threshold for building and training deep learning models in application domains.

[0128] A prototype system was developed based on the method of this invention. This system includes a visual model building and interactive framework loading module, a visual supervised learning model interactive building module, a model file generation module oriented towards a unified modeling standard, a visual supervised learning distributed parallel training module, and a visual supervised learning iterative modeling support module, wherein:

[0129] The Visual Model Building Interactive Framework Loading Module serves as the initial entry point for users to build and train visual supervised learning models using the prototype system. Before users actually create visual modeling tasks, this module supports the initialization of basic visual modeling capabilities and supporting services, including loading the visual operator structure system and initializing the operation area of ​​the visual model building interactive framework. Regarding basic visual modeling capabilities, it primarily loads the fundamental elements and standard specifications in the process of visual modeling and standardized joint modeling, involving visual building components, visual modeling operator libraries, and basic mapping and transformation logic. In terms of visual modeling support services, it primarily initializes the important supporting capabilities and processing frameworks required during the visual modeling process, including a visual drag-and-drop interactive sub-framework, a built-in visual model sub-framework, a standard pattern verification sub-framework, and a transformation exception handling sub-framework.

[0130] The Visual Supervised Learning Model Interaction Builder module is the foundational and core module supporting users in building visual supervised learning models using the prototype system. It primarily provides a selection of visual operator components and is customized based on the interface of the open-source library GraphEditor mxGraph. It supports supervised learning model construction through visual drag-and-drop operators and appropriately granular scaling interactions. This visual operator component is built upon a visual operator library that supports integrated modeling and training. Based on its function, it is mainly divided into data processing operators, training configuration operators, and model structure operators, further subdivided into three levels according to their relationship. Model structure operators are further divided into layer-level operators (level 1, 2, and 3) and block-level operators (level 1, 2, and 3) based on application needs. Layer-level operators are built-in visual modeling operators, while block-level operators provide users with custom operator templates. Block-level operators allow users to build composite operator structures as needed using layer-level operators.

[0131] For example, the first-level operator in the training configuration class can be weights, used to set the weight initialization method for convolutional kernels and fully connected layers. The second-level operator is the weight type, used to specify the weight type. The third-level operator is random normal distribution initialization, used to sample and initialize weights from a normal distribution.

[0132] For example, the first-level operator in a layer hierarchy can be a convolutional layer, used to extract features from the input data, and the size of the output feature map is determined by the parameters of the convolutional layer. The second-level operator is the activation function, used to specify the activation function of the convolutional layer. The third-level operator is the softmax activation function, used for the output of a multi-class neural network.

[0133] The model file generation module, designed for a unified modeling standard, is a core module that enables seamless training of visually supervised learning models after their construction. Under the visualization and standardization joint modeling system planned in this invention, the influence of specific deep learning framework syntax and network frameworks is reduced. Through this joint modeling response, the model structure, visually constructed, can be converted into a standardized model file that can be easily trained across platforms. This module is implemented using a model file generation algorithm based on a unified modeling standard. The specific algorithms are as follows:

[0134] A model file generation algorithm that standardizes modeling.

[0135] Input: A visual model edited by a visual editor.

[0136] Output: A model XML text file (file) conforming to the unified modeling standard.

[0137] process:

[0138] (1) Obtain the visualization model document tree from the visualization editor.

[0139] (1.1) Call the getGraphXml interface of the visual editor to obtain the XML DOMDocument instance corresponding to the current drawing, and name it "graph".

[0140] (2) Traverse the visualization model document tree graph and convert the element nodes and element connections in the document into a model document tree with a unified modeling standard.

[0141] (2.1) Call the parseFromString method of the window.DOMParser class to initialize a unified model document XML DOM Document object instance named "model".

[0142] (2.2) Traverse all nodes in the graph document using the multi-way tree preorder traversal algorithm (ignore virtual nodes with id 0).

[0143] (2.2.1) If a node with an ID of 1 is encountered, its attribute information is read only, and a “model” node is constructed, the corresponding attribute information is set, and then it is inserted into the model document tree to form the root node of the model document.

[0144] (2.2.2) If a node with an ID greater than 1 is encountered, its type attribute information is read, and a corresponding model node (dataset, parameters, block, layer, vertex and its associated attribute information) is created and inserted into the model document tree.

[0145] (2.2.3) If a drawing node with an edge attribute of "1" is encountered, it is determined to be a connection node. The connection relationship is calculated based on its source attribute and target attribute information, and the parent attribute information is set for the corresponding node in the model.

[0146] (2.2.4) Adjust the parent-child relationship of nodes in the model for all traversed drawing nodes based on the parent attribute information to form the correct tree structure.

[0147] (3) Serialize the model document tree into text and save it as an XML text file.

[0148] (3.1) Call the getPrettyXml interface of the mxUtils class in the mxGraph library to perform text serialization conversion on the model document tree instance.

[0149] (3.2) Create a Blob instance from the converted text serialization content using the Blob class, and call the window.URL.createObjectURL interface to convert the Blob instance into an ObjectUrl instance.

[0150] (3.3) Create an HTML hyperlink element a, set the href attribute of a to an instance of ObjectUrl, and call its click method to download the model XML content to a local file as a text attachment.

[0151] The Visualized Supervised Learning Distributed Parallel Training Module is an execution module that supports users in performing distributed training using model files conforming to the unified modeling standard generated by the aforementioned modules. In this module, the model files conforming to the unified modeling standard serve as the output of the joint modeling response of visualization and standardization. These outputs can then be used as input for cross-platform neural network standardized model training tasks based on existing distributed parallel training frameworks. Through the cross-platform neural network standardization definition technology provided in the preceding invention, these model files conforming to the unified modeling standard can be converted into trainable PyTorch or TensorFlow neural network models via this module, and further training can be performed.

[0152] The Visualized Supervised Learning Iterative Modeling Support Module provides users with a convenient way to iteratively build and train visualized supervised learning models. This module supports saving visualized model files for subsequent iterations and provides an interface for importing visualized model files. It also supports parsing these files into a drag-and-drop display format within the interactive framework for building visualized models. This parsing process is implemented using a model file visualization algorithm. The specific algorithms are as follows:

[0153] Model file visualization algorithm.

[0154] Input: XML instance text of the model file.

[0155] Output: A visual model displayed using a visual editor.

[0156] process:

[0157] (1) Load the XML text content of the model file and parse it into an XML document tree model.

[0158] (1.1) Read the text content of the local model XML document through the browser's input form control.

[0159] (1.2) Call the parseFromString method of the window.DOMParser class to parse the XML text content into an XML DOM Document object instance, named model.

[0160] (2) Initialize the visualization model mxGraph document tree graph.

[0161] (2.1) Construct an instance of the XML DOM Document object for initializing mxGraph using the parseFromString method of the window.DOMParser class, and name it graph.

[0162] (2.2) Set the basic attribute information of the drawing document, and insert the drawing root node and the first element for subsequent node addition and management.

[0163] (3) Traverse all nodes in the model and node attributes, and generate corresponding graph nodes and connections based on node names.

[0164] (3.1) Use the multi-branch tree preorder traversal algorithm to traverse all nodes in the model.

[0165] (3.1.1) If the node name is “model”, record the basic attribute information of the node and continue to traverse.

[0166] (3.1.2) If the node name is “dataset” or “parameters”, then insert the parameter configuration information plotting node into the graph document.

[0167] (3.1.3) If the node name is "block", "layer", "vertex", etc., then insert the corresponding drawing node information into the graph document. At the same time, read the detailed attribute information of the node using the getAttribute method and insert it into the corresponding drawing node in the graph document as a basic text node.

[0168] (3.1.3.1) The detailed attribute information of the node read includes the parent attribute field.

[0169] (3.1.3.1.1) Traverse all drawing nodes in the current graph, find matching nodes, and construct connection relationships between the drawing nodes.

[0170] (3.1.3.1.2) Insert the connecting plotting nodes into the graph document.

[0171] (3.1.3.2) The detailed attribute information of the node read does not include the parent attribute field.

[0172] (3.1.3.2.1) By default, the drawing node establishes a connection relationship with the previous drawing node.

[0173] (3.1.3.2.2) Insert the connecting plot nodes into the graph document.

[0174] (4) Pass the graph document tree content to the mxGraph visualization editor for visualization display.

[0175] (4.1) Create a drawing editor instance named "editor" using the Editor class of the mxGraph library.

[0176] (4.2) Pass the graph document instance to the editor for plot display.

[0177] (4.3) Call the editor's layout interface to automatically lay out the graph nodes.

[0178] (4.4) Call the expand node interface of the editor to expand all expandable nodes in the graph.

[0179] (4.5) The nodes of the graph are automatically resized using the editor’s automatic node resizing interface.

[0180] (4.6) Call the editor's layout interface again to perform a second automatic layout on the nodes after they have been expanded and automatically resized.

[0181] The method for generating and training visualized neural network models provided in this application addresses the difficulty and threshold issues in the research and application of artificial intelligence in key fields. Under the constraints of a visualization and standardized joint modeling system, it utilizes the service calls provided by the visualization model building interaction framework. Users can drag and scale operators as needed to build supervised learning intelligent models, and it supports generating model files with unified modeling standards as input for standardized model training tasks, thus completing model training in a coordinated manner and reducing the professional threshold for building and training deep learning models in application fields.

[0182] Figure 3 This is a schematic diagram of the structure of a device for generating and training a visual neural network model, as provided in an embodiment of this application. Figure 3 As shown in the embodiments of this application, the apparatus for generating and training a visual neural network model includes a first acquisition module 301, a first conversion module 302, and a first generation module 303, wherein:

[0183] The first acquisition module 301 acquires the visualization model file of the neural network model generated by the visualization editor;

[0184] The first conversion module 302 is used to convert the visualization model file into an XML text file; the XML text file is a model file that conforms to a preset unified modeling standard.

[0185] The first generation module 303 is used to generate a target neural network model based on the XML text file; the target neural network model is a neural network model that has completed model training under a specified deep learning framework based on a preset unified modeling standard.

[0186] In some embodiments, the first conversion module includes a first generation submodule, a first conversion submodule, and a first execution submodule, wherein:

[0187] The first generation submodule is used to generate a first model document tree based on the visualized model file; the first model document tree is a visualized model document tree;

[0188] The first conversion submodule is used to convert the first model document tree into a second model document tree; the second model document tree is a model document tree that conforms to a preset unified modeling standard;

[0189] The first execution submodule is used to serialize the second model document tree text to obtain the XML text file.

[0190] In some embodiments, the first conversion submodule includes a first traversal unit, a first determination unit, a first conversion unit, and a second conversion unit, wherein:

[0191] The first traversal unit is used to traverse the element nodes in the first model document tree;

[0192] The first determining unit is used to determine the element nodes and element connections in the first model document tree;

[0193] The first conversion unit is used to convert element nodes in the first model document tree into element nodes in the second model document tree;

[0194] The second conversion unit is used to convert the element connection relationship in the first model document tree into the element connection relationship in the second model document tree.

[0195] In some embodiments, the first conversion unit includes:

[0196] The first reading subunit is used to read the attribute information of the current node number in the first model document tree when the node number is a first preset value;

[0197] The first generation subunit is used to generate the root node of the second model document tree based on the attribute information of the current node number.

[0198] In some embodiments, the first conversion unit includes:

[0199] The second reading subunit is used to read the node type information of the current node number in the first model document tree when the node number is greater than the first preset value;

[0200] The second generation subunit is used to generate the corresponding node of the second model document tree based on the node type information of the current node number.

[0201] In some embodiments, the second conversion unit includes:

[0202] The first determining subunit is used to determine that nodes in the first model document tree whose edge attribute is a second preset value are connecting nodes.

[0203] The second determining subunit is used to determine the element connection relationship of the connecting nodes;

[0204] The third determining subunit is used to determine the parent-child relationship of the nodes in the second model document tree corresponding to the connecting nodes based on the element connection relationship of the connecting nodes.

[0205] In some embodiments, the second conversion unit includes:

[0206] The first adjustment subunit is used to adjust the parent-child relationship of nodes in the second model document tree based on the parent attribute information of element nodes in the first model document tree.

[0207] In some embodiments, the first generation module includes a second generation submodule and a third generation submodule, wherein:

[0208] The second generation submodule is used to parse the XML text file based on the preset unified modeling standard and generate model training information under the specified deep learning framework;

[0209] The third generation submodule is used to train a neural network model under a specified deep learning framework based on the model training information, and generate the target neural network model.

[0210] Specifically, the apparatus for generating and training the above-mentioned visual neural network model provided in this application embodiment can implement all the method steps implemented in the above-mentioned method embodiment for generating and training the visual neural network model, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0211] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a method for generating and training a visual neural network model, the method including:

[0212] Obtain the visualization model file of the neural network model generated by the visualization editor;

[0213] The visualization model file is converted into an XML text file; the XML text file is a model file that conforms to a preset unified modeling standard.

[0214] Based on the XML text file, a target neural network model is generated; the target neural network model is a neural network model that has completed model training under a specified deep learning framework based on a preset unified modeling standard.

[0215] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0216] In some embodiments, converting the visualization model file into an XML text file includes:

[0217] A first model document tree is generated based on the visualized model file; the first model document tree is a visualized model document tree.

[0218] The first model document tree is converted into a second model document tree; the second model document tree is a model document tree that conforms to a preset unified modeling standard;

[0219] The second model document tree text is serialized to obtain the XML text file.

[0220] In some embodiments, converting the first model document tree into a second model document tree includes:

[0221] Traverse the element nodes in the first model document tree;

[0222] Determine the element nodes and element connections in the first model document tree;

[0223] Convert the element nodes in the first model document tree into element nodes in the second model document tree;

[0224] The element connection relationships in the first model document tree are converted into element connection relationships in the second model document tree.

[0225] In some embodiments, converting element nodes in the first model document tree into element nodes in the second model document tree includes:

[0226] When the node number is a first preset value, the attribute information of the current node number in the first model document tree is read, and the root node of the second model document tree is generated based on the attribute information of the current node number.

[0227] In some embodiments, converting element nodes in the first model document tree into element nodes in the second model document tree includes:

[0228] If the node number is greater than a first preset value, read the node type information of the current node number in the first model document tree, and generate the corresponding node of the second model document tree based on the node type information of the current node number.

[0229] In some embodiments, converting the element connection relationships in the first model document tree into element connection relationships in the second model document tree includes:

[0230] Nodes in the first model document tree whose edge attribute is a second preset value are identified as connection nodes;

[0231] Determine the element connection relationships of the connected nodes;

[0232] Based on the element connection relationship of the connected nodes, the parent-child relationship of the nodes in the second model document tree corresponding to the connected nodes is determined.

[0233] In some embodiments, converting the element connection relationships in the first model document tree into element connection relationships in the second model document tree includes:

[0234] Based on the parent attribute information of the element nodes in the first model document tree, the parent-child relationship of the nodes in the second model document tree is adjusted.

[0235] In some embodiments, generating the target neural network model based on the XML text file includes:

[0236] Based on the preset unified modeling standard, the XML text file is parsed to generate model training information under the specified deep learning framework;

[0237] Based on the model training information, the training of a neural network model under the specified deep learning framework is completed, and the target neural network model is generated.

[0238] Specifically, the electronic device provided in this application embodiment can implement all the method steps implemented by the method embodiment with the execution subject being an electronic device, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0239] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the methods for generating and training visual neural network models provided by the above methods, the method comprising:

[0240] Obtain the visualization model file of the neural network model generated by the visualization editor;

[0241] The visualization model file is converted into an XML text file; the XML text file is a model file that conforms to a preset unified modeling standard.

[0242] Based on the XML text file, a target neural network model is generated; the target neural network model is a neural network model that has completed model training under a specified deep learning framework based on a preset unified modeling standard.

[0243] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for generating and training a visual neural network model provided by the methods described above, the method comprising:

[0244] Obtain the visualization model file of the neural network model generated by the visualization editor;

[0245] The visualization model file is converted into an XML text file; the XML text file is a model file that conforms to a preset unified modeling standard.

[0246] Based on the XML text file, a target neural network model is generated; the target neural network model is a neural network model that has completed model training under a specified deep learning framework based on a preset unified modeling standard.

[0247] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0248] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0249] It should also be noted that the terms "first," "second," etc., used in the embodiments of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, and the number of objects is not limited. For example, the first object can be one or more.

[0250] In the embodiments of this application, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0251] In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0252] In this application, "determining B based on A" means that factor A must be considered when determining B. It is not limited to "B can be determined based solely on A," but should also include: "determining B based on A and C," "determining B based on A, C, and E," "determining C based on A, and further determining B based on C," etc. It can also include using A as a condition for determining B, for example, "when A satisfies the first condition, B is determined using the first method"; or "when A satisfies the second condition, B is determined," or "when A satisfies the third condition, B is determined based on the first parameter," etc. Of course, it can also be a condition where A is a factor in determining B, for example, "when A satisfies the first condition, C is determined using the first method, and B is further determined based on C," etc.

[0253] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for visualizing neural network model generation and training, characterized in that, include: Obtain the visualization model file of the neural network model generated by the visualization editor; The visualization model file is converted into an XML text file; the XML text file is a model file that conforms to a preset unified modeling standard. Based on the XML text file, a target neural network model is generated; the target neural network model is a neural network model that has been trained under a specified deep learning framework based on a preset unified modeling standard. The step of converting the visualization model file into an XML text file includes: A first model document tree is generated based on the visualized model file; the first model document tree is a visualized model document tree. The first model document tree is converted into a second model document tree; the second model document tree is a model document tree that conforms to a preset unified modeling standard; The second model document tree text is serialized to obtain the XML text file; The step of converting the first model document tree into a second model document tree includes: Traverse the element nodes in the first model document tree; Determine the element nodes and element connections in the first model document tree; Convert the element nodes in the first model document tree into element nodes in the second model document tree; Convert the element connection relationships in the first model document tree into element connection relationships in the second model document tree; The step of converting element nodes in the first model document tree into element nodes in the second model document tree includes: If the node number is greater than the first preset value, read the node type information of the current node number in the first model document tree; Based on the node type information of the current node number, generate the corresponding node of the second model document tree; The step of converting the element connection relationships in the first model document tree into element connection relationships in the second model document tree includes: Nodes in the first model document tree whose edge attribute is a second preset value are identified as connection nodes; Determine the element connection relationships of the connected nodes; Based on the element connection relationship of the connected nodes, the parent-child relationship of the nodes in the second model document tree corresponding to the connected nodes is determined.

2. The method for generating and training a visual neural network model according to claim 1, characterized in that, The step of converting element nodes in the first model document tree into element nodes in the second model document tree includes: If the node number is a first preset value, read the attribute information of the current node number in the first model document tree; Based on the attribute information of the current node number, the root node of the second model document tree is generated.

3. The method for generating and training a visual neural network model according to claim 1, characterized in that, The step of converting the element connection relationships in the first model document tree into element connection relationships in the second model document tree includes: Based on the parent attribute information of the element nodes in the first model document tree, the parent-child relationship of the nodes in the second model document tree is adjusted.

4. The method for generating and training a visual neural network model according to claim 1, characterized in that, The process of generating the target neural network model based on the XML text file includes: Based on the preset unified modeling standard, the XML text file is parsed to generate model training information under the specified deep learning framework; Based on the model training information, the training of a neural network model under the specified deep learning framework is completed, and the target neural network model is generated.

5. An apparatus for visualizing neural network model generation and training, characterized in that, include: The first acquisition module acquires the visualization model file of the neural network model generated by the visualization editor; The first conversion module is used to convert the visualization model file into an XML text file; the XML text file is a model file that conforms to a preset unified modeling standard. The first generation module is used to generate a target neural network model based on the XML text file; the target neural network model is a neural network model that has been trained under a specified deep learning framework based on a preset unified modeling standard. The first conversion module specifically includes: The second generation unit is used to generate a first model document tree based on the visualization model file; the first model document tree is a visualization model document tree. The second conversion unit is used to convert the first model document tree into a second model document tree; the second model document tree is a model document tree that conforms to a preset unified modeling standard. A serialization unit is used to serialize the second model document tree text to obtain the XML text file; The second conversion unit specifically includes: The traversal unit is used to traverse the element nodes in the first model document tree; Determine connection units, used to determine element nodes and element connections in the first model document tree; The element node conversion unit is used to convert element nodes in the first model document tree into element nodes in the second model document tree; The connection relationship conversion unit is used to convert the element connection relationship in the first model document tree into the element connection relationship in the second model document tree; The transformation element node unit is specifically used for: If the node number is greater than the first preset value, read the node type information of the current node number in the first model document tree; Based on the node type information of the current node number, generate the corresponding node of the second model document tree; The conversion connection relationship unit is specifically used for: Nodes in the first model document tree whose edge attribute is a second preset value are identified as connection nodes; Determine the element connection relationships of the connected nodes; Based on the element connection relationship of the connected nodes, the parent-child relationship of the nodes in the second model document tree corresponding to the connected nodes is determined.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for generating and training a visual neural network model as described in any one of claims 1 to 4.

Images