Intelligent model arrangement method and device, equipment and storage medium

Abstract

The invention relates to an intelligent model arrangement method and device, equipment and a storage medium, and the method comprises the steps: receiving a current component triggering instruction, and determining a current triggered component according to the received current component triggering instruction; determining a corresponding data processing model according to the currently triggered component; rendering the data processing models, and arranging and combining the rendered data processing models according to a received connection instruction; wherein the data processing models corresponding to the components are single processing logics of the data, and the data processing models are obtained by decoupling the neural network model. Compared with a conventional model arrangement method, when each new business appears and the model is developed based on the new business scene, the whole model does not need to be developed, and the single data processing logic required in the business scene is directly reconnected, so that the model arrangement mode is greatly simplified, and the development efficiency is improved. And the model arrangement in the new business scene is more concise and efficient.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to an intelligent model compilation method, device, equipment and storage medium. Background Art

[0002] Commonly used foreign deep learning frameworks include PyTorch and Tensorflow, and domestic ones include PaddlePaddle and MindSpore. Different developers use different frameworks, and the model formats trained by different frameworks are different. When users use the above deep learning frameworks to develop models, they often accumulate a large amount of data and models during the use process. For each new business, a completely new development method needs to be adopted based on the new business scenario, and the developed models cannot be directly reused, which makes the model development process slower and the cost of manpower and time higher. Summary of the invention

[0003] In view of this, the present disclosure proposes an intelligent model orchestration method, which effectively improves the model development process and saves manpower and time costs.

[0004] According to one aspect of the present application, there is provided an intelligent model arrangement method, comprising: Receive a current component trigger instruction, and determine a currently triggered component according to the received current component trigger instruction; Determine a corresponding data processing model according to the currently triggered component; Rendering the data processing model, and arranging and combining the rendered data processing model according to the received connection instruction; Among them, the data processing models corresponding to the components are respectively single processing logics of data, and the data processing models are obtained by decoupling the neural network model.

[0005] In a possible implementation, rendering the data processing model includes rendering the data processing model to a visualization canvas.

[0006] In a possible implementation, the components are divided into two or more modules according to the data processing flow, and different modules correspond to different data processing flows.

[0007] In a possible implementation, the module corresponding to the component includes: at least one of a preprocessing module, a model module and a post-processing module.

[0008] In a possible implementation, the data processing model is obtained by decoupling the neural network model, including: Importing the neural network model currently input, and determining the model framework of the neural network model; When it is determined that the model frame of the neural network model does not belong to the preset frame, frame conversion is performed according to the current model frame of the neural network model so that the model frame of the neural network model is converted to the preset frame; After converting the model framework of the neural network model into the preset framework, acquiring model information of the neural network model, and storing the neural network model based on the acquired model information; At the same time, according to the model information of the network model, the neural network model is decoupled to obtain a single data processing logic contained in the neural network model; The data processing model is generated according to each single data processing logic included in the neural network model.

[0009] In a possible implementation, after obtaining the data processing model by decoupling the neural network model, the method further includes: encapsulating the data processing model into an operation of the component.

[0010] In a possible implementation, when the data processing model is encapsulated as a component, the method further includes: performing statistical classification on the data processing model.

[0011] According to another aspect of the present application, there is also provided an intelligent model arrangement device, comprising: a component triggering module, a model determination module and a model rendering combination module; The component triggering module is configured to receive a current component triggering instruction, and determine a currently triggered component according to the received current component triggering instruction; The model determination module is configured to determine a corresponding data processing model according to the currently triggered component; The model rendering and combining module is configured to render the data processing model and arrange and combine the rendered data processing model according to the received connection instruction; Among them, the data processing models corresponding to the components are respectively single processing logics of data, and the data processing models are obtained by decoupling the neural network model.

[0012] According to another aspect of the present application, there is also provided an intelligent model arrangement device, including: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement any of the above-mentioned methods when executing the executable instructions.

[0013] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, any of the above-mentioned methods is implemented.

[0014] By decoupling the neural network model, each single data processing logic is obtained, and each single data processing logic is encapsulated as a component. Then, when the model is orchestrated, the component corresponding to the required data processing logic is directly triggered, and the components are connected through connection instructions to complete the model orchestration. Compared with the conventional model orchestration method, when each new business appears and the model is developed based on the new business scenario, there is no need to develop the overall model. The single data processing logic required in the business scenario can be directly reconnected. This greatly simplifies the model orchestration method and makes the model orchestration in the new business scenario more concise and efficient.

[0015] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0017] Figure 1 A flowchart showing an embodiment of the intelligent model arrangement method of the present application; Figure 2 A screenshot of the interface for intelligent model arrangement using an embodiment of the present application is shown; Figure 3 A flowchart showing the model import when decoupling the neural network model in the intelligent model arrangement method of the present application is shown; Figure 4 A schematic diagram showing model classification management in an embodiment of the present application is shown; Figure 5 A schematic diagram showing statistical classification of data preprocessing in an embodiment of the present application; Figure 6 A schematic diagram of statistical classification of data post-processing in an embodiment of the present application is shown; Figure 7 A structural block diagram showing an intelligent model arrangement device according to an embodiment of the present application is shown; Figure 8 A structural block diagram of an intelligent model orchestration device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0018] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0019] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0020] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0021] Example 1 Figure 1 FIG. 1 is a flow chart showing a method for arranging an intelligent model according to an embodiment of the present disclosure. Figure 1 As shown, the method includes: step S100, receiving a current component trigger instruction, and determining the currently triggered component according to the received current component trigger instruction. Step S200, determining the corresponding data processing model according to the currently triggered component. Then, through step S300, the data processing model is rendered, and the rendered data processing model is arranged and combined according to the received connection instruction. Here, it should be noted that the data processing models corresponding to the triggered components are respectively single processing logics of the data, and the data processing models are obtained by decoupling the neural network model.

[0022] Therefore, the intelligent model orchestration method of the embodiment of the present application decouples the neural network model to obtain each single data processing logic, and encapsulates each single data processing logic as a component. Then, when performing model orchestration, the component corresponding to the required data processing logic is directly triggered, and the components are connected through connection instructions to complete the model orchestration. Compared with the conventional model orchestration method, when each new business appears and the model is developed based on the new business scenario, there is no need to develop the overall model. The single data processing logic required in the business scenario can be directly reconnected. This greatly simplifies the model orchestration method and makes the model orchestration in the new business scenario more concise and efficient.

[0023] In a possible implementation, when rendering the data processing model, it includes rendering the data processing model to a visualization canvas. Figure 2In the intelligent model arrangement method of the embodiment of the present application, after each single data processing model is encapsulated as a corresponding component, each component is rendered on the side of the visualization canvas in combination with the visualization operation. Therefore, when the user arranges the intelligent model, he only needs to select the required component from the side to trigger the operation of the component to obtain its corresponding data processing logic, and place the selected component on the canvas by dragging and dropping, and then connect the components in the form of lines. After the connection, the final intelligent model application is generated by one-click generation.

[0024] It should be noted that the current component can be triggered in two ways: one is that the user directly triggers the component through selection and clicking operations, and the other is that the component can be triggered through intelligent recommendation.

[0025] Specifically, when triggering components through intelligent recommendation, users can enter a business description or select a business type when entering the model orchestration canvas. After receiving the business description or selected business type entered by the user, the system identifies the business description or business type entered by the user, and determines the component that matches the business description or business type entered by the current user through the pre-trained natural language processing model based on the identification result. For example, when the business type entered by the user is an image recognition task, the corresponding recommended matching components are: data preprocessing (image enhancement component), model reasoning (CNN component), and post-processing (classification result processing component).

[0026] Here, it should be pointed out that when identifying the business description or business type input by the user to determine the matching component based on the identification result, the identification of the business description or business type can be implemented using the conventional intelligent identification network model in the field, which is not elaborated here.

[0027] At the same time, after obtaining the currently required component through any one of the above two component triggering methods, it also includes the operation of parameter configuration of the currently required component. Specifically, in the method of the present application, the corresponding component configuration interface is configured with a "parameter guidance" function, which triggers the parameter guidance instruction, and after receiving the parameter guidance instruction, pushes parameter explanation information for the commonly used parameters of each component according to the business scenario (or business type) input by the user, and pushes the recommended value of each component parameter at the same time. Among them, the recommended value of each component parameter is pre-stored in the database, and different components have corresponding mappings with matching recommended values ​​in different business scenarios. The recommended value can be directly obtained according to the mapping relationship and pushed to the user. The operation is simple and easy to implement.

[0028] After the recommended value of the component parameter of the component is pushed, the information whether to adjust the recommended value is waited for at the same time. After receiving the information of adjusting the recommended value input by the user, the value of the component parameter of the current component is modified according to the recommended value adjustment information input by the user. If the recommended value adjustment information is not received within the preset time, it is assumed that the user determines to directly use the recommended value, so the current component parameter of the current component can be directly configured according to the recommended value.

[0029] Furthermore, after determining the currently required components according to any of the methods described above, step S200 can be executed to determine the corresponding data processing model based on the currently triggered component, and through step S300, the currently selected required data processing model is rendered, and the rendered data processing model is arranged and combined according to the received connection instructions.

[0030] In one possible implementation, while rendering the currently selected data processing model to the canvas, the data processing model rendered to the canvas can be automatically and preliminarily laid out simultaneously, thereby providing a draft diagram for the user. When automatically and preliminarily laying out the data processing model, the component placement work of the user can be simplified based on the best practices of common task flows, so that the user can make a small adjustment on this basis or directly use the preliminary layout, which further reduces the time cost.

[0031] That is to say, in the intelligent model arrangement method of the present application, when rendering the selected data processing model to the canvas, the determination of the connection (ie, connection instruction) between the data processing models can also be achieved in two ways.

[0032] One is that the user directly connects the data processing models manually to achieve the arrangement and combination between the data processing models.

[0033] The other is: according to the currently triggered component, the corresponding data processing models are automatically arranged and connected to form a preliminary layout, and the formed preliminary layout is displayed in the canvas so that the user can judge whether the preliminary layout needs to be adjusted. When adjustment is required, the user manually adjusts the connection arrangement, so as to adaptively adjust the arrangement layout of the data processing model according to the connection instruction received again. When no adjustment is required, the layout automatically generated by the system is directly determined as the final arrangement layout of the data processing model, thereby realizing a one-click generation of intelligent model arrangement, thereby greatly improving the model arrangement efficiency. That is, after determining the components currently needed, the data flow connection is automatically generated based on predefined business logic rules, such as: automatically connecting the data preprocessing module to the model inference model, and then connecting the model inference model to the post-processing model, so as to achieve seamless connectivity of key components.

[0034] In addition, when the user manually adjusts the connection between models or the user directly connects the models, it also includes pushing connection suggestions based on the currently connected model and prompting the connection results, so that the user can further determine the current connection based on the pushed suggestions and connection result prompts.

[0035] That is, when the user adjusts a component parameter or connection, it also includes an instant feedback operation. That is, obtain the connection or component parameters currently adjusted by the user, determine the impact on the overall effect of the intelligent model based on the currently adjusted connection or component parameters, and issue corresponding prompts based on the determined impact results. For example: when it is determined that the efficiency of the intelligent model will be affected based on the currently adjusted connection or component parameters, a prompt of "preprocessing time is too long" or "model reasoning time is slow" will be issued, thereby helping users to complete the optimization of the model synchronously during the process of model arrangement. In other words, different connection or component parameters actually correspond to different data flows and processing. During the adjustment process, the software will promptly perform pre-calculations in the background and implement the display of calculation effects. Users can get different results based on different connection or component parameters, and then judge the impact on the overall effect of the intelligent model.

[0036] Furthermore, after completing the arrangement and combination of the data processing models through any of the above methods, the step of verifying the arrangement and layout of the connected models is also included. That is, the connection between the data processing models is checked (specifically, the input and output data dimension size and data type corresponding to the current connection are checked). When the unmatched data flow connection is checked, it is automatically marked in red and a prompt for modification is issued, thereby ensuring the accuracy of the final intelligent model arrangement and layout result.

[0037] Among them, it should be pointed out that when checking the connection between each data processing model, it can be achieved through the following steps. That is, obtain the connection between each data processing model, and judge whether the logic of the connection is accurate based on the type of data processing model connected to the two ends of the connection and the data processing logic executed. When it is judged that the data processing logic between the data processing models at both ends of the connection is accurate, the connection is determined to be a matching accurate connection. When it is judged that the data processing logic between the data processing models at both ends of the connection is inaccurate, the connection is determined to be unmatched. At this time, the connection is marked in red and the user is prompted to check the connection of the connection. Through the operation of this verification and detection, the situation of incorrect connection caused by complex logic is avoided.

[0038] In one possible implementation, after the arrangement and combination of the data processing models is completed, the arranged model can be verified again by real-time simulation (i.e., preview). That is, the intelligent model after the current arrangement and combination is obtained, and the pre-built input data is input into the intelligent model obtained by the arrangement and combination, and the intelligent model processes the input data according to the current combination form and obtains the final processing result. In this process, the data flow of the intelligent model during data processing is obtained in real time, and the data flow is displayed in the simulation interface, so that the user can directly view the results of the data inflow and data outflow of each data processing model in the intelligent model from the canvas, which more intuitively realizes the further verification of the intelligent model obtained by the arrangement and combination.

[0039] Here, it should be pointed out that the input data for the simulation operation of the intelligent model is pre-stored. Among them, for the input data of different intelligent models, corresponding selection can be made according to the business scenario (or business type) corresponding to the intelligent model. That is to say, in the technical solution of the present application, a mapping relationship between input data and business type is pre-stored, and the mapping relationship shows the correspondence between different business types and different input data. Therefore, when starting the intelligent model simulation operation verification process, appropriate input data can be selected according to the business type corresponding to the intelligent model to be simulated, and the input data can be used as the initial input of the data flow of the intelligent model to perform simulation operation verification on the intelligent model.

[0040] At the same time, when verifying the orchestrated and combined intelligent model through simulation operation, there are multiple verification methods, that is, diversified verification modes are set. Therefore, when simulating and verifying the intelligent model application, you can first obtain the currently selected verification mode (the verification mode can set different scenarios according to different business types, and configure corresponding verification input data for different scenarios), and then obtain the corresponding simulation data stream according to the obtained verification mode, such as: noise data, abnormal data, etc., and then verify the performance of the intelligent model application in different scenarios based on the obtained simulation data stream.

[0041] Furthermore, after the intelligent model application is finally generated by any of the above methods, an operation of generating application code with one click may also be included. That is, by triggering the one-click generation instruction, the logic, configuration parameters, data flow, optimization rules, etc. of each component are automatically summarized to generate the final intelligent model application code.

[0042] Furthermore, in the embodiment of the present application, the component can be divided into two or more modules according to the data processing flow, and different modules correspond to different data processing flows. Specifically, the module corresponding to the component can include at least one of a preprocessing module, a model module, and a post-processing module.

[0043] That is, the components are divided into three modules: preprocessing module, model module and post-processing module. Different modules correspond to different data processing flows. The components contained in the preprocessing module are all operation logics for preprocessing the data (such as grayscale processing, scale adjustment, normalization processing, etc. of image data), and the components contained in the model module all contain operation logics for reprocessing the data after preprocessing (such as feature extraction, feature recognition, etc.), and the components contained in the post-processing module are operation logics for post-processing the results after data processing by the model module (such as non-maximum suppression processing, border regression processing, threshold processing, etc. for image type data). By dividing the components into different modules, it is more conducive to model compilers to obtain the required components more intuitively and quickly when using the model compilation method of the present application for model compilation, which further improves the efficiency of model compilation.

[0044] In a possible implementation, when a data processing model is obtained by decoupling a neural network model, it can be implemented in the following manner.

[0045] First, the currently input neural network model is imported, and the model framework of the neural network model is judged. When it is judged that the model framework of the neural network model does not belong to the preset framework, the framework conversion is performed according to the current model framework of the neural network model, so that the model framework of the neural network model is converted to the preset framework. After the model framework of the neural network model is converted to the preset framework, the model information of the neural network model is obtained, and the neural network model is stored based on the obtained model information. At the same time, according to the model information of the network model, the neural network model is decoupled to obtain a single data processing logic contained in the neural network model. A data processing model is generated according to each single data processing logic contained in the neural network model.

[0046] That is, see Figure 3 As shown, when the model is decoupled to obtain each data processing model, the model framework of the imported neural network model can be judged first. In an embodiment of the present application, the preset framework is the Paddle framework. When it is determined that the framework of the imported neural network model does not belong to the Paddle framework, the neural network model is converted to the Paddle framework before proceeding to the next step.

[0047] Among them, in order to effectively determine the model file types of different deep learning frameworks, this application designs a new model file type identification process based on file structure, binary features and custom parsers to avoid direct reliance on existing technologies and libraries. This method covers steps such as file extension, feature byte detection of model files, and custom binary parsing strategies, so that the process can more effectively and accurately identify model frameworks.

[0048] Specifically, when the model file type recognition technology of this application is used for recognition, first, the file extension and format of the model are preliminarily detected. This is because different frameworks use different file formats to save models. Preliminary screening can be performed through file extensions, but it should be noted that some common extensions may be reused in multiple frameworks, such as .ckpt in TensorFlow and MindSpore. Therefore, the extension judgment is only for reference and needs further verification.

[0049] Specifically, when performing a preliminary check on the file extension and format of the model, you can use Python to read the file path extension and match it with common model file formats. Compare different formats with possible frameworks to establish a preliminary framework guess.

[0050] Then, the file characteristic byte pattern of the model is identified. Since each model format has some unique byte features in the file header and specific offset positions, it can be used to confirm its framework. For example, TensorFlow's .pb file uses Google's Protocol Buffers, and the header usually contains specific magic bytes. By analyzing the binary pattern of the model's file characteristic bytes, further verification of the framework to which the file belongs is achieved.

[0051] When identifying the file feature byte pattern of the model, first, binary reading: read the first 256 bytes (or key byte positions) of the file in binary mode through Python's file I / O module. Then, pattern matching: design specific byte matching rules to capture the file header features of common formats such as TensorFlow and PyTorch.

[0052] Next, the custom model file is parsed. That is, after finding the pattern in the file header and the specific offset position, the custom parser is further used to parse the file structure information, such as determining the hierarchy of the graph definition or the way the weight parameters are stored. Each framework has a specific data sequence pattern, and parsers for different frameworks can be designed to determine the file content characteristics. Specifically, first, the file is deconstructed and read: the header and keyword segments are parsed, and key information is extracted based on the rules inferred from the known format. Then, the data structure is parsed: for example, the GraphDef file of TensorFlow is mainly serialized using protobuf, and its layers are deconstructed by a custom Protobuf parser in this application. Then, a conditional judgment is performed: that is, the framework type is determined based on the content of the structure obtained by parsing the layer.

[0053] After identifying the model framework type in the above way, the identified model framework type can also be verified by model loading. That is, without using a specific library, you can try to load the model file content based on native Python to confirm whether certain key information can be read correctly. Although model inference cannot be performed, the correctness of the data block structure can be detected to further confirm the model framework. Specifically, first, customize the reading: read specific byte segments of the file, analyze parameters, hierarchical structures, etc. Then, perform exception capture: design logic to ensure that once an unexpected structure is detected (such as a TensorFlow file mistakenly treated as a PyTorch file), the program can capture and mark this exception.

[0054] Therefore, by combining any one or more of the above methods, the multi-layer judgment logic of extension name, file signature, content parsing and loading verification is used to ensure accurate identification of the model framework.

[0055] After identifying the model framework, you can judge the model framework to determine whether the currently imported model belongs to the Paddle framework. Specifically, when judging the model framework, you can use the following method.

[0056] Since the formats of models trained by different frameworks are different, but each framework has its own command rules, we can directly distinguish them by the format of the model.

[0057] Among them, it is well known that the models trained by the PyTorch framework are as follows: pt or .pth: This is the commonly used model saving format in the PyTorch framework. Use torch.save(model.state_dict(), 'model_weights.pth') to save the model's weights, while torch.save(model, 'model.pth') can save the entire model. The .pt and .pth formats are functionally equivalent, and the main difference lies in the suffix.

[0058] ‌.pkl‌: Although the .pkl format is not common in PyTorch, it can also be used to save models. Using the pickle library, you can save the model as a .pkl file, which is functionally equivalent to the .pt and .pth formats.

[0059] ‌TorchScript‌: This is a feature in PyTorch that allows models to be converted into an intermediate representation (IR) that can be optimized and run independently of Python. TorchScript aims to improve the deployment performance, portability, and scalability of models, especially in production environments.

[0060] ‌ONNX format: Although not a native format for PyTorch, ONNX (Open Neural Network Exchange) is an open standard format for representing neural network models, allowing models to be exchanged between different deep learning frameworks. PyTorch can export models to ONNX format through the torch.onnx.export() function.

[0061] The models trained by the TensorFlow framework are as follows: Checkpoint (*.ckpt): This format is used to save the model weights, mainly for parameter backup and model training hot start during model training. Checkpoint files do not contain graph structures, only serialized variable objects, so the network structure needs to be redefined when restoring the model.

[0062] GraphDef (*.pb): This format contains the serialized data of the protobuf object, including the computational graph, operators, tensors, and variable definitions, but does not contain the values ​​of the variables. Files in the GraphDef format can be used to build computational graphs, but the trained weights still need to be restored from Checkpoint.

[0063] ‌SavedModel‌: This is the model format recommended by TensorFlow, which contains the model graph and weights and can be used directly online. The SavedModel format is the core part of TensorFlow 2.x version and is suitable for saving and loading TensorFlow and Keras models.

[0064] ‌FrozenGraph‌: Checkpoint and GraphDef are integrated and optimized through the freeze_graph.py tool to generate the FrozenGraphDef format. This format does not contain the Variables node, converts all variables into constants, and is suitable for direct deployment on mobile devices.

[0065] TFLite: This format is used for device-side inference and optimizes the model based on flatbuf. It is suitable for mobile devices such as Android and iOS. The TFLite format is slightly different from FrozenGraph and is mainly used for deployment on mobile devices.

[0066] The model formats of the Paddle framework are as follows: .pdparams: This is the training model file of PaddlePaddle, which contains all the parameter information obtained by training. By merging the training model file and the model structure file, a complete prediction model file can be obtained.

[0067] .infer.model: This is the prediction model file of PaddlePaddle, which contains the complete model structure and parameter information. Use the Paddle Inference library to load the prediction model file to directly perform model inference.

[0068] The model formats of the MindSpore framework are as follows: Checkpoint: It uses the Protocol Buffers format to store all parameter values ​​in the network. It is mainly used to resume training after a training task is interrupted, or for fine-tuning tasks after training.

[0069] MindIR‌: MindSpore's intermediate representation format, a functional IR based on graph representation, eliminates model differences between different backends and supports reasoning tasks across hardware platforms.

[0070] ‌AIR‌: Ascend Intermediate Representation, is an open file format designed for machine learning defined by Huawei, suitable for inference tasks on Huawei AI processors such as Ascend 310.

[0071] From the above, we can know that each framework has its own model format, so the model training framework can be identified through the preliminary distinction of the format.

[0072] When it is determined that the imported model is not a Paddle framework, the currently imported model needs to be converted to a Paddle framework. In a possible implementation, when converting the model to a Paddle framework, it can be based on the following principles.

[0073] 1. Intermediate representation layer: Convert models of different frameworks into intermediate representation layers (such as ONNX format). This intermediate format is universal and can express the model structure and weights of different frameworks. By converting the model to the intermediate layer first and then parsing it into Paddle format, the compatibility and accuracy of the conversion are improved.

[0074] 2. Format parsing and conversion: Design a custom parser for each input format (such as TensorFlow, ONNX, etc.) to convert its key features (model structure, parameter weights, etc.) into a format that Paddle can understand. The parsing logic for each format will run in a multi-threaded environment to speed up the conversion.

[0075] 3. Model information extraction and synchronization: After conversion, various types of model information (such as name, purpose, etc.) are parsed and acquired synchronously using multi-threading to improve extraction efficiency.

[0076] Based on the above principles, when converting the model to the Paddle framework, first determine the intermediate representation layer (IR) of the model, and convert models such as TensorFlow, Caffe, and PyTorch into intermediate representations such as ONNX. For models that are already in ONNX format, they can be directly parsed and converted, while models of other frameworks will be converted to ONNX through the following steps. Specifically, ①TensorFlow to ONNX: Parse TensorFlow's .pb file into ONNX format, and implement it through custom graph structure conversion logic. ②PyTorch to ONNX: Use the .pt and .pth files saved by PyTorch to convert its parameter hierarchy into an ONNX graph. ③Caffe to ONNX: Read Caffe's .prototxt file structure and .caffemodel weights, and convert them to ONNX through a hierarchical mapping table.

[0077] More specific: # Custom TensorFlow to ONNX conversion custom_tf_to_onnx --input_model=tf_model.pb --output_model=converted_model.onnx # Custom PyTorch to ONNX conversion custom_pytorch_to_onnx --input_model=pt_model.pth --output_model=converted_model.onnx # Custom Caffe to ONNX conversion custom_caffe_to_onnx --prototxt=deploy.prototxt --weights=deploy.caffemodel --output_model=converted_model.onnx Then, parse the ONNX intermediate representation and convert it to Paddle format. That is, after converting to ONNX format, it is further parsed and converted to the format of the Paddle framework. This involves node and operator mapping between ONNX and Paddle formats. The specific steps are as follows: ① Read ONNX graph: parse the hierarchy, operators, and parameters in the ONNX file to build a model graph compatible with the Paddle format.

[0078] ②ONNX to Paddle mapping table: Create a layer mapping table between ONNX and Paddle, and map the operators and parameters in ONNX to the corresponding layers of Paddle.

[0079] ③ Parameter weight conversion: For the weight parameters of each layer, convert the floating point format of ONNX into the format supported by Paddle.

[0080] # Convert ONNX to Paddle convert_onnx_to_paddle --input_model=converted_model.onnx --save_dir=pd_model Next, the model information is extracted and stored. That is, after the model is converted to Paddle format, the relevant information of the model (such as name, purpose, creation time, version and category, etc.) is parsed and obtained synchronously using multi-threading.

[0081] The specific steps are: ① Basic information extraction: Extract basic information such as name, category, creation time, model version, etc. from the model structure.

[0082] ② Multi-threaded parallel extraction: Create independent threads for different information extraction tasks to achieve synchronous processing.

[0083] ③Information storage: Save the extracted model information to the database or configuration file for subsequent management and query.

[0084] Finally, the final command summary is generated. That is, based on the above designed steps, these operations can be integrated into a terminal command line tool to support the conversion of different model formats and information extraction.

[0085] # TensorFlow ->ONNX ->Paddle custom_tf_to_paddle --input_model=tf_model.pb --output_dir=pd_model # PyTorch ->ONNX ->Paddle custom_pytorch_to_paddle --input_model=pt_model.pth --output_dir=pd_model # Caffe ->ONNX ->Paddle custom_caffe_to_paddle --prototxt=deploy.prototxt --weights=deploy.caffemodel --output_dir=pd_model Therefore, in the intelligent model arrangement and combination method of the present application, when converting the model, the purpose of converting different model formats into a unified ONNX representation is achieved through the design of the intermediate representation layer, and the conversion accuracy across frameworks is improved. At the same time, a parallel extraction method can also be used to improve the efficiency of model information acquisition, which is particularly suitable for large-scale model management. In addition, since it integrates custom commands and tools, the model conversion process is independently implemented, independent of existing conversion tools, demonstrating the practicality and flexibility of custom tools. This process not only supports cross-framework model conversion, but also supports efficient model information extraction and management, providing an efficient and universal solution for cross-platform applications of deep learning models.

[0086] For example, X2Paddle is a model conversion tool under the Paddle ecosystem, which supports converting other deep learning framework models into Paddle framework models.

[0087] TensorFlow model conversion uses the following command: x2paddle --framework=tensorflow --model=tf_model.pb --save_dir=pd_model ONNX model conversion uses the following command: x2paddle --framework=onnx --model=onnx_model.onnx --save_dir=pd_model Caffe model conversion uses the following command: x2paddle --framework=caffe --prototxt=deploy.prototxt --weight=deploy.caffemodel -- The PyTorch and MindSpore framework models are first converted into ONNX models, and then converted into Paddle framework models.

[0088] After the model framework of the imported neural network model is set to the preset framework in the above manner, the model information of the imported neural network model can be obtained. Here, it should be noted that the obtained model information includes: model name, model purpose, creation time, model version and model category, and at least one of other information. And the above model information can be obtained synchronously in a multi-threaded manner, or one by one in a single-threaded manner (such as Figure 3 as shown).

[0089] Here, it is necessary to explain in detail that when obtaining the model information, the model information can be obtained through human-computer interaction. Specifically, it includes: 1. Start the interactive interface Create a user-friendly interactive interface to facilitate users to supplement model information. You can use command line interaction, graphical interface (such as Tkinter, Qt), web form (such as Flask + HTML) or API interface, and flexibly choose according to the user's usage scenario.

[0090] 2. Automatically detect basic information In the interactive interface, the program automatically displays basic information of the model, such as the model name and file path, and displays this information to the user. The user can modify the model name or select the file path.

[0091] Prompt users to confirm automatically extracted information, such as model file size, number of parameters, number of network layers, etc., to help users understand the structure of the model more clearly.

[0092] 3. Human-computer interaction input model meta-information Model purpose: Prompts the user to enter the specific purpose of the model, such as "image classification", "object detection", "natural language processing", etc.

[0093] Model version: Users can specify the model version (such as v1.0, v2.1, etc.) in the interface to track it in version management.

[0094] Creation Time: Provides automatic detection of the creation time of the model file, but allows users to enter or modify it manually. If the actual creation time of the model is different from the file save time, the user can enter the correct time.

[0095] Model Category: Provide users with an option menu or selection box to select the model category from preset categories (such as "Classification Model", "Generative Model", etc.), or manually enter a custom category.

[0096] 4. Input confirmation and feedback After the user completes entering the information, all the information entered by the user will be displayed, and the user will be prompted to confirm and submit.

[0097] If the information entered by the user is incomplete or not in the expected format, provide timely feedback and allow the user to modify it.

[0098] 5. Multithreaded or asynchronous acquisition optimization Provide multi-threaded or asynchronous functions so that the system can simultaneously execute automatically extracted steps (such as file attributes, number of parameters, etc.) and steps manually entered by the user (such as purpose, version, etc.).

[0099] Display the information being obtained in the interface and let users see real-time feedback from the system to optimize the user experience.

[0100] 6. Storage and Recording Store all model information (including user input and automatically detected information) in a configuration file (such as JSON or YAML) for easy subsequent management. Save records to a database or log to support future queries and retrieval.

[0101] After obtaining these model information, the neural network model can be decoupled to obtain the single data processing logic contained in the neural network model, and a data processing model can be generated according to the single data processing logic contained in the neural network model.

[0102] In a possible implementation, decoupling the neural network model can be achieved in the following manner.

[0103] That is, in the process of decoupling the neural network model, the model can be divided from the overall structure into reusable, independent functional components, each of which contains independent data processing logic. Through this modular processing, flexible component combination is achieved to efficiently build a complex intelligent model system. The following are the detailed steps and principles for decoupling the model in this application: 1. Determine the decoupling target. The decoupling target can be determined based on the principle of model disassembly: that is, when decoupling the neural network model, divide the model into independent functional units, such as data preprocessing, feature extraction, classifier and other modules. Each module performs a specific data processing task, and different tasks are independent of each other. By decomposing the model into functional modules at different levels, the reusability and combination flexibility of the model can be achieved, so that it can be called on demand in different application scenarios.

[0104] Then, automatically detect and separate the sub-modules of the model. Specifically, automatically detect the logical hierarchy: automatically identify the different logical levels of the model, such as the input preprocessing layer, feature extraction layer, classification layer, etc., through model parsing tools (such as the layered API that comes with the model building framework). Separate sub-modules: Based on the detected hierarchy, extract each sub-module separately to generate independent sub-model files. Each sub-model file only contains specific processing logic, which is convenient for calling in different scenarios.

[0105] Next, reconstruct and encapsulate the submodules. Specifically, encapsulate the data processing logic: encapsulate each extracted submodule to make it an independent component. Each component contains specific data processing logic, and the corresponding data processing function can be implemented by calling the component. By encapsulating each submodule, the submodule has good reusability and isolation, reduces the coupling of the model, and realizes a more flexible combination of data processing logic.

[0106] In addition, after decoupling the neural network model to obtain the data processing model, it also includes: encapsulating the data processing model into components. That is, the received data processing model is componentized and an API interface is generated. Among them, componentization processing: componentization of each submodule, that is, designing an independent API interface for each submodule to facilitate external system calls. Generate API interface: encapsulate the API interface for each component to support the RESTful calling method. In this way, users can call these API interfaces to realize on-demand calls of various data processing modules. Through APIization, each submodule of the model can be called independently, which is convenient for expansion and integration, and can realize dynamic combination and configuration.

[0107] At the same time, when the data processing model is encapsulated as a component, it also includes: an operation of statistically classifying the data processing model.

[0108] That is, see Figure 4 , Figure 5 and Figure 6 As shown, for the components included in the data preprocessing module, when performing statistical classification, they can be classified according to the type of preprocessed data. Specifically, the preprocessing components include components associated with image preprocessing, components associated with text preprocessing, and components associated with speech preprocessing. Among them, the data oriented by the components associated with image preprocessing are all data in image format, the data oriented by the components associated with text preprocessing are all data in text format, and the data oriented by the components associated with speech preprocessing are all data in speech format.

[0109] Model classification management includes components associated with the type of data processed, components associated with the type of technology used, and components associated with the type of business application.

[0110] Data post-processing is similar to the statistical classification of data pre-processing, and also includes components associated with image post-processing, components associated with text post-processing, and components associated with speech post-processing. We will not go into details here.

[0111] That is, classify and manage data processing models. The classification basis is: classify each data processing model according to data type, technology type and application scenario, so that the system can manage and schedule components. Specifically, classification by data type: image data, text data and voice data correspond to image preprocessing components, text preprocessing components and voice preprocessing components respectively. Classification by technology type: classification according to the algorithm technology used, such as CNN, RNN, Transformer, etc. Classification by application scenario: such as classification tasks, regression tasks, generation tasks, etc., to facilitate the selection of appropriate modules for specific tasks. By classifying data processing models in detail, efficient management of model components is achieved, which facilitates the system to automatically schedule and call components.

[0112] Finally, the data post-processing component design is performed. Among them, post-processing componentization: the decoupled data processing logic is further encapsulated according to the post-processing requirements, such as annotation after image processing, conversion after speech recognition, etc. Component classification and management: The classification of data post-processing components is similar to that of data pre-processing components, and is classified according to data type, technology type, and application scenario. Through componentized design, the full process of data processing is automated, and through componentized encapsulation of pre-processing-data processing-post-processing, automatic model orchestration under complex business logic is supported.

[0113] It should be noted that although Figures 1 to 6 The above-mentioned intelligent model arrangement method is introduced as an example, but those skilled in the art will understand that the present disclosure should not be limited thereto. In fact, the user can flexibly set the specific implementation method of each step according to personal preferences and / or actual application scenarios, as long as the technical concept of intelligent model arrangement can be realized by calling the corresponding data processing model through the trigger component of the present application.

[0114] Correspondingly, based on any of the above-mentioned intelligent model arrangement methods, the present application also provides an intelligent model arrangement device. Figure 7 The intelligent model arrangement device 100 provided in the present application includes: a component triggering module 110, a model determination module 120 and a model rendering combination module 130.

[0115] The component trigger module 110 is configured to receive the current component trigger instruction, and determine the currently triggered component according to the received current component trigger instruction. The model determination module 120 is configured to determine the corresponding data processing model according to the currently triggered component. The model rendering combination module 130 is configured to render the data processing model, and arrange and combine the rendered data processing model according to the received connection instruction. The data processing models corresponding to the components are respectively single processing logics of the data, and the data processing models are obtained by decoupling the neural network model.

[0116] Furthermore, according to another aspect of the present disclosure, an intelligent model arrangement device 200 is also provided. Figure 8 The intelligent model arrangement device 200 of the embodiment of the present disclosure includes a processor 210 and a memory 220 for storing executable instructions of the processor 210. The processor 210 is configured to implement any of the above-mentioned intelligent model arrangement methods when executing the executable instructions.

[0117] Here, it should be noted that the number of processors 210 may be one or more. At the same time, in the intelligent model arrangement device 200 of the embodiment of the present disclosure, an input device 230 and an output device 240 may also be included. Among them, the processor 210, the memory 220, the input device 230 and the output device 240 may be connected through a bus or in other ways, which are not specifically limited here.

[0118] The memory 220 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and various modules, such as the program or module corresponding to the intelligent model arrangement method of the embodiment of the present disclosure. The processor 210 executes various functional applications and data processing of the intelligent model arrangement device 200 by running the software programs or modules stored in the memory 220.

[0119] The input device 230 may be used to receive input numbers or signals. The signals may be key signals related to user settings and function control of the device / terminal / server. The output device 240 may include display devices such as display screens.

[0120] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by the processor 210, any of the above-mentioned intelligent model compilation methods is implemented.

[0121] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An intelligent model arrangement method, characterized in that: include: Receive a current component trigger instruction, and determine a currently triggered component according to the received current component trigger instruction; Determine a corresponding data processing model according to the currently triggered component; Rendering the data processing model, and arranging and combining the rendered data processing model according to the received connection instruction; Among them, the data processing models corresponding to the components are respectively single processing logics of data, and the data processing models are obtained by decoupling the neural network model.

2. The method according to claim 1, characterized in that Rendering the data processing model includes rendering the data processing model to a visualization canvas.

3. The method according to claim 1, characterized in that The components are divided into two or more modules according to the data processing flow, and different modules correspond to different data processing flows.

4. The method according to claim 3, characterized in that The modules corresponding to the component include: at least one of a pre-processing module, a model module and a post-processing module.

5. The method according to any one of claims 1 to 4, characterized in that: The data processing model is obtained by decoupling the neural network model, including: Importing the neural network model currently input, and determining the model framework of the neural network model; When it is determined that the model frame of the neural network model does not belong to the preset frame, performing frame conversion according to the current model frame of the neural network model so that the model frame of the neural network model is converted to the preset frame; After converting the model framework of the neural network model into the preset framework, acquiring model information of the neural network model, and storing the neural network model based on the acquired model information; At the same time, according to the model information of the network model, the neural network model is decoupled to obtain a single data processing logic contained in the neural network model; The data processing model is generated according to each single data processing logic included in the neural network model.

6. The method according to any one of claims 1 to 4, characterized in that: After obtaining the data processing model by decoupling the neural network model, the method further includes: encapsulating the data processing model into an operation of the component.

7. The method according to claim 6, characterized in that When the data processing model is encapsulated as a component, it also includes: an operation of statistically classifying the data processing model.

8. An intelligent model arrangement device, characterized in that: include: Component triggering module, model determination module and model rendering combination module; The component triggering module is configured to receive a current component triggering instruction, and determine a currently triggered component according to the received current component triggering instruction; The model determination module is configured to determine a corresponding data processing model according to the currently triggered component; The model rendering and combining module is configured to render the data processing model and to arrange and combine the rendered data processing model according to the received connection instruction; Among them, the data processing models corresponding to the components are respectively single processing logics of data, and the data processing models are obtained by decoupling the neural network model.

9. An intelligent model arrangement device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 7 when executing the executable instructions.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images