Data processing method, device, equipment and medium based on deep learning model

By filtering target nodes in the node network and starting the corresponding mirror, the problem of low model format and chip compatibility in the existing technology is solved, and the operation of multiple model formats in heterogeneous environments is realized, and the deep learning model is improved. compatibility and application scope.

CN117519736BActive Publication Date: 2024-12-03YITONG TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311572571.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-22
Publication Date
2024-12-03
Estimated Expiration
2043-11-22

Smart Images

  • Figure CN117519736B_ABST
    Figure CN117519736B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a data processing method and device, equipment and medium based on a deep learning model, which belongs to the field of deep learning technology. The method includes: obtaining a model package, the model package includes a model configuration file and a model file, the model configuration file includes chip information and a model format; based on the chip information and the chip owned by the node, the node is screened to obtain a target node; according to the model format and the model file, a start service command is sent to the target node; through the start service command, the target node is controlled to select a target image from the preset candidate images according to the model format, and the inference service code of the target image is started according to the model file to perform the inference task; wherein, multiple candidate images are run on each node, and a model format is run on each candidate image. The embodiment of the present application achieves the technical effect of supporting heterogeneous hardware platforms and multiple model structures.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a data processing method and apparatus, device and medium based on a deep learning model. Background Art

[0002] In the field of computer vision, deep learning models are widely used to perform reasoning tasks, such as image classification, object detection, instance segmentation, and text recognition.

[0003] When deploying deep learning models, due to high requirements for performance indicators such as speed and throughput, deep learning models are usually deployed on a dedicated inference framework to provide inference services. An inference framework generally refers to a computer program or system used for reasoning and logical inference, which can be used for reasoning, inference, and decision-making processes. On the hardware platform, some inference frameworks usually do not support chips other than NVIDIA GPUs. For example, TensorRT and Triton only support NVIDIA GPUs, and OpenVINO only supports Intel's GPU products. These inference frameworks only support a few model formats, such as TorchServe only supports Pytorch format, TensorflowServing only supports Tensorflow format, and OnnxRuntime only supports ONNX format.

[0004] The disadvantage of existing technologies is that the few model formats supported by a single inference framework can only run on specific devices with specific chips, limiting the application scope of deep learning models. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a data processing method and apparatus, equipment and medium based on a deep learning model, aiming to achieve the technical goal of scheduling multiple models in a heterogeneous environment to improve the compatibility of the reasoning framework with multiple model formats and multiple types of devices.

[0006] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a data processing method based on a deep learning model, the method being applied to a main node among multiple nodes of a node network, the method comprising:

[0007] Obtain a model package, wherein the model package includes a model configuration file and a model file, wherein the model configuration file includes chip information and a model format;

[0008] The nodes are screened based on the chip information and the chips owned by the nodes to obtain a target node;

[0009] Sending a service start command to the target node according to the model format and the model file;

[0010] Through the start service command, the target node is controlled to select a target image from preset candidate images according to the model format, and the inference service code of the target image is started according to the model file to perform the inference task; wherein, multiple candidate images are run on each node, and one model format is run on each candidate image.

[0011] In some embodiments, the step of screening the nodes based on the chip information and the chips owned by the nodes to obtain the target nodes includes:

[0012] Sending a status detection instruction to the node, and the node sending feedback information;

[0013] Obtaining a candidate node according to the sending node identifier in the feedback information;

[0014] Obtain the chip owned by the candidate node to obtain the candidate chip;

[0015] The candidate chip is selected based on the chip information to obtain a target chip, and the candidate node to which the target chip belongs is used as the target node.

[0016] In some embodiments, the method further comprises:

[0017] Receiving an operation instruction sent by a target object; wherein the operation instruction includes a stop service command or an exit service command;

[0018] The stop service command is sent to the target node, and the target node is controlled to stop running the target image through the stop service command; or the exit service command is sent to each of the nodes, and each of the nodes is controlled to stop running the candidate image through the exit service command.

[0019] In some embodiments, the target image is a common image, and the method further comprises constructing the common image:

[0020] Determine an image according to the chip to obtain a basic image;

[0021] Acquire a first target tool software and a first target software package according to the model format, and install the first target tool software and the first target software package into the base image to obtain a first preliminary image;

[0022] Acquire a first target reasoning service code according to the model format, and copy the first target reasoning service code to the first preliminary image to obtain the common image;

[0023] Obtain the name of the common image to obtain the common image name;

[0024] The common image name and the model format are broadcasted to the nodes in the node network.

[0025] In some embodiments, starting the inference service code of the target image according to the model file includes:

[0026] Mounting the model file to a predetermined location of the target image and starting the target model;

[0027] Starting a calling interface, receiving a first reasoning request for a reasoning task sent by the master node through the calling interface, executing the reasoning request through the target model, and obtaining first task reasoning data;

[0028] The first task reasoning data is sent to the master node through the calling interface.

[0029] In some embodiments, the target image is a preset image, and the method further includes constructing the preset image:

[0030] Obtain an image according to the chip to obtain a basic image;

[0031] Acquire a target operating environment and a code dependency library according to a preset code, acquire a second target tool software and a second target software package according to the model format, and install the target operating environment, the code dependency library, the second target tool software and the second target software package into the base image to obtain a second preliminary image;

[0032] The target reasoning service code is obtained according to the model format, and the target reasoning service code is copied to the second preliminary image to obtain the preset image.

[0033] In some embodiments, the model package further includes a preset target code file; the inference service code for starting the target image includes:

[0034] The inference service code for starting the target image includes:

[0035] Execute the preset code in the target code file, mount the model file to a predetermined position of the target image according to the initialization part of the preset code, and start the target model;

[0036] Starting a calling interface, receiving a second reasoning request for a reasoning task sent by the master node through the calling interface, executing the second reasoning request through the target model, and obtaining second task reasoning data;

[0037] The second task reasoning data is sent through the calling interface.

[0038] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a data processing device based on a deep learning model, wherein the device is applied to a main node among multiple nodes of a node network, and the device comprises:

[0039] An acquisition module, used to acquire a model package, wherein the model package includes a model configuration file and a model file, wherein the model configuration file includes chip information and a model format;

[0040] A screening module, used for screening the nodes based on the chip information and the chips owned by the nodes to obtain target nodes;

[0041] A sending module, used for sending a service start command to the target node according to the model format and the model file;

[0042] A startup module is used to control the target node to select a target image from preset candidate images according to the model format through the startup service command, and to start the inference service code of the target image according to the model file to perform the inference task; wherein, multiple candidate images are run on each node, and one model format is run on each candidate image.

[0043] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0044] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0045] The present application proposes a data processing method and apparatus, device and medium based on a deep learning model. In the present application, the model configuration file in the model package includes chip information and model format, and each node in the node network has certain chips and supports certain model formats. Therefore, the target node can be obtained by screening the chip information in the model configuration file and the chips owned by the node. The chip owned by the target node supports the execution of the reasoning task. Since multiple candidate images are run on each node, and a model format is run on each candidate image, the target image can be determined from the multiple candidate images running on the target node based on the model format in the model configuration file. Then, the reasoning service code of the target image is started according to the model file to perform the reasoning task. It can be seen that no matter which chips and / or model formats are required to perform the reasoning task, a suitable target node can be found in the node network to perform the reasoning task, with high versatility and availability. In addition, each node in the node network of the present application has a separate chip and a supported model format, which belongs to a heterogeneous distributed environment, and can run reasoning frameworks of multiple model formats, thereby improving the compatibility of deep learning models with different hardware platforms and reasoning frameworks. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of a data processing method based on a deep learning model provided in an embodiment of the present application;

[0047] Figure 2 yes Figure 1 Flowchart of step 102 in FIG.

[0048] Figure 3 is a flowchart of a data processing method based on a deep learning model provided by another embodiment of the present application;

[0049] Figure 4 is a flowchart of a data processing method based on a deep learning model provided by another embodiment of the present application;

[0050] Figure 5 yes Figure 1 Flowchart of step 104 in;

[0051] Figure 6 is a flowchart of a data processing method based on a deep learning model provided by another embodiment of the present application;

[0052] Figure 7 Another embodiment of the present application provides Figure 1 Flowchart of step 104 in;

[0053] Figure 8 It is a flowchart of an application example provided by an embodiment of the present application;

[0054] Fig. 9 is a flowchart of another application example provided by an embodiment of the present application;

[0055] Fig.10 This is a flowchart of common mirror operation provided by an embodiment of the present application;

[0056] Fig.11 It is a flow chart about the operation of preset mirror provided by an embodiment of the present application;

[0057] Fig.12 is a structural schematic diagram of a data processing device based on a deep learning model provided in an embodiment of the present application;

[0058] Fig.13 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] First, some nouns involved in this application are analyzed:

[0063] Deep learning reasoning framework: After completing model training using a deep learning training framework, you need to use a specialized deep learning reasoning framework when deploying it in a production environment. They usually provide functions such as concurrency, model optimization and acceleration, and provide interfaces such as HTTP or GRPC for calling.

[0064] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0065] Image classification is an image processing method that distinguishes different categories of targets based on their different characteristics reflected in the image information. It uses computers to perform quantitative analysis on images and classify the image or each pixel or area in the image into one of several categories to replace human visual interpretation.

[0066] Object detection: Use a box to mark the location of the object in the image and give the category of the object.

[0067] Instance segmentation is a combination of object detection and semantic segmentation. First, the object is detected in the image (object detection, identifying the content in the image and detecting its location), and then each pixel is labeled (semantic segmentation, each pixel in the image is labeled with a category for classification).

[0068] Text recognition refers to the process in which an electronic device (such as a scanner or digital camera) examines characters printed on paper and then uses character recognition methods to translate the shapes into computer text. That is, the text data is scanned and then the image file is analyzed and processed to obtain text and layout information.

[0069] The data processing method, device, equipment and medium based on the deep learning model provided in the embodiments of the present application are specifically illustrated by the following embodiments. First, the data processing method based on the deep learning model in the embodiments of the present application is described.

[0070] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0071] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0072] The data processing method based on the deep learning model provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data processing method based on the deep learning model, etc., but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0074] Figure 1 is an optional flowchart of a data processing method based on a deep learning model provided in an embodiment of the present application. Figure 1 The method is applied to a master node among multiple nodes in a node network, and the method may include but is not limited to steps 101 to 104.

[0075] Step 101, obtaining a model package, the model package including a model configuration file and a model file, the model configuration file including chip information and a model format;

[0076] Step 102, screening nodes based on chip information and chips owned by the nodes to obtain target nodes;

[0077] Step 103, sending a service start command to the target node according to the model format and the model file;

[0078] Step 104, by starting the service command, controls the target node to select the target image from the preset candidate images according to the model format, and starts the reasoning service code of the target image according to the model file to perform the reasoning task; wherein, multiple candidate images are run on each node, and a model format is run on each candidate image.

[0079] It should be noted that the node in the embodiment of the present application refers to a physical machine. Running an image on a node is equivalent to dividing CPU or memory resources in the node to form a virtual machine.

[0080] It should be noted that the image can be regarded as a container template, which contains the file system structure and its content required to start the container, and is the basis for starting the container. The image is a static perspective of the container, and the container is the running state of the image. The image can be regarded as a structure at the time of construction, and the container can be regarded as a structure at runtime. The running image in the embodiment of the present application is equivalent to the container.

[0081] In step 101 of some embodiments, the model package can be a combination of data such as a program and a text file, a picture file, etc. The files in the model package are used when starting the inference service of the target image. The model configuration file includes chip information and model format. The chip is a GPU. The model format includes but is not limited to Pytorch, Tensorflow, ONNX, Mindspore, and PaddlePaddle.

[0082] In step 102 of some embodiments, based on the chip information and the chips owned by the node, the node is screened to obtain the target node, which may be to search multiple nodes in the node network and select the node whose chip matches the data of the chip information as the target node. For example, a node that is consistent with the chip model, operating frequency, and interface type information in the model configuration file is selected.

[0083] In step 103 of some embodiments, multiple nodes of the node network are already in communication connection. The master node sends a start service command to the target node, so that the target node starts the service process.

[0084] In step 104 of some embodiments, multiple images can be run on the target node at the same time, and different images will not affect each other.

[0085] It should be noted that the data processing method based on a deep learning model provided in the embodiment of the present application can be an image processing method based on a deep learning model. Specifically, for the inference model that implements the object detection function, an inference service command for starting image object detection is sent to the target node according to the model format and the model file; by starting the service command, the target node is controlled to select a target image from the preset candidate images according to the model format, and the inference service code for image object detection of the target image is started according to the model file, so as to perform the object detection task in the image; wherein, multiple candidate images are run on each node, and a model format is run on each candidate image.

[0086] After starting the inference service of the target image according to the model file, you can also verify whether the inference service is started normally. Specifically, for the inference model that implements the object detection function, the method to verify whether the inference service is started normally is to upload a test image and check the inference result of the image. If the object in the test image is correctly identified and labeled, it means that the inference service of the target image is started normally.

[0087] It should be noted that a heterogeneous distributed environment usually refers to a system composed of different types of processors and accelerators such as CPU (Central Processing Unit), GPU (graphics processing unit), FPGA (Field Programmable Gate Array). In terms of software platform support, the current inference framework only supports a few model formats, such as TorchServe only supports Pytorch format, TensorflowServing only supports Tensorflow format, and OnnxRuntime only supports ONNX format. However, deep learning models are updated very quickly, and existing software platforms are difficult to be compatible with enough model formats.

[0088] Steps 101 to 104 shown in the embodiment of the present application screen the target node according to the node configuration file, send a start service command to the target node, and control the target node to select the target image and start the inference service. Each node runs multiple images, and runs a model format on each image. In multiple nodes of the node network, multiple inference models are run and corresponding inference services are started, thereby supporting heterogeneous distributed environments, and can run inference frameworks of multiple model formats on devices with different chip architectures, thereby improving the compatibility of deep learning models with different hardware platforms and inference models.

[0089] See also Figure 2 In some embodiments, step 102 may include, but is not limited to, steps 201 to 204:

[0090] Step 201, sending a status detection instruction to the node, and the node sends feedback information;

[0091] Step 202, obtaining a candidate node according to the sending node identifier in the feedback information;

[0092] Step 203, obtaining the chips owned by the candidate nodes to obtain the candidate chips;

[0093] Step 204 , selecting candidate chips based on chip information to obtain a target chip, and taking the candidate node to which the target chip belongs as the target node.

[0094] In step 201 of some embodiments, the node performs self-checking according to the received status detection instruction. If the self-checking result of the node is abnormal, an error feedback message is sent to the main node. The node with abnormal self-checking is not selected as a candidate node.

[0095] In step 202 of some embodiments, each node corresponds to a unique node identifier, and the feedback information contains the node identifier corresponding to the node that sent the feedback information. According to the sending node identifier in the feedback information, the node that sent the feedback information to the main node is obtained and used as a candidate node.

[0096] In step 203 of some embodiments, after the candidate nodes are determined, the chip of each candidate node is a candidate chip.

[0097] In step 204 of some embodiments, candidate nodes are screened according to chip information in the model configuration file, a target chip that matches the chip information is found, and the candidate node to which the target chip belongs is used as the target node.

[0098] Specifically, each node of a heterogeneous platform has a fixed architecture, such as the CPU is x86 or arm, but the entire platform is composed of nodes with multiple different architectures. The master node records the architecture information of all nodes. When starting the inference service of an inference model, the master node uses the scheduling algorithm to select a suitable idle node based on the architecture information in the model package configuration file and sends a start service command. The node selects a suitable image based on the model format in the model package configuration file, such as pytorch, tensorflow, or onnx, and starts the inference service.

[0099] In steps 201 to 204 shown in the embodiment of the present application, during the screening of candidate chip architectures, the status of the candidate nodes is detected. If the status detection is normal, the candidate nodes are obtained, thereby improving the startup success rate of the target nodes selected from the candidate nodes.

[0100] See also Figure 3In some embodiments, after step 102, the data processing method based on the deep learning model further includes steps 301 to 302:

[0101] Step 301, receiving an operation instruction sent by a target object; wherein the operation instruction includes a stop service command or an exit service command;

[0102] Step 302, a stop service command is sent to the target node, and the target node is controlled to stop running the target image through the stop service command; or an exit service command is sent to each node, and each node is controlled to stop running the candidate image through the exit service command.

[0103] In step 301 of some embodiments, the operation instruction includes a stop service command or an exit service command. The function of the stop service command is to stop running the target image on the target node, thereby shutting down the reasoning service of the image. The function of the exit service command is to stop running the target image running on any node, thereby shutting down the reasoning service of the image.

[0104] In step 302 of some embodiments, upon receiving a stop service command, the target node only stops running the target image, while other images running on the target node are not affected.

[0105] In the step 301 to step 302 shown in the embodiment of the present application, the stop service command stops any reasoning process running on the node according to the specified reasoning model, and does not affect other reasoning processes running on the same node, which increases the flexibility of selecting data processing methods and the diversity of operation. When exiting the program, first exit the target image of the target node, and then exit the main node, to ensure the normal exit of the program and the normal preservation of the reasoning results, and avoid losing control of the target node due to exiting the main node program alone.

[0106] See also Figure 4 , the target image is a common image. After step 104 in some embodiments, the data processing method based on the deep learning model further includes constructing a common image, that is, including steps 401 to 405:

[0107] Step 401, determine the image according to the chip to obtain a basic image;

[0108] Step 402, obtaining a first target tool software and a first target software package according to the model format, and installing the first target tool software and the first target software package into the base image to obtain a first preliminary image;

[0109] Step 403, obtaining a first target reasoning service code according to the model format, and copying the first target reasoning service code to the first preliminary image to obtain a common image;

[0110] Step 404, obtaining the name of the common image, and obtaining the common image name;

[0111] Step 405, broadcast the common image name and model format to the nodes in the node network.

[0112] In step 401 of some embodiments, one method of determining an image according to a chip is to select an image that can run on the chip according to chip information as a basic image.

[0113] It should be noted that the model formats supported by the common image are fixed, such as pth format and onnx format. Users only need to provide model files and configuration files, and do not need to provide preset code files. The model is inferred in a fixed way and cannot be changed.

[0114] In step 402 of some embodiments, a first target tool software and a first target software package are obtained according to a model format, and the first target tool software and the first target software package are installed into a base image to obtain a first preliminary image.

[0115] In step 403 of some embodiments, the first target reasoning service code is a user, and the first target reasoning service code is copied to the first preliminary image to obtain a common image.

[0116] In step 404 of some embodiments, the name of the common image is unique in the node network. In addition to the name, other types of identifiers can also be used as the identifier of the image.

[0117] In step 405 of some embodiments, the common image name and model format are broadcast to the nodes in the node network, so that the master node receives the common image name and model format fed back by the node where the image is successfully deployed.

[0118] Steps 401 to 405 shown in the embodiment of the present application construct corresponding images according to the candidate chip architecture, and different images do not affect each other, so that they can run on a heterogeneous distributed platform and are applicable to a variety of devices and model formats.

[0119] See also Figure 5 In some embodiments, step 104 may include but is not limited to steps 501 to 503:

[0120] Step 501, mount the model file to a predetermined location of the target image, and start the target model;

[0121] Step 502, start the calling interface, receive the first reasoning request for the reasoning task sent by the master node through the calling interface, execute the reasoning request through the target model, and obtain the first task reasoning data;

[0122] Step 503: Send the first task reasoning data to the main node by calling the interface.

[0123] In step 501 of some embodiments, the target model is started after the model file is mounted.

[0124] In step 502 of some embodiments, different images execute different reasoning services and may execute a variety of reasoning requests, which are not limited here.

[0125] In step 503 of some embodiments, the inference data is aggregated to the main node and presented to the user through the main node.

[0126] Steps 501 to 503 shown in the embodiment of the present application can ensure that the model file is not damaged or erroneous by checking the model file, thereby ensuring the accuracy and reliability of subsequent reasoning results. No additional repair or processing of the model file is required after startup, saving time and resources and improving the efficiency of model use.

[0127] See also Figure 6 In some embodiments, the target image is a preset image. After step 104 in some embodiments, the data processing method based on the deep learning model further includes constructing a preset image, that is, including steps 601 to 603:

[0128] Step 601, obtaining an image according to the chip to obtain a basic image;

[0129] Step 602, obtaining a target operating environment and a code dependency library according to a preset code, obtaining a second target tool software and a second target software package according to a model format, and installing the target operating environment, the code dependency library, the second target tool software and the second target software package into the base image to obtain a second preliminary image;

[0130] Step 603 , obtaining the target reasoning service code according to the model format, and copying the target reasoning service code to the second preliminary image to obtain a preset image.

[0131] In step 601 of some embodiments, the model format supported by the preset image is variable, and the user can provide a code file or an executable file. The preset image calls the entry function to start the inference service, and then the custom code can be used for inference, or the user can perform any operation in the image.

[0132] It should be noted that the main difference between a common image and a preset image is whether custom code can be run. The preset image is configured with the operating environment and code dependency library corresponding to the custom code to support running the custom code.

[0133] In step 602 of some embodiments, for different images running on different nodes, the obtained second target tool software and second target software package are also different.

[0134] For example, to build an inference image that supports the onnx format on x86 CPUs and NVIDIA GPUs, first select a basic image that supports the above architecture on the official Docker hub, then install the required tool software on the basis of the image, such as C++ compiler, Python interpreter, ssh or vim, etc., and then install the software packages that the code depends on, such as opencv or numpy.

[0135] In step 603 of some embodiments, the target inference service code is copied to the second preliminary image to obtain a preset image, corresponding to the above-mentioned example of building an inference image supporting the onnx format on the x86 cpu and nvidia gpu, specifically, the inference service code supporting the onnx format is copied to the image, and compiled if necessary. The image is saved and uploaded to the docker hub in the platform, and the name of the new image and the supported model format are notified to all nodes in the node network by broadcasting.

[0136] Specifically, the preset image supports C++ and Python interfaces. Correspondingly, the "operating environment and code dependency library corresponding to the custom code" are the operating environment and code dependency library of C++ and Python.

[0137] In the steps 601 to 603 shown in the embodiment of the present application, the preset image supports the use of custom codes to expand the reasoning model, and the preset code is called to execute the user's extended functions and build complex services. Since the preset image is established by the user, the functions it implements are more suitable for special reasoning application scenarios than universal functions. Triton and other reasoning models also provide Python interfaces to execute user extended functions, but Triton's preset code interface is very complex and difficult to use for algorithm engineers. Compared with the complex preset interface of Triton, the embodiment of the present application uses a simpler preset code interface, and the interface has fewer parameters and options, which helps to reduce user input errors and thus improve the stability and reliability of the code, and improves the convenience of the user to use the interface. In addition, Triton establishes an image, and because it has many functions, the image is large, and the time resources and memory resources spent on deploying the image are high. The mirroring function of the embodiment of the present application is streamlined, the running speed of the reasoning service is improved, and it can be flexibly switched between different images.

[0138] See also Figure 7In some embodiments, the model package also includes a preset target code file. Step 104 of starting the inference service of the target image may include but is not limited to steps 701 to 703:

[0139] Step 701, execute the preset code in the target code file, mount the model file to the predetermined position of the target image according to the initialization part of the preset code, and start the target model;

[0140] Step 702, start the calling interface, receive the second reasoning request for the reasoning task sent by the master node through the calling interface, execute the second reasoning request through the target model, and obtain the second task reasoning data;

[0141] Step 703: Send the second task reasoning data by calling the interface.

[0142] It should be noted that the preset image can run the preset code provided by the user and perform related operations, and is limited to the preset image and will not affect other images or hosts.

[0143] In step 701 of some embodiments, the initialization part of the preset code can be an entry script file. For a preset image, its model package contains not only model files and configuration files, but also code files or executable files. Among them, at least one entry script file is required. The file is located in the root directory of the model package, and its file name needs to be set in the configuration file (such as start.sh). After the image is started, the file name is first checked in the configuration file. If the file exists, the file is executed, and some parameters (such as the model package root directory address, architecture information, etc.) are passed in as command lines. All subsequent operations are controlled by the code in the model package. The so-called correct format means that there is an entry file name in the configuration file and this file exists.

[0144] In step 702 of some embodiments, the manner of executing the second reasoning request varies according to different reasoning models, which is not limited in the embodiments of the present application.

[0145] In step 703 of some embodiments, sending the second reasoning data by calling the interface may be sending the second reasoning data to the master node by calling the interface.

[0146] In the embodiment of the present application, steps 701 to 703 are used to call a preset code interface to execute preset functions, thereby building a complex service. The interface is relatively simple, which improves the convenience of use and makes reasoning more flexible.

[0147] See also Figure 8In one application example, after the master node is started, it detects all nodes in the node network, sends commands to start the nodes, and records the information of the nodes that can be started normally. Then the master node listens to user commands.

[0148] If a command to start a service is detected, the appropriate node is selected according to the model configuration, and a start service command is sent to the corresponding node of the service. If a command to stop a service is detected, a stop service command is sent to the corresponding node of the service. If a command to exit is detected, an exit command is sent to all nodes, and the main node ends its operation.

[0149] See also Fig. 9 In an application example, after the child node is started, it performs a self-check. If the self-check fails, it exits. If it is normal, it listens to the main node command.

[0150] If a command to start the service is heard, the appropriate image is selected according to the model configuration, the model file is mounted and the image is started. If a command to stop the service is heard, the corresponding image is stopped. If a command to exit is heard, all images running on the child node are stopped.

[0151] See also Fig.10 In an application example, for a common image, the target node checks the model file. If an error occurs, the image's reasoning service is exited. If normal, the model is loaded, the model file is mounted to the predetermined location of the image, and the HTTP (Hypertext Transfer Protocol) or gRPC (PRC stands for Remote Procedure Call, gRPC is an open source RPC framework) service is started. The target node listens to the reasoning request sent by the master node, executes the request and returns the result to the master node.

[0152] See also Fig.11 In an application example, for a preset image, the target node checks the model file. If an error occurs, the image's reasoning service is exited. If normal, the custom program file is checked. If the custom program file format is correct, the custom program is executed. The so-called correct format means that there is an entry script file name in the configuration file and the entry script file exists. If the custom program file format is incorrect, the running image is exited.

[0153] See also Fig.12 The embodiment of the present application also provides a data processing device based on a deep learning model, which can implement the above-mentioned data processing method based on a deep learning model and is applied to a main node among multiple nodes of a node network. The device includes:

[0154] An acquisition module 801 is used to acquire a model package, the model package includes a model configuration file and a model file, the model configuration file includes chip information and a model format;

[0155] A screening module 802 is used to screen nodes based on chip information and chips owned by the nodes to obtain target nodes;

[0156] The sending module 803 is used to send a start service command to the target node according to the model format and the model file;

[0157] The startup module 804 is used to control the target node to select a target image from preset candidate images according to the model format through the startup service command, and start the inference service code of the target image according to the model file to perform the inference task; wherein, multiple candidate images are run on each node, and a model format is run on each candidate image.

[0158] In one embodiment, the data processing device based on the deep learning model further includes a first sending module, which is used to:

[0159] Receiving an operation instruction sent by a target object; wherein the operation instruction includes a stop service command or an exit service command;

[0160] A stop service command is sent to the target node, and the target node is controlled to stop running the target image through the stop service command; or an exit service command is sent to each node, and each node is controlled to stop running the candidate image through the exit service command.

[0161] In one embodiment, the data processing device based on the deep learning model also includes a first construction module, which is used to: construct a common image.

[0162] In one embodiment, the data processing device based on the deep learning model also includes a second construction module, which is used to: construct a preset image.

[0163] The specific implementation of the data processing device based on the deep learning model is basically the same as the specific implementation of the data processing method based on the deep learning model mentioned above, and will not be repeated here.

[0164] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned data processing method based on the deep learning model when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0165] Fig.13 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0166] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0167] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the data processing method based on the deep learning model in the embodiment of this application;

[0168] Input / output interface 903, used to implement information input and output;

[0169] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0170] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0171] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0172] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned data processing method based on the deep learning model.

[0173] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0174] The data processing method and apparatus, device and medium based on the deep learning model provided in the embodiments of the present application screen the target node through the node configuration file, send a start service command to the target node, thereby controlling the target node to select the target image and start the inference service. Each node runs multiple images, and a model format is run on each image. In multiple nodes of the node network, multiple inference models are run and corresponding inference services are started, thereby supporting heterogeneous distributed environments, and can run inference frameworks of multiple model formats on devices with different chip architectures, thereby improving the compatibility of deep learning models with different hardware platforms and inference models.

[0175] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0176] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0177] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0179] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0180] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0181] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0182] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0184] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0185] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data processing method based on a deep learning model, characterized in that: The data processing method is applied to a master node among a plurality of nodes in a node network, and the method comprises: Obtain a model package, wherein the model package includes a model configuration file and a model file, wherein the model configuration file includes chip information and a model format; Filtering nodes other than the main node among the multiple nodes based on the chip information and chips owned by the nodes other than the main node among the multiple nodes to obtain a target node; Sending a service start command to the target node according to the model format and the model file; By means of the start service command, the target node is controlled to select a target image from preset candidate images according to the model format, and the inference service code of the target image is started according to the model file to perform the inference task; wherein, a plurality of the candidate images are run on each node, and a model format is run on each of the candidate images; The step of screening nodes other than the main node among the multiple nodes based on the chip information and the chips owned by the nodes other than the main node among the multiple nodes to obtain a target node includes: Sending a status detection instruction to nodes other than the main node among the multiple nodes, and the nodes other than the main node among the multiple nodes sending feedback information; Obtaining a candidate node according to the sending node identifier in the feedback information; Obtain the chip owned by the candidate node to obtain the candidate chip; The candidate chip is selected based on the chip information to obtain a target chip, and the candidate node to which the target chip belongs is used as the target node; When the target image is a common image, the method further includes constructing the common image: Determine an image according to the target chip to obtain a base image; Acquire a first target tool software and a first target software package according to the model format, and install the first target tool software and the first target software package into the base image to obtain a first preliminary image; Acquire a first target reasoning service code according to the model format, and copy the first target reasoning service code to the first preliminary image to obtain the common image; Obtain the name of the common image to obtain the common image name; Broadcasting the common image name and the model format to nodes in the node network; When the target image is a preset image, the method further includes constructing the preset image: Acquire an image according to the target chip to obtain a basic image; Acquire a target operating environment and a code dependency library according to a preset code, acquire a second target tool software and a second target software package according to the model format, and install the target operating environment, the code dependency library, the second target tool software and the second target software package into the base image to obtain a second preliminary image; The target reasoning service code is obtained according to the model format, and the target reasoning service code is copied to the second preliminary image to obtain the preset image.

2. The data processing method based on the deep learning model according to claim 1 is characterized in that: The data processing method further includes: Receiving an operation instruction sent by a target object; wherein the operation instruction includes a stop service command or an exit service command; The stop service command is sent to the target node, and the target node is controlled to stop running the target image through the stop service command; or the exit service command is sent to each node other than the main node among the multiple nodes, and each node other than the main node among the multiple nodes is controlled to stop running the candidate image through the exit service command.

3. The data processing method based on the deep learning model according to claim 1 is characterized in that: The inference service code for starting the target image according to the model file includes: Mounting the model file to a predetermined location of the target image and starting the target model; Starting a calling interface, receiving a first reasoning request for a reasoning task sent by the master node through the calling interface, executing the reasoning request through the target model, and obtaining first task reasoning data; The first task reasoning data is sent to the master node through the calling interface.

4. The data processing method based on the deep learning model according to claim 1 is characterized in that: The model package also includes a preset target code file; The inference service code for starting the target image includes: Execute the preset code in the target code file, mount the model file to a predetermined position of the target image according to the initialization part of the preset code, and start the target model; Starting a calling interface, receiving a second reasoning request for a reasoning task sent by the master node through the calling interface, executing the second reasoning request through the target model, and obtaining second task reasoning data; The second task reasoning data is sent through the calling interface.

5. A data processing device based on a deep learning model, characterized in that: The data processing device is applied to a master node among a plurality of nodes in a node network, and the data processing device comprises: An acquisition module, used for acquiring a model package, wherein the model package includes a model configuration file and a model file, wherein the model configuration file includes chip information and a model format; A screening module, configured to screen nodes other than the main node among the multiple nodes based on the chip information and the chips owned by the nodes other than the main node among the multiple nodes to obtain a target node; A sending module, used for sending a service start command to the target node according to the model format and the model file; A startup module, used to control the target node to select a target image from preset candidate images according to the model format through the startup service command, and start the reasoning service code of the target image according to the model file to perform the reasoning task; wherein, multiple candidate images are run on each node, and one model format is run on each candidate image; The apparatus also includes a module for performing the following method: The step of screening nodes other than the main node among the multiple nodes based on the chip information and the chips owned by the nodes other than the main node among the multiple nodes to obtain a target node includes: Sending a status detection instruction to nodes other than the main node among the multiple nodes, and the nodes other than the main node among the multiple nodes sending feedback information; Obtaining a candidate node according to the sending node identifier in the feedback information; Obtain the chip owned by the candidate node to obtain the candidate chip; The candidate chip is selected based on the chip information to obtain a target chip, and the candidate node to which the target chip belongs is used as the target node; When the target image is a common image, the method further includes constructing the common image: Determine an image according to the target chip to obtain a base image; Acquire a first target tool software and a first target software package according to the model format, and install the first target tool software and the first target software package into the base image to obtain a first preliminary image; Acquire a first target reasoning service code according to the model format, and copy the first target reasoning service code to the first preliminary image to obtain the common image; Obtain the name of the common image to obtain the common image name; Broadcasting the common image name and the model format to nodes in the node network; When the target image is a preset image, the method further includes constructing the preset image: Acquire an image according to the target chip to obtain a basic image; Acquire a target operating environment and a code dependency library according to a preset code, acquire a second target tool software and a second target software package according to the model format, and install the target operating environment, the code dependency library, the second target tool software and the second target software package into the base image to obtain a second preliminary image; The target reasoning service code is obtained according to the model format, and the target reasoning service code is copied to the second preliminary image to obtain the preset image.

6. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the data processing method based on the deep learning model described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data processing method based on a deep learning model described in any one of claims 1 to 4 is implemented.