A model edge deployment method based on multi-task fusion

Abstract

The present invention discloses a model edge deployment method based on multi-task fusion, which collects image samples in a multi-task scenario and establishes a training sample library; uses a lightweight network to train the mixed sample library to obtain a one-stage mixed target detection model file; according to the characteristics of the detection target in the multi-task scenario, each image sample that needs to be processed in the second stage is labeled in the second stage, and a lightweight network under the specific task is designed and trained to obtain a number of two-stage lightweight processing model files; takes the image to be detected as input to obtain the classification information and position information of the rectangular key area of ​​the multi-task target; takes the information of each rectangular key area as input to obtain the result output in the multi-task mixed application scenario. The method of the present invention integrates the mixed detection scenario of multiple tasks into a set of application processes, establishes an end-to-end task implementation method, can improve the execution efficiency of multiple tasks in the edge terminal, and effectively improves the user experience.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a model edge deployment method based on multi-task fusion. Background Art

[0002] In the traditional way, when deploying and applying deep learning models in edge terminals to implement various AI tasks, due to the scheduling of hardware resources, various tasks are generally executed in a serial synchronous manner, that is, after a task is completed and the result is obtained, the next task is executed. This leads to problems in execution efficiency in complex multi-task scenarios. First, each task is run in series, which will cause a single task cycle to take up a lot of time. Due to the existence of some redundant detection processes, it also brings more time-consuming occupation; secondly, each task needs to load at least one model file. In the multi-task usage scenario, a lot of model files are required to detect targets, resulting in the occupation of memory and storage resources, which poses a greater challenge to the already scarce hardware resources on the edge terminal. Therefore, the traditional multi-task serial execution method in the edge terminal will cause a decrease in reasoning efficiency and an increase in hardware resource requirements, thereby reducing the user experience. In order to solve the deployment problem in such multi-task scenarios, multiple edge terminals are often used or several external computing devices are expanded on the edge terminal to perform division of labor and parallel processing, but this also increases the hardware cost, which is not a good solution. Summary of the invention

[0003] The purpose of the present invention is to provide a model edge deployment method based on multi-task fusion, which solves the problems of redundant time consumption, high hardware resource occupancy, etc. brought about by the traditional method of multi-model serial execution when performing multi-task deployment by establishing a set of end-to-end multi-task fusion detection and recognition methods. It can effectively improve the execution efficiency of multiple tasks in the edge terminal and achieve beneficial detection effects.

[0004] The technical solution adopted by the present invention is as follows:

[0005] A model edge deployment method based on multi-task fusion, comprising:

[0006] S1: Obtain image samples and corresponding manual annotation information in a multi-task mixed detection scenario to build a training sample library, where the manual annotation information includes classification information and location information of rectangular key areas of each task target in the image sample;

[0007] S2: Use the training sample library in S1 to train the lightweight network and obtain a one-stage hybrid target detection model file;

[0008] S3: According to the setting requirements of each task target in the multi-task mixed detection scenario, the image samples corresponding to each task target that needs to be processed in the second stage are respectively annotated in the second stage, and a second stage training sample set for each task target that needs to be processed in the second stage is constructed based on the classification information and position information of the rectangular key area of ​​each task target that needs to be processed in the second stage and the second stage annotation;

[0009] S4: For each task target that needs to be processed in the second stage, the lightweight network is trained using the corresponding second-stage training sample set to obtain several second-stage processing model files;

[0010] S5: Build a one-stage detection program at the edge of the model, input the image to be detected into the one-stage hybrid target detection model file, run forward reasoning calculation, and obtain the classification information and location information of the rectangular key area of ​​each task target in the image to be detected;

[0011] S6: Build a multi-branch two-stage processing program at the edge of the model. For each task target that needs to be processed in the second stage in S5, input the classification information and location information of the rectangular key area into the corresponding two-stage processing model file under different branches to obtain the result output in the multi-task mixed detection scenario.

[0012] Furthermore, the step S1 also includes:

[0013] The python script is used to count the number of rectangular key areas of each category of task targets. For categories whose number is less than the set threshold, the training samples are expanded by re-shooting or transforming the original images using data enhancement technology.

[0014] Furthermore, the lightweight network selected in step S2 is a YOLOX-Tiny or NanoDet network.

[0015] Further, in step S2: a training set and a test set are randomly divided in the training sample library at a ratio of 4:1, and the lightweight network is trained using the training set to obtain an intermediate model file; the intermediate model file is then verified using the test set. If the verification result does not meet the set requirements, each image sample in the training sample library is optimized and labeled, and the training set and the test set are re-established. The intermediate model file is iteratively trained and verified using the current training set and the test set until the verification result meets the set requirements, and the intermediate model file is used as the final one-stage hybrid target detection model file.

[0016] Furthermore, before step S5, the method also includes using a model conversion tool provided by the edge terminal chip manufacturer to perform model conversion processing on the first-stage mixed target detection model file and the second-stage processing model file to obtain several model files suitable for deployment in the edge terminal.

[0017] Further, in step S5: a first-stage detection program is constructed based on the API interface provided by the edge chip manufacturer, and is encoded in a heterogeneous computing manner to implement forward reasoning calculations for the first-stage detection.

[0018] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following method is implemented:

[0019] S1: Obtain image samples and corresponding manual annotation information in a multi-task mixed detection scenario to build a training sample library, where the manual annotation information includes classification information and location information of rectangular key areas of each task target in the image sample;

[0020] S2: Use the training sample library in S1 to train the lightweight network and obtain a one-stage hybrid target detection model file;

[0021] S3: According to the setting requirements of each task target in the multi-task mixed detection scenario, the image samples corresponding to each task target that needs to be processed in the second stage are respectively annotated in the second stage, and a second stage training sample set for each task target that needs to be processed in the second stage is constructed based on the classification information and position information of the rectangular key area of ​​each task target that needs to be processed in the second stage and the second stage annotation;

[0022] S4: For each task target that needs to be processed in the second stage, the lightweight network is trained using the corresponding second-stage training sample set to obtain several second-stage processing model files;

[0023] S5: Build a one-stage detection program at the edge of the model, input the image to be detected into the one-stage hybrid target detection model file, run forward reasoning calculation, and obtain the classification information and location information of the rectangular key area of ​​each task target in the image to be detected;

[0024] S6: Build a multi-branch two-stage processing program at the edge of the model. For each task target that needs to be processed in the second stage in S5, input the classification information and location information of the rectangular key area into the corresponding two-stage processing model file under different branches to obtain the result output in the multi-task mixed detection scenario.

[0025] The present invention also provides a model edge deployment device based on multi-task fusion, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following method when executing the computer program:

[0026] S1: Obtain image samples and corresponding manual annotation information in a multi-task mixed detection scenario to build a training sample library, where the manual annotation information includes classification information and location information of rectangular key areas of each task target in the image sample;

[0027] S2: Use the training sample library in S1 to train the lightweight network and obtain a one-stage hybrid target detection model file;

[0028] S3: According to the setting requirements of each task target in the multi-task mixed detection scenario, the image samples corresponding to each task target that needs to be processed in the second stage are respectively annotated in the second stage, and a second stage training sample set for each task target that needs to be processed in the second stage is constructed based on the classification information and position information of the rectangular key area of ​​each task target that needs to be processed in the second stage and the second stage annotation;

[0029] S4: For each task target that needs to be processed in the second stage, the lightweight network is trained using the corresponding second-stage training sample set to obtain several second-stage processing model files;

[0030] S5: Build a one-stage detection program at the edge of the model, input the image to be detected into the one-stage hybrid target detection model file, run forward reasoning calculation, and obtain the classification information and location information of the rectangular key area of ​​each task target in the image to be detected;

[0031] S6: Build a multi-branch two-stage processing program at the edge of the model. For each task target that needs to be processed in the second stage in S5, input the classification information and location information of the rectangular key area into the corresponding two-stage processing model file under different branches to obtain the result output in the multi-task mixed detection scenario.

[0032] Beneficial Effects

[0033] Compared with the prior art, the advantages of the present invention are: (1) It proposes an end-to-end multi-task fusion detection and recognition method, which solves the problems of time-consuming and high hardware resource occupation caused by the serial execution of multiple tasks on the edge terminal, and can effectively improve the execution efficiency of multiple tasks in the edge terminal; (2) It proposes an improved lightweight network structure implementation idea suitable for the characteristics of the embedded hardware platform, which can perform targeted optimization processing on the original algorithm model and improve the reasoning efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1A flow chart of a model edge deployment method based on multi-task fusion according to the present invention;

[0035] Figure 2 It is a flowchart of the traditional serial execution method when deploying multiple tasks in an embodiment of the present invention;

[0036] Figure 3 It is a flowchart of the multi-task fusion method based on the present invention when deploying multiple tasks in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention proposes a model edge deployment method based on multi-task fusion, which establishes an end-to-end multi-target detection and recognition method, and uses an improved lightweight idea to build a network model. It can maximize the use of edge terminal hardware resources to complete business needs in complex multi-task scenarios, avoid the redundant time and hardware resource occupation caused by the serial execution of multiple tasks, improve the efficiency of task execution, and thus improve the user experience.

[0038] The invention is further described below with reference to the accompanying drawings and specific embodiments.

[0039] This embodiment introduces a model edge deployment method based on multi-task fusion. The specific process is as follows: Figures 1 to 3 1. Acquisition and labeling of image samples in the first stage

[0040] Use cameras to directly collect data in substations, use crawler tools to collect data online, or use public standard data sets to establish image samples in hybrid multi-task detection scenarios, and manually screen to obtain a sample library with sufficient number and obvious features.

[0041] Use Python scripts to count the number of each category, and take additional photos for categories with a small number, or use data enhancement techniques such as pixel content transformation and spatial geometric transformation on the original image to obtain an expanded sample library.

[0042] This embodiment considers the detection and recognition of multiple task scenarios in substation operation and maintenance, including helmet wearing detection, face detection and recognition, text detection and recognition, small animal (foreign object) intrusion detection, personnel intrusion detection, screen pressure plate status recognition, knife switch opening and closing status recognition, switch opening and closing indication recognition, pointer and digital instrument reading recognition, fire detection, smoking detection, personnel fall detection, etc. The total number of training samples is about 68,747 images, including 27 classification detection targets.

[0043] Open source image annotation tools are used to manually annotate the key areas of the images in the sample library. The annotated information includes the location coordinate information and classification information of the key areas. After the annotation is completed, an annotation file in json format is obtained.

[0044] Use a python script to convert the annotation file in json format into an annotation file in the COCO standard dataset format supported by the YOLOX algorithm, and randomly divide the sample library of each recognition task into training sets and test sets in a 4:1 ratio as the data source for subsequent training of the network model.

[0045] 2. Construction of a lightweight network for improved target detection in one stage

[0046] This embodiment selects the YOLOX-Tiny lightweight network based on the Pytorch deep learning framework as the target detection benchmark network. The YOLOX algorithm is a new generation of real-time target detection network proposed by Megvii Technology in July 2021. The algorithm switches the YOLO detector to anchor-free and performs other advanced detection technologies, namely decoupled head and leading label assignment strategy SimOTA, achieving state-of-the-art results in a large number of models. YOLOX-Tiny is one of the lightweight networks in this series of algorithms that is suitable for porting to edge devices for operation. It can greatly reduce the number of model parameters while maintaining high detection accuracy, and its overall performance is excellent.

[0047] This embodiment selects Rockchip's RK3399Pro platform terminal as the edge terminal device for multi-task deployment applications. The RK3399Pro platform uses an AI chip with a CPU+GPU+NPU hardware structure design, which has extremely strong AI computing performance. Its integrated NPU integrates Rockchip's many years of experience in machine vision, voice processing, deep learning and other fields, and has achieved multi-field, multi-industry, and multi-scenario commercial use in the field of AI artificial intelligence, including smart homes, AI smart sweeping robots, IoT AI speakers, and more.

[0048] According to the structural characteristics of the NPU hardware deployed on the edge terminal RK3399Pro, the YOLOX-Tiny network structure is optimized. These optimization measures can effectively improve the inference efficiency under the target hardware environment, including: (1) replacing the activation function from Leaky Relu to Relu activation function, realizing the fusion of convolution layer and Relu activation function during model construction, forming the basic calculation unit of ConvReLU, thereby saving bandwidth, improving performance, and having higher accuracy; (2) replacing the slice structure with an equivalent convolutional network to solve the problem that rknpu cannot perform slice operations; (3) replacing the large-size maxpool structure with multiple small-size maxpool structures, which can significantly improve the inference speed on rknpu.

[0049] Of course, this embodiment only takes Rockchip's RK3399Pro platform as an example of an edge terminal device. In actual implementation, other edge hardware platforms can also be used, such as an edge device platform based on Intel x86 CPU or an edge device platform based on Nvidia GPU. In this case, the network structure can be optimized according to the characteristics of the edge hardware platform to improve the performance of the model in the reasoning stage.

[0050] 3. Training of the one-stage detection model

[0051] In this embodiment, a containerized model training environment is built on the GPU server of TITAN V, and the intermediate detection model file of the multi-task target is obtained by training 300 rounds based on the improved YOLOX-Tiny network. Then the intermediate model file is evaluated using the test set samples, and the evaluation indicators can comprehensively consider the mainstream mAP (multi-category average precision), Precision (accuracy), Recall (recall rate), Flops (computing power required for the model) and other indicators in the deep learning model evaluation. It is judged whether the evaluation indicators meet the technical specifications or the requirements for promotion and application. If not, the training environment is re-established by optimizing sample annotation, image data enhancement processing, adjusting hyperparameters, etc. Then, through repeated iterative training and evaluation, the final one-stage mixed target detection model file and the corresponding training mixed sample library are obtained.

[0052] 4. Construction of two-stage multi-task network structure and model training

[0053] Analyzing the detection target recognition characteristics in multiple task scenarios in the first stage, the second stage processing process in this embodiment can be divided into the following task types:

[0054] (1) Semantic segmentation task: In this embodiment, pointer instrument reading recognition belongs to this type of task. Pointer instrument reading recognition requires that after the pointer instrument target is detected in the first stage, the instrument scale and pointer are fine-grained semantic segmentation is performed in the second stage to determine the scale position information pointed by the current pointer, so as to identify the reading value marked by the current instrument; this embodiment adopts the lightweight segmentation algorithm Unet for this semantic segmentation task.

[0055] (2) Classification task: In this embodiment, the screen pressure plate state recognition belongs to this type of task. The screen pressure plate state recognition requires that after the small pressure plate target is detected in the first stage, the actual insertion and retraction state of the pressure plate is subdivided into two classification categories, namely, inserted and not inserted, in the second stage, so as to identify the current insertion and retraction state of the pressure plate; this embodiment adopts the lightweight classification algorithm GhostNet for this classification task.

[0056] (3) Recognition tasks: In this embodiment, face detection and recognition, text detection and recognition, and person tracking belong to this type of task.

[0057] (a) Face recognition requires that after the key areas of the face are detected in the first stage, the features of the face area are encoded in the second stage, and the difference between the feature encoding and the face database is calculated to verify the identity of the face. This embodiment uses the lightweight face recognition algorithm ArcFace for this face recognition task;

[0058] (b) Text recognition requires that after the key area of ​​the text is detected in the first stage, the direction of the text area and the text semantics recognition are performed in the second stage to extract the text content in the key area. This embodiment uses MobileNetV3 and CRNN recognition algorithms for the text classification and recognition tasks respectively.

[0059] (4) Discrimination task: The recognition of the switch opening and closing indication state in this embodiment belongs to this type of task. The characteristic is that no model needs to be built in the second stage, and some discrimination processing logic needs to be executed in the CPU. The recognition of the switch opening and closing indication state requires that after the key target areas such as the switch opening and closing indication "open", "closed", and "pointer" are detected in the first stage, the discrimination algorithm based on the overlap degree is used to judge the switch opening and closing state represented in the current image.

[0060] (5) Display tasks: Most of the tasks in this embodiment belong to this type of task. The characteristics are that the second stage does not need to build a model or execute some judgment processing logic. The results of the first stage can be directly displayed and output. For example, the second stage directly displays that there is a fire in the current image, or that someone is smoking, etc.

[0061] Of course, the above classification only represents the task classification type of this embodiment. The two-stage processing process may also have other task categories. In this case, it is necessary to construct a two-stage model based on the actual task characteristics.

[0062] According to the characteristics of each task scenario, determine whether the task needs to build a model. If the model needs to be built, collect the second-stage training sample images under the current task, and build the respective processing model files based on the designed lightweight network structure; if the model does not need to be built, there is no need to collect samples and train.

[0063] 5. Conversion and optimization of original model

[0064] Using the model conversion tool provided by the edge terminal chip manufacturer, the models trained in all the first-stage detection and second-stage processing processes in multi-task scenarios are optimized to obtain model files suitable for deployment in edge terminals.

[0065] In this embodiment, the improved YOLOX-Tiny network model obtained by training in deep learning frameworks such as pytorch and tensorflow, as well as the network models of Unet, GhostNet, ArcFace, MobileNetV3, CRNN, etc. in the second stage, are converted into RKNN format model files supported by edge chips through the RKNN tool provided by Rockchip. The RKNN tool can convert deep learning models of different frameworks such as Pytorch, TensorFlow, Caffe, etc. into RKNN models that can be parsed by NPU, and it is necessary to grasp the loss of conversion accuracy. When the loss is too large, it is necessary to consider adopting a hybrid quantization mode. It should be noted that not all operators in the above algorithm can be converted by the conversion tool. It may be necessary to replace the network structure that is not supported in the original algorithm with an equivalent structure, or to build a custom operator in the RKNN tool to successfully complete the model conversion. In the above model conversion process, some optimization measures are performed on the original model, mainly including: model quantization, kernel fusion, calculation graph optimization, etc.

[0066] Model quantization refers to converting the storage format of model parameters from Float32 to int8 / uint8 format to reduce the amount of calculation and model size; kernel fusion refers to fusing the kernel functions corresponding to the operation nodes in a computational graph into a sequence function to increase the granularity of a single operation and reduce the number of kernel calls; computational graph optimization refers to performing operator fusion, constant parameter pre-computation, data layout conversion and other processing on the network during the inference stage to reduce the number of parameters and increase the inference speed.

[0067] In this embodiment, an asymmetric uint8 quantization method is used to convert the FP32 floating-point tensor in the original model into a Uint8 type tensor. The remaining optimization measures are completed by the RKNN tool based on the network structure improvement in the original model.

[0068] 6. Construction of the first-stage detection program

[0069] A first-stage detection algorithm program segment is constructed based on the API interface provided by the edge chip manufacturer. The program segment includes sub-modules such as data acquisition and preprocessing, model loading and calculation, detection post-processing, and result display.

[0070] In order to make full use of the hardware resources of the edge chip, this embodiment is based on the API interface provided by Rockchip and adopts a heterogeneous computing method based on CPU+NPU chips to implement the construction of the first-stage detection program.

[0071] The general implementation process is as follows:

[0072] (a) Use the CPU to load the RKNN model into the memory and pass it to the NPU device during initialization. Set some necessary parameters to complete the model loading and context initialization.

[0073] (b) Processing the image data obtained from the camera through the Rockchip hardware codec application (MPP) to obtain input image data to be detected;

[0074] (c) Using the CPU to schedule the NPU chip to perform forward reasoning calculations on the model, and performing necessary data post-processing operations in the CPU to obtain the reasoning results of the input image data, which includes the classification information and position coordinate information of several rectangular key areas;

[0075] (d) Based on the obtained target classification information and location information, obtain input data for the two-stage processing process.

[0076] 7. Construction of the second-stage processing program

[0077] A two-stage processing algorithm program segment is constructed based on the API interface provided by the edge chip manufacturer. The program segment includes sub-modules such as model loading and calculation, post-processing, result analysis, and result display.

[0078] Similarly, this embodiment is based on the API interface provided by Rockchip and adopts a heterogeneous computing method based on CPU+NPU chips to implement the construction of the two-stage processing program.

[0079] The two-stage processing procedure is different from the one-stage detection procedure in that the one-stage detection procedure takes the entire image to be detected as input, and outputs the classification information and position information of several rectangular key areas through forward processing of a lightweight model; while the two-stage processing procedure takes the information of several rectangular key areas output by the first stage as input, and executes the branch processing process according to the task type to which the classification information belongs.

[0080] In this embodiment, the branch processing flow is generally as follows:

[0081] The loop processes each rectangular key area:

[0082] (a) If the classification information of the rectangular key area is "pointer instrument", the two-stage Unet network model trained under the semantic segmentation task is used, the processing program is built using the API interface, and the forward reasoning of the model is completed using heterogeneous computing to obtain the pixel segmentation information of the "pointer" and "scale". Then, the pointer reading value of the segmented key area is recognized in the CPU chip based on the traditional OpenCV library and recognition logic;

[0083] (b) If the classification information of the rectangular key area is "screen pressure plate", the two-stage MobileNet network model trained under the classification task is used, the processing program is built using the API interface, and the forward reasoning of the model is completed using heterogeneous computing to obtain the "separated" or "merged" state of the pressure plate;

[0084] (c) If the classification information of the rectangular key area is a recognition task, taking "face" as an example, the two-stage ArcFace network model trained under the recognition task is used, the processing program is constructed using the API interface, and the forward reasoning of the model is completed using heterogeneous computing to obtain the feature encoding information of the input face area. According to the matching degree of the encoding information with the Euclidean distance of all face encodings in the face database, and the preset threshold, the information of the current face is determined and output;

[0085] (d) If the classification information of the rectangular key area is "switch opening and closing indication state identification", there is no need to use the API interface and network model, and the CPU is directly used to build the state identification logic, and the discrimination algorithm based on the overlap degree is used to determine the switch opening and closing state represented by the current rectangular key area;

[0086] (e) If the classification information of the rectangular key area is a regular display task, taking "fire detection" as an example, the classification and location information obtained in the first stage is directly output.

[0087] 8. Results display and storage

[0088] After the first and second stage reasoning calculations of the image to be detected, the processing results of the key areas in the current image are output. The results can be annotated and output in the original image through the output interface, or saved in a file for output.

[0089] The invention discloses a model edge deployment method based on multi-task fusion, which collects sample image sets in multi-task scenarios, performs data enhancement processing and manual annotation on the acquired sample images, and establishes a unified multi-task mixed sample library; according to the characteristics of the edge terminal hardware unit to be deployed, an improved lightweight network is used to train the mixed sample library to obtain a lightweight one-stage mixed target detection model file based on multi-task fusion; according to the characteristics of the detection targets in multiple task scenarios, for each task that needs to build a two-stage processing model, the collected images under the task are annotated in two stages, and a lightweight processing network under the specific task is designed and trained to obtain the following: A two-stage lightweight processing model file is constructed; the model conversion tool provided by the edge terminal chip manufacturer is used to perform model conversion processing on all the above model files to obtain several model files suitable for deployment in edge terminals; a one-stage detection program is constructed, the image data to be detected is used as input, and forward reasoning calculations are run on heterogeneous chips to obtain the classification information and position information of the rectangular key areas of the multi-task targets in the image; a multi-branch two-stage processing program is constructed, the information of each rectangular key area is used as input, and the processing programs under different branches are executed according to the task type to which the classification information of the key area belongs, and the result output under the multi-task mixed application scenario is obtained. The method of the present invention integrates the mixed detection scenario of multiple tasks into a set of application processes, establishes an end-to-end task implementation method, and the models in the process all adopt the design idea of ​​an improved lightweight network, which solves the problems of redundant time consumption and high hardware resource occupancy caused by the multi-model serial execution method commonly used in multi-task deployment in the traditional way, and can improve the execution efficiency of multiple tasks in the edge terminal and effectively improve the user experience.

[0090] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0091] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0094] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A model edge deployment method based on multi-task fusion, characterized in that: include: S1: Obtain image samples and corresponding manual annotation information in a multi-task mixed detection scenario to build a training sample library, where the manual annotation information includes classification information and location information of rectangular key areas of each task target in the image sample; S2: Use the training sample library in S1 to train the lightweight network and obtain a one-stage hybrid target detection model file; S3: According to the setting requirements of each task target in the multi-task mixed detection scenario, the image samples corresponding to each task target that needs to be processed in the second stage are respectively annotated in the second stage, and a second stage training sample set for each task target that needs to be processed in the second stage is constructed based on the classification information and position information of the rectangular key area of ​​each task target that needs to be processed in the second stage and the second stage annotation; S4: For each task target that needs to be processed in the second stage, the lightweight network is trained using the corresponding second-stage training sample set to obtain several second-stage processing model files; S5: Build a one-stage detection program at the edge of the model, input the image to be detected into the one-stage hybrid target detection model file, run forward reasoning calculation, and obtain the classification information and location information of the rectangular key area of ​​each task target in the image to be detected; S6: Build a multi-branch two-stage processing program at the edge of the model. For each task target that needs to be processed in the second stage in S5, input the classification information and location information of the rectangular key area into the corresponding two-stage processing model file under different branches to obtain the result output in the multi-task mixed detection scenario.

2. According to the method for edge deployment of a model based on multi-task fusion according to claim 1, it is characterized in that: The step S1 also includes: The python script is used to count the number of rectangular key areas of each category of task targets. For categories whose number is less than the set threshold, the training samples are expanded by re-shooting or transforming the original images using data enhancement technology.

3. According to the method for edge deployment of a model based on multi-task fusion according to claim 1, it is characterized in that: In step S2, the lightweight network selected is a YOLOX-Tiny or NanoDet network.

4. According to the method for edge deployment of a model based on multi-task fusion according to claim 1, it is characterized in that: In step S2: The training sample library is randomly divided into training set and test set in a ratio of 4:1, and the lightweight network is trained with the training set to obtain an intermediate model file. The intermediate model file is then verified with the test set. If the verification result does not meet the set requirements, each image sample in the training sample library is optimized and labeled, and the training set and test set are re-established. The intermediate model file is iteratively trained and verified using the current training set and test set until the verification result meets the set requirements. The intermediate model file is used as the final one-stage hybrid target detection model file.

5. According to the method for edge deployment of a model based on multi-task fusion according to claim 1, it is characterized in that: Before step S5, the method also includes using a model conversion tool provided by the edge terminal chip manufacturer to perform model conversion processing on the first-stage mixed target detection model file and the second-stage processing model file to obtain several model files suitable for deployment in the edge terminal.

6. According to the method for edge deployment of a model based on multi-task fusion according to claim 1, it is characterized in that: In the step S5: a first-stage detection program is constructed based on the API interface provided by the edge chip manufacturer, and is encoded in a heterogeneous computing manner to implement forward reasoning calculations for the first-stage detection.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A model edge deployment device based on multi-task fusion, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images