Design method, system, equipment and deployment method of multi-task neural network model

Abstract

The present invention discloses a design method, system, device, and deployment method for a multi-task neural network model. The design method includes: using a target detection network model as a backbone network architecture, defining the output of branch feature maps within the backbone network architecture; constructing a semantic segmentation model and a lane recognition model; connecting the semantic segmentation model and lane recognition model to the backbone network architecture based on the output of the branch feature maps to build an overall network model; and training the overall network model using a road condition dataset to obtain a multi-task neural network model. The present invention integrates each subclass task required for autonomous driving within an embedded operating system with limited computing power, significantly reducing the computational complexity, parameter count, and hardware memory consumption of the overall task, thereby meeting the requirements of multi-task autonomous driving in a conventional embedded environment.

Description

Technical Field

[0001] The present invention relates to a design method, system, equipment and deployment method of a multi-task neural network model, and belongs to the technical field of autonomous driving scenarios. Background Art

[0002] Autonomous driving technology is developing rapidly. Since the release of the AlexNet convolutional neural network model in 2012, deep learning has been widely applied in autonomous driving scenarios. Both emerging car manufacturers and traditional automakers are continuously promoting the application of deep learning in autonomous driving scenarios, and the field is showing positive signs of development.

[0003] In deep learning technology, convolutional neural network models, as a technique for simulating human vision, have been continuously innovated over the past decade and are being applied to autonomous driving scenarios. Typical applications of convolutional neural network models in autonomous driving scenarios include lane detection, drivable area detection, object detection, path planning, depth estimation, and more. Each function requires a separate neural network model for input and output, which is extremely challenging for embedded operating systems with limited computing power. Conventional methods for reducing computational complexity include reducing overall model parameters, modifying related models, or lowering the resolution of input images to achieve real-time functionality. However, these approaches still struggle to address the context switching delays associated with memory access for multiple models. Furthermore, the allocation and blocking of computing resources during the switching process between multiple models can also degrade the performance of real-time functions.

[0004] For deployment in real-world embedded scenarios, the multi-task neural network model, as a shared backbone network model, can significantly improve the model's real-time performance. This is reflected in more efficient memory utilization, with no context switching. Furthermore, the data pipeline is easily optimized, facilitating actual data processing and output. For example:

[0005] (1) CN201611199772.8, this invention provides a vehicle attribute recognition method based on a multi-task convolutional neural network, which includes two parts: a training process and a recognition process. The specific process includes: obtaining a vehicle image to be recognized, designing a multi-task convolutional neural network structure and training a network model for vehicle attribute recognition, recognizing the vehicle type and regressing the vehicle window position coordinates, designing a vehicle image mask and generating a new vehicle image, extracting multi-task convolutional neural network features of the new vehicle image, training an SVM classification model, and recognizing the vehicle color. Although this solution utilizes a multi-task model, it uses a traditional support vector machine for the classification part, which has shortcomings in recognition performance and end-to-end convenience. In addition, the conventional NVIDIA architecture has limited optimization of traditional support vector machines.

[0006] (2) CN202011328854.4, this invention provides a method for planning the drivable area of a vehicle based on a multi-task neural network. Based on a hard parameter sharing mechanism, it constructs a lightweight multi-task neural network with two functions: semantic segmentation and target detection. The functions include lane segmentation, pedestrian detection, vehicle detection, monocular ranging, etc. However, this solution uses a typical model such as PSPNet, which has heavy computational performance and a low input resolution of 416x416. For lane line detection, a segmentation method is used, which requires post-processing clustering and does not have an end-to-end output type. In actual application scenarios, there are certain post-processing steps. Summary of the Invention

[0007] In view of this, the present invention provides a design method, system, computer device, storage medium of a multi-task neural network model and a deployment method, system, computer device and storage medium of a multi-task neural network model, which integrates each subclass task required for autonomous driving in an embedded operating system with limited computing power, greatly reducing the amount of calculation, parameter quantity and hardware memory consumption required for the overall task, thereby meeting the requirements of multi-task autonomous driving in a conventional embedded environment.

[0008] The first object of the present invention is to provide a design method for a multi-task neural network model.

[0009] The second object of the present invention is to provide a design system for a multi-task neural network model.

[0010] The third object of the present invention is to provide a deployment method for a multi-task neural network model.

[0011] A fourth object of the present invention is to provide a deployment system for a multi-task neural network model.

[0012] A fifth object of the present invention is to provide a computer device.

[0013] A sixth object of the present invention is to provide a storage medium.

[0014] The first object of the present invention can be achieved by adopting the following technical solutions:

[0015] A method for designing a multi-task neural network model, the method comprising:

[0016] The target detection network model is used as the backbone network architecture, and the output of the branch feature map is defined in the backbone network architecture;

[0017] Build semantic segmentation model and lane recognition model;

[0018] Based on the output of the branch feature maps, the semantic segmentation model and lane recognition model are connected to the backbone network architecture to build the overall network model.

[0019] The overall network model is trained using the road condition dataset to obtain a multi-task neural network model.

[0020] Furthermore, the semantic segmentation model includes one of the Base semantic segmentation model, the BiSeNet semantic segmentation model, the DDRNet semantic segmentation model, the PSPNet semantic segmentation model and the DeepLabV3+ semantic segmentation model; and the lane line recognition model is the LaneATT network model.

[0021] Furthermore, the BiSeNet semantic segmentation model includes an ARM module and an FFM module;

[0022] The ARM module includes a GlobalPool module, a Conv2d module, and a Sigmoid module connected in sequence;

[0023] The FFM module includes a CONV module, a GlobalPool module, a Conv2d module, a Silu module, a Conv2d module, and a Sigmoid module connected in sequence.

[0024] Furthermore, the target detection network model is a YOLOv5s network model.

[0025] Furthermore, the output of the branch feature map includes a first feature output, a second feature output, and a third feature output;

[0026] The first feature output is the sixteenth layer feature map output of the YOLOv5s network model;

[0027] The second feature output is the sixteenth layer feature map output, the nineteenth layer feature map output and the twenty-second layer feature map output of the YOLOv5s network model;

[0028] The third feature output is the fourth-layer feature map output and the nineteenth-layer feature map output of the YOLOv5s network model.

[0029] Furthermore, based on the output of the branch feature map, the semantic segmentation model and the lane recognition model are connected to the backbone network architecture to build an overall network model, which specifically includes:

[0030] Connect the input of the Base semantic segmentation model to the first feature output, and connect the input of the lane recognition model to the first feature output or the second feature output to build the overall network model.

[0031] Alternatively, connect the input of the BiSeNet semantic segmentation model to the second feature output, and connect the input of the lane recognition model to the first feature output or the second feature output to build the overall network model.

[0032] Alternatively, the input of the DDRNe semantic segmentation model is connected to the second feature output, and the input of the lane recognition model is connected to the first feature output or the second feature output, thereby building an overall network model.

[0033] Alternatively, connect the input of the PSPNet semantic segmentation model to the second or third feature output, and connect the input of the lane recognition model to the first or second feature output to build the overall network model.

[0034] Alternatively, connect the input of the DeepLabV3+ semantic segmentation model with the second or third feature output, and connect the input of the lane recognition model with the first or second feature output to build the overall network model.

[0035] The second object of the present invention can be achieved by adopting the following technical solutions:

[0036] A multi-task network model design system, the system comprising:

[0037] A definition unit is used to use the target detection network model as the backbone network architecture and define the output of the branch feature map in the backbone network architecture;

[0038] Construction unit, used to build semantic segmentation model and lane line recognition model;

[0039] The building unit is used to connect the semantic segmentation model and lane recognition model with the backbone network architecture based on the output of the branch feature map to build the overall network model;

[0040] The training unit is used to train the overall network model using the road condition dataset to obtain a multi-task neural network model.

[0041] The third object of the present invention can be achieved by adopting the following technical solutions:

[0042] A method for deploying a multi-task neural network model, the method comprising:

[0043] Get a multi-task neural network model;

[0044] Perform model conversion and acceleration on multi-task neural network models;

[0045] Modify the multi-task neural network model after accelerated processing;

[0046] Deploy the modified multi-task neural network model to the hardware board;

[0047] The multi-task neural network model is obtained through the above-mentioned design method.

[0048] The fourth object of the present invention can be achieved by adopting the following technical solutions:

[0049] A multi-task neural network model deployment system, the system comprising:

[0050] An acquisition unit, used to acquire a multi-task neural network model;

[0051] Conversion and acceleration unit, used for model conversion and acceleration processing of multi-task neural network models;

[0052] A modification unit, used to modify the multi-task neural network model after accelerated processing;

[0053] A deployment unit, used to deploy the modified multi-task neural network model to the hardware board;

[0054] The multi-task neural network model is obtained through the above-mentioned design method.

[0055] The fifth object of the present invention can be achieved by adopting the following technical solutions:

[0056] A computer device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the above-mentioned design method or the above-mentioned deployment method is implemented.

[0057] The sixth object of the present invention can be achieved by adopting the following technical solutions:

[0058] A storage medium stores a program, which, when executed by a processor, implements the above-mentioned design method or the above-mentioned deployment method.

[0059] The present invention has the following beneficial effects compared to the prior art:

[0060] Taking into account the application scenarios of autonomous driving and NVIDIA's general-purpose boards, the present invention adds other branches on the basis of the YOLOv5s network model to build a multi-task neural network model and realize multi-task output; among them, the multi-task neural network model mainly adds a lane line detection branch based on the LaneATT algorithm idea and a semantic segmentation branch based on classic semantic style network ideas such as BiseNet, and integrates each subclass task required for autonomous driving, greatly reducing the amount of calculation, parameter amount and hardware memory consumption required for the overall task, thereby meeting the requirements of multi-task autonomous driving in a conventional embedded environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0062] Figure 1 Flowchart of the design method of the multi-task neural network model of Example 1 of the present invention.

[0063] Figure 2 This is a schematic diagram of the YOLOv5s network model and the connection head part of Example 1 of the present invention.

[0064] Figure 3 This is a schematic diagram of the DAPPM module of Example 1 of the present invention.

[0065] Figure 4 This is a schematic diagram of the LaneATT network model of Example 1 of the present invention.

[0066] Figure 5 Schematic diagram of the CONV module, Focus module, SPP module, BottleNeck module, ARM module and C3 module of Example 1 of the present invention.

[0067] Figure 6 Schematic diagram of the FFM module and ASPP module of Example 1 of the present invention.

[0068] Figure 7 This is a flowchart of the deployment method of the multi-task neural network model of Example 2 of the present invention.

[0069] Figure 8 This is a structural block diagram of the design system of the multi-task neural network model of Example 3 of the present invention.

[0070] Figure 9This is a structural block diagram of the deployment system of the multi-task neural network model of Example 4 of the present invention.

[0071] Figure 10 This is a structural block diagram of a computer device according to embodiment 5 of the present invention. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0073] Example 1:

[0074] According to the definition of actual needs, the multi-task neural network model of this embodiment needs to identify the following contents: target detection, semantic segmentation and lane line detection based on a specific network model; among them: in the target detection content, since the YOLOv5s network model has a good comprehensive effect of inference speed and mAP value, and this network model adopts the CSPNet architecture and OSANet architecture (backbone network architecture), it has a good comprehensive consideration for the board end, so its backbone network architecture remains unchanged; in the semantic segmentation content, taking the typical lightweight BiSeNet semantic segmentation model as an example, the ARM module, FFM module and other sub-modules within the BiSeNet semantic segmentation model have good real-time performance, so when considering the design of the semantic segmentation model, each sub-module within it is tried; in the lane line detection content, in order to avoid the subsequent clustering of the lane line recognition method of the semantic segmentation model, a lane line detection solution based on a specific network model is adopted.

[0075] like Figure 1 As shown, based on the above ideas, this embodiment provides a design method for a multi-task neural network model, which includes the following steps:

[0076] S101. Use the target detection network model as the backbone network architecture and define the output of the branch feature map in the backbone network architecture.

[0077] Step S101 includes the following steps:

[0078] S1011. Use the YOLOv5s network model as the backbone network architecture. The YOLOv5s network model is used to adjust the image size of the overall recognition input to minimize the computational complexity of the overall data. The image size is 640x640x3.

[0079] S1012. Define the output of the branch feature map in the backbone network architecture.

[0080] like Figure 2 As shown, the output of the branch feature map is defined in the backbone network architecture, specifically including: defining the first feature output, the second feature output and the third feature output; wherein: the first feature output is the sixteenth-layer feature map output of the YOLOv5s network model; the second feature output is the sixteenth-layer feature map output, the nineteenth-layer feature map output and the twenty-second-layer feature map output of the YOLOv5s network model; the third feature output is the fourth-layer feature map output and the nineteenth-layer feature map output of the YOLOv5s network model.

[0081] It is worth noting that the YOLOv5s network model, as the overall skeleton (backbone network architecture) of the multi-task neural network model, has remained basically unchanged. The modifications mainly focus on the output definition of the branch feature map and the data construction of the YOLOv5s network model.

[0082] S102: Build a semantic segmentation model and a lane line recognition model.

[0083] Furthermore, the semantic segmentation model includes one of the Base semantic segmentation model, the BiSeNet semantic segmentation model, the DDRNet semantic segmentation model, the PSPNet semantic segmentation model and the DeepLabV3+ semantic segmentation model, and the lane line recognition model is the LaneATT network model.

[0084] In this embodiment, the Base semantic segmentation model is constructed with reference to the Base benchmark reference head paper. The Base semantic segmentation model directly selects the first feature output, and after the C3, C3SPP, Dropout, and Conv2d modules, it is upsampled 8 times to the original image for training; the BiSeNet semantic segmentation model is constructed with reference to the original BiSeNet paper, in which the original ARM module and the original FFM module are improved; the DDRNet semantic segmentation model is constructed with reference to the original DDRNet paper; the PSPNet semantic segmentation model is constructed with reference to the original PSPNet paper, in which the PSPNet semantic segmentation model includes the DAPPM module, such as Figure 3 As shown in the figure, the DeepLabV3+ semantic segmentation model is constructed with reference to the original DeepLabV3+ paper, in which the original ASPP module is improved. The lane line recognition model is constructed and improved with reference to the original LaneATT part. The improvements include: adding a feature pyramid architecture to the original LaneATT part to obtain the LaneATT network model, as shown in the figure. Figure 4 shown.

[0085] like Figure 5 and Figure 6 As shown, in this embodiment, the ARM module includes a GlobalPool module, a Conv2d module, and a Sigmoid module connected in sequence; the FFM module includes a CONV module, a GlobalPool module, a Conv2d module, a Silu module, a Conv2d module, and a Sigmoid module connected in sequence; the ASPP module includes a parallel 1×1CONV module, a 3×3CONV rate6 module, a 3×3CONV rate12 module, a 3×3CONV rate18 module, and a GlobalPool module.

[0086] S103. Based on the output of the branch feature map, the semantic segmentation model and the lane line recognition model are connected to the backbone network architecture to build an overall network model.

[0087] Before step S103, debug each connection head part (interface); after debugging, combine the semantic segmentation model, lane line recognition model and backbone network architecture, i.e. step S103, to form a unified network model system, i.e., the overall network model.

[0088] In this embodiment, the input of the BiSeNet semantic segmentation model is connected to the second feature output, and the input of the lane recognition model is connected to the second feature output, thereby building an overall network model.

[0089] In another embodiment, the input of the BiSeNet semantic segmentation model is connected to the second feature output, and the input of the lane recognition model is connected to the first feature output, thereby building an overall network model.

[0090] In other embodiments, the input of the Base semantic segmentation model is connected to the first feature output, and the input of the lane recognition model is connected to the first feature output or the second feature output, thereby building an overall network model.

[0091] In other embodiments, the input of the DDRNe semantic segmentation model is connected to the second feature output, and the input of the lane recognition model is connected to the first feature output or the second feature output, thereby building an overall network model.

[0092] In other embodiments, the input of the PSPNet semantic segmentation model is connected to the second feature output or the third feature output, and the input of the lane recognition model is connected to the first feature output or the second feature output, thereby building an overall network model.

[0093] In other embodiments, the input of the DeepLabV3+ semantic segmentation model is connected to the second feature output or the third feature output, and the input of the lane recognition model is connected to the first feature output or the second feature output, so as to build an overall network model.

[0094] In addition, this embodiment verifies each connector part to achieve consistency, so that the model can be freely configured and combined.

[0095] S104: Using the traffic data set to train the overall network model to obtain a multi-task neural network model.

[0096] Based on the actual performance requirements, some detailed architecture details of the overall network model were tested and modified. A separate dataset corresponding to each semantic segmentation model was constructed as input for the semantic part. Based on the size of the output feature map, the model was upsampled to the original image and the relevant loss function was defined. Similarly, based on the analysis of the lane line recognition model, the relevant loss function and the fitted lane line output were defined.

[0097] Specifically, during the training process, it is necessary to comprehensively consider the outputs of the three branches. The three branches correspond to the three models in the overall network model. The consideration principles are as follows:

[0098] A. Define the loss function weights for the target task: To ensure the reliability of the target task of each branch, each branch uses the average loss function as its loss function. During initial training, adjust the loss function weights to make the weights of the loss functions of each branch equal. During actual training, after a certain number of rounds, adjust the branches with large loss function weights to keep the overall loss function weight within a certain range as much as possible.

[0099] B. Define separate datasets: Due to the incompatibility of datasets, a separate dataset is constructed for each subclass task, and the data of each branch is input into the pipeline for testing and optimization to ensure its reliability.

[0100] Specifically, a separate dataset is constructed for each sub-category task. For the YOLOv5s network model, a separate dataset based on target detection is constructed based on the original open source code. For each semantic segmentation model, a separate dataset corresponding to each semantic segmentation model is constructed as the input of the semantic part. For the lane line recognition model, a lane line image dataset is constructed.

[0101] In this embodiment, the dataset based on target detection, the dataset corresponding to the BiSeNet semantic segmentation model, and the lane line image dataset are collectively referred to as a road condition dataset.

[0102] C. Use forward reasoning and reverse gradient optimization: Perform forward reasoning and reverse iteration on each branch to ensure that each branch can fully learn the capabilities of its branch task.

[0103] Finally, based on principles A to C, this embodiment uses the road condition dataset to train the overall network model to obtain a trained overall network model, namely a multi-task neural network model.

[0104] Those skilled in the art will appreciate that all or part of the steps in the method for implementing the above embodiments may be completed by instructing related hardware through a program, and the corresponding program may be stored in a computer-readable storage medium.

[0105] It should be noted that although the method operations of the above embodiments are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed to achieve the desired results. Rather, the depicted steps may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0106] Example 2:

[0107] like Figure 7 As shown, this embodiment provides a deployment method for a multi-task neural network model, which includes the following steps:

[0108] S701. Obtain a multi-task neural network model, wherein the multi-task neural network model is obtained by the design method of Example 1.

[0109] S702: Perform model conversion and acceleration processing on the multi-task neural network model.

[0110] Specifically, the multi-task neural network model is converted into an ONNX model; the ONNX model is converted into a TRT (TensorRT) model and accelerated.

[0111] S703. Modify the multi-task neural network model after the accelerated processing to overcome the incompatibility problem.

[0112] Specifically, for the incompatible parts of the multi-task neural network model after accelerated processing, OP is written or part of OP is modified to overcome the incompatibility problem.

[0113] S704: Deploy the modified multi-task neural network model to the hardware board.

[0114] Specifically, the modified multi-task neural network model is migrated to the NVIDIA NX board and quasi-real-time calculations are performed in the NVIDIA NX board to realize the deployment of the network model; wherein: the NVIDIA NX board is connected to the surround-view camera, and the surround-view camera is a six-way surround-view camera; the NVIDIA NX board is used to obtain the output required for the overall autonomous driving control.

[0115] After final testing, this embodiment achieved a single inference speed of 70ms on an NVIDIA NX board while ensuring high recall and accuracy. The speed was achieved when a surround-view camera inputted a 640x640x3 image. This can basically meet the processing requirements of this type of board for real-time tasks.

[0116] Those skilled in the art will appreciate that all or part of the steps in the method for implementing the above embodiments may be completed by instructing related hardware through a program, and the corresponding program may be stored in a computer-readable storage medium.

[0117] It should be noted that although the method operations of the above embodiments are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed to achieve the desired results. Rather, the depicted steps may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0118] Example 3:

[0119] like Figure 8 As shown, this embodiment provides a design system for a multi-task neural network model, which includes a definition unit 801, a construction unit 802, a building unit 803 and a training unit 804. The specific functions of each unit are as follows:

[0120] A definition unit 801 is used to use the target detection network model as a backbone network architecture and define the output of the branch feature map in the backbone network architecture;

[0121] A construction unit 802 is used to construct a semantic segmentation model and a lane recognition model;

[0122] A building unit 803 is used to connect the semantic segmentation model and the lane recognition model with the backbone network architecture based on the output of the branch feature map to build an overall network model;

[0123] The training unit 804 is used to train the overall network model using the road condition data set to obtain a multi-task neural network model.

[0124] Example 4:

[0125] like Figure 9 As shown, this embodiment provides a deployment system for a multi-task neural network model. The system includes an acquisition unit 901, a conversion and acceleration unit 902, a modification unit 903, and a deployment unit 904. The specific functions of each unit are as follows:

[0126] An acquisition unit 901 is used to acquire a multi-task neural network model;

[0127] The conversion and acceleration unit 902 is used to perform model conversion and acceleration processing on the multi-task neural network model;

[0128] A modifying unit 903 is used to modify the multi-task neural network model after the accelerated processing;

[0129] A deployment unit 904 is used to deploy the modified multi-task neural network model to a hardware board;

[0130] The multi-task neural network model is obtained through the design method of Example 1.

[0131] Example 5:

[0132] like Figure 10 As shown, this embodiment provides a computer device, which includes a processor 1002, a memory, an input device 1003, a display device 1004, and a network interface 1005 connected via a system bus 1001. The processor 1002 is used to provide computing and control capabilities, and the memory includes a non-volatile storage medium 1006 and an internal memory 1007. The non-volatile storage medium 1006 stores an operating system, a computer program, and a database. The internal memory 1007 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium 1006. When the computer program is executed by the processor 1002, the design method of the above-mentioned embodiment 1 and / or the deployment method of the embodiment 2 are implemented.

[0133] Example 6:

[0134] This embodiment provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, it implements the design method of the above-mentioned embodiment 1 and / or the deployment method of embodiment 2.

[0135] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0136] In this embodiment, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Furthermore, in this embodiment, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0137] The computer readable storage medium can be written in one or more programming languages or a combination thereof to execute the computer program for performing the present embodiment, including object-oriented programming languages such as Java, Python, C++, and conventional procedural programming languages such as C or similar programming languages. The program can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect via the Internet).

[0138] In summary, the present invention makes full use of the public skeleton network to collect image features, outputs each branch on each feature map, and completes the task requirements of each branch; the main branches include: a semantic segmentation branch, which is constructed based on the current classic semantic segmentation model and improves the structure of some semantic segmentation models, and the semantic segmentation model can be selected according to the actual board performance requirements. The semantic segmentation model includes one of them: Base semantic segmentation model, BiSeNet semantic segmentation model, DDRNet semantic segmentation model, PSPNet semantic segmentation model and DeepLabV3+ semantic segmentation model, and also includes: the design of a direct upsampling segmentation network; the target detection branch directly adopts the YOLOv5s network model to realize typical target detection. It is the backbone network architecture of the multi-task neural network model and has good comprehensive considerations for hardware boards; the lane line detection branch adopts the original LaneATT model and adds a feature pyramid architecture to the original LaneATT model, thereby improving the overall actual model performance.

[0139] The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes based on the technical solution and inventive concept of the present invention within the scope disclosed by the present invention, which falls within the scope of protection of the present invention.

Claims

1. A design method for a multi-task neural network model, characterized in that: The method comprises: The target detection network model is used as the backbone network architecture, and the output of the branch feature map is defined in the backbone network architecture; Build semantic segmentation model and lane recognition model; Based on the output of the branch feature maps, the semantic segmentation model and lane recognition model are connected to the backbone network architecture to build the overall network model. The overall network model is trained using the traffic data set to obtain a multi-task neural network model; The output of the branch feature map includes a first feature output, a second feature output and a third feature output; The first feature output is the sixteenth layer feature map output of the YOLOv5s network model; The second feature output is the sixteenth layer feature map output, the nineteenth layer feature map output and the twenty-second layer feature map output of the YOLOv5s network model; The third feature output is the fourth-layer feature map output and the nineteenth-layer feature map output of the YOLOv5s network model; Based on the output of the branch feature map, the semantic segmentation model and lane recognition model are connected to the backbone network architecture to build an overall network model, which specifically includes: Connect the input of the Base semantic segmentation model to the first feature output, and connect the input of the lane recognition model to the first feature output or the second feature output to build the overall network model. Alternatively, connect the input of the BiSeNet semantic segmentation model to the second feature output, and connect the input of the lane recognition model to the first feature output or the second feature output to build the overall network model. Alternatively, the input of the DDRNe semantic segmentation model is connected to the second feature output, and the input of the lane recognition model is connected to the first feature output or the second feature output, thereby building an overall network model. Alternatively, connect the input of the PSPNet semantic segmentation model to the second or third feature output, and connect the input of the lane recognition model to the first or second feature output to build the overall network model. Alternatively, connect the input of the DeepLabV3+ semantic segmentation model with the second or third feature output, and connect the input of the lane recognition model with the first or second feature output to build the overall network model.

2. The design method according to claim 1, characterized in that: The semantic segmentation model includes one of the Base semantic segmentation model, the BiSeNet semantic segmentation model, the DDRNet semantic segmentation model, the PSPNet semantic segmentation model and the DeepLabV3+ semantic segmentation model; the lane line recognition model is the LaneATT network model.

3. The design method according to claim 2, characterized in that: The BiSeNet semantic segmentation model includes an ARM module and an FFM module; The ARM module includes a GlobalPool module, a Conv2d module, and a Sigmoid module connected in sequence; The FFM module includes a CONV module, a GlobalPool module, a Conv2d module, a Silu module, a Conv2d module, and a Sigmoid module connected in sequence.

4. The design method according to any one of claims 1 to 3, characterized in that: The target detection network model is a YOLOv5s network model.

5. A deployment method, characterized in that: The method comprises: Get a multi-task neural network model; Perform model conversion and acceleration on multi-task neural network models; Modify the multi-task neural network model after accelerated processing; Deploy the modified multi-task neural network model to the hardware board; The multi-task neural network model is obtained by the design method described in any one of claims 1-4.

6. A design system for a multi-task network model, characterized in that: The system comprises: A definition unit is used to use the target detection network model as the backbone network architecture and define the output of the branch feature map in the backbone network architecture; Construction unit, used to build semantic segmentation model and lane line recognition model; The building unit is used to connect the semantic segmentation model and lane recognition model with the backbone network architecture based on the output of the branch feature map to build the overall network model; A training unit is used to train the overall network model using a road condition dataset to obtain a multi-task neural network model; The output of the branch feature map includes a first feature output, a second feature output and a third feature output; The first feature output is the sixteenth layer feature map output of the YOLOv5s network model; The second feature output is the sixteenth layer feature map output, the nineteenth layer feature map output and the twenty-second layer feature map output of the YOLOv5s network model; The third feature output is the fourth-layer feature map output and the nineteenth-layer feature map output of the YOLOv5s network model; Based on the output of the branch feature map, the semantic segmentation model and lane recognition model are connected to the backbone network architecture to build an overall network model, which specifically includes: Connect the input of the Base semantic segmentation model to the first feature output, and connect the input of the lane recognition model to the first feature output or the second feature output to build the overall network model. Alternatively, connect the input of the BiSeNet semantic segmentation model to the second feature output, and connect the input of the lane recognition model to the first feature output or the second feature output to build the overall network model. Alternatively, the input of the DDRNe semantic segmentation model is connected to the second feature output, and the input of the lane recognition model is connected to the first feature output or the second feature output, thereby building an overall network model. Alternatively, connect the input of the PSPNet semantic segmentation model to the second or third feature output, and connect the input of the lane recognition model to the first or second feature output to build the overall network model. Alternatively, connect the input of the DeepLabV3+ semantic segmentation model with the second or third feature output, and connect the input of the lane recognition model with the first or second feature output to build the overall network model.

7. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, it implements the design method described in any one of claims 1 to 4 or the deployment method described in claim 5.

8. A storage medium storing a program, characterized in that: When the program is executed by a processor, the design method described in any one of claims 1 to 4 or the deployment method described in claim 5 is implemented.

Images