Target identification method

Abstract

The invention relates to a target identification method. The method comprises the following steps: acquiring an environment image in a scene to be recognized; the environment image comprises at least one type of target recognition task; inputting the environment image into a pre-trained multi-task target recognition model to obtain recognition results of various types of targets in the scene to be recognized; a recognition head network of the multi-task target recognition model comprises a target detection module and a semantic segmentation module, and the target detection module and the semantic segmentation module are respectively used for processing different types of target recognition tasks. A target detection module and a semantic segmentation module are arranged in an identification head network of a multi-task target identification model, and the target detection module and the semantic segmentation module are used for processing different types of target identification tasks respectively, equivalently, on the basis that the identification head network realizes a target detection function, a semantic segmentation function is also realized; therefore, one model can complete multi-task identification, and occupation of computing resources is reduced.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a target recognition method. Background Technology

[0002] With the continuous development of autonomous driving technology, market demand is also increasing. In addition to the driver assistance functions installed in passenger cars, there are also unmanned logistics delivery vehicles, unmanned vehicles in mining areas, and other projects.

[0003] Taking urban driving scenarios as an example, the target recognition tasks involved in autonomous driving include identifying vehicles, pedestrians, traffic lights, green belts, lane lines, etc. For these different recognition tasks, related technologies often need to use different models for target recognition, which leads to a large amount of system computing resources being consumed when performing target recognition in related technologies, affecting the response speed of the autonomous driving system. Summary of the Invention

[0004] Therefore, it is necessary to provide a target recognition method that reduces the system's computational resource consumption during target recognition, addressing the aforementioned technical problems.

[0005] In a first aspect, embodiments of this application provide a target recognition method, including:

[0006] Acquire environmental images of the scene to be identified; the environmental images include at least one type of target recognition task;

[0007] The environmental image is input into a pre-trained multi-task target recognition model to obtain the recognition results of various types of targets in the scene to be recognized. The recognition head network of the multi-task target recognition model includes a target detection module and a semantic segmentation module, which are used to process different types of target recognition tasks.

[0008] The target recognition method provided in this application acquires an environmental image of the scene to be recognized, and then inputs the environmental image into a pre-trained multi-task target recognition model to obtain the recognition results of various types of targets in the scene to be recognized. The environmental image includes at least one type of target recognition task. The recognition head network of the multi-task target recognition model includes a target detection module and a semantic segmentation module, which are used to process different types of target recognition tasks, respectively. In this method, by setting a target detection module and a semantic segmentation module in the recognition head network of the multi-task target recognition model, and having the target detection model and semantic segmentation module process different types of target recognition tasks respectively, multiple types of target recognition tasks can be completed by a single model during multi-task target recognition. That is, the multi-task target recognition model processes the environmental image of the scene to be recognized and directly outputs the recognition results of various types of targets in the scene to be recognized. Essentially, on the basis of the target detection function implemented in the recognition head network, semantic segmentation function is also implemented, enabling one model to complete the recognition of multiple tasks and reducing the consumption of computing resources.

[0009] In one embodiment, the multi-task target recognition model further includes a backbone network, which is connected to a feature extraction and fusion module in the recognition head network. The feature extraction and fusion module is connected to the target detection module and the semantic segmentation module, respectively.

[0010] The backbone network is used to extract features from environmental images to obtain multi-scale feature maps;

[0011] The feature extraction and fusion module is used to perform feature extraction and fusion processing on multi-scale feature maps to obtain a fused feature map.

[0012] The object detection module and the semantic segmentation module are used to process corresponding types of object recognition tasks based on fused feature maps, respectively.

[0013] The target recognition method provided in this application includes a multi-task target recognition model that further comprises a backbone network. The backbone network is connected to a feature extraction and fusion module in the recognition head network. This feature extraction and fusion module is connected to both a target detection module and a semantic segmentation module. The backbone network is used to extract features from the environmental image to obtain multi-scale feature maps. The feature extraction and fusion module is used to perform feature extraction and fusion processing on the multi-scale feature maps to obtain fused feature maps. The target detection module and the semantic segmentation module are used to process corresponding types of target recognition tasks based on the fused feature maps. In this method, by setting a backbone network in the multi-task target recognition model, feature maps can be obtained by extracting features from the environmental image. By setting a feature extraction and fusion module in the recognition head network, multi-level feature extraction and fusion of the feature maps can be performed, which helps improve the recognition accuracy of the multi-task target recognition model.

[0014] In one embodiment, the semantic segmentation module includes a semantic segmentation unit; the semantic segmentation unit includes a convolutional layer and a confidence output layer and a category output layer connected to the convolutional layer;

[0015] Convolutional layers are used to extract features from the fused feature map to obtain an enhanced feature map;

[0016] The confidence output layer is used to predict whether each pixel in the enhanced feature map belongs to the foreground or the background.

[0017] The category output layer is used to predict the category of foreground pixels in the enhanced feature map.

[0018] In the target recognition method provided in this application embodiment, the semantic segmentation module includes a semantic segmentation unit; the semantic segmentation unit includes a convolutional layer and a confidence output layer and a category output layer connected to the convolutional layer; the convolutional layer is used to extract features from the fused feature map to obtain an enhanced feature map; the confidence output layer is used to predict whether each pixel in the enhanced feature map belongs to the foreground or background; the category output layer is used to predict the category of the foreground pixels in the enhanced feature map. In this method, by setting a semantic segmentation unit in the semantic segmentation module, the convolutional layer in the semantic segmentation unit can continue to extract features from the fused feature map to obtain an enhanced feature map, the confidence output layer can predict whether each pixel in the enhanced feature map belongs to the foreground or background, and the category output layer can predict the category of the foreground pixels in the enhanced feature map, thus realizing a type of target recognition task.

[0019] In one embodiment, the semantic segmentation module further includes an optimized semantic segmentation unit; the optimized semantic segmentation unit includes a convolutional layer and a confidence output layer, an offset output layer, and a category output layer connected to the convolutional layer;

[0020] Convolutional layers are used to extract features from the fused feature map to obtain an enhanced feature map;

[0021] The confidence output layer is used to predict whether each pixel in the enhanced feature map belongs to the foreground or the background.

[0022] The offset output layer is used to predict the offset when the enhanced feature map is reconstructed into the environment image;

[0023] The category output layer is used to predict the category of foreground pixels in the enhanced feature map.

[0024] In the target recognition method provided in this application embodiment, the semantic segmentation module further includes an optimized semantic segmentation unit. The optimized semantic segmentation unit includes a convolutional layer and a confidence output layer, an offset output layer, and a category output layer connected to the convolutional layer. The convolutional layer is used to extract features from the fused feature map to obtain an enhanced feature map. The confidence output layer is used to predict whether each pixel in the enhanced feature map belongs to the foreground or background. The offset output layer is used to predict the offset when the enhanced feature map is restored to the environmental image. The category output layer is used to predict the category of the foreground pixels in the enhanced feature map. In this method, by setting an optimized semantic segmentation unit in the semantic segmentation module, the enhanced feature map can be obtained by further extracting features from the fused feature map through the convolutional layer in the optimized semantic segmentation unit. The confidence output layer predicts whether each pixel in the enhanced feature map belongs to the foreground or background, the offset output layer predicts the offset when the enhanced feature map is restored to the environmental image, and the category output layer predicts the category of the foreground pixels in the enhanced feature map, thus achieving another type of target recognition task.

[0025] In one embodiment, the construction process of the multi-task target recognition model includes:

[0026] Obtain multiple sample environment images and the label information for each sample environment image;

[0027] Using environmental images and label information from each sample, the model to be trained is trained to obtain a multi-task target recognition model.

[0028] The target recognition method provided in this application acquires multiple sample environment images and their label information, and then uses these images and label information to train a training model to obtain a multi-task target recognition model. This method provides an optional approach for rapidly constructing a multi-task target recognition model; by acquiring multiple sample environment images and their label information as training data, and then using this training data to train the training model, a multi-task target recognition model is obtained, laying the foundation for target recognition of environment images in the scene to be recognized.

[0029] In one embodiment, the model to be trained includes a recognition head network; the model is trained using each sample environment image and each label information to obtain a multi-task target recognition model, including:

[0030] The training loss is determined based on the environmental images of each sample, the label information, and the recognition head network.

[0031] Based on the training loss, the model parameters of the model to be trained are adjusted until training is completed, resulting in a multi-task target recognition model.

[0032] In the target recognition method provided in this application, the model to be trained includes a recognition head network. Based on the environmental images of each sample, the label information, and the recognition head network, a training loss is determined. Then, based on the training loss, the model parameters of the model to be trained are adjusted until training is complete, resulting in a multi-task target recognition model. In this method, a loss value is first calculated based on the environmental images of each sample, the label information, and the recognition head network as the training loss. Then, the model parameters of the model to be trained are continuously adjusted based on the training loss until training is completed, resulting in a multi-task target recognition model. The training loss is used to optimize the parameters of the model to be trained, thereby improving the model's predictive performance and accuracy.

[0033] In one embodiment, the model to be trained further includes a backbone network; the recognition head network includes a feature extraction and fusion module, an object detection module, and a semantic segmentation module; the training loss is determined based on the environmental images of each sample, the label information, and the recognition head network, including:

[0034] The environmental images of each sample are input into the backbone network to obtain multi-scale feature maps;

[0035] The multi-scale feature map is input into the feature extraction and fusion module for upsampling and downsampling to obtain the intermediate feature map;

[0036] The training loss is determined based on the intermediate feature map, the label information, and the object detection and semantic segmentation modules.

[0037] In the target recognition method provided in this application embodiment, the model to be trained further includes a backbone network; the recognition head network includes a feature extraction and fusion module, a target detection module, and a semantic segmentation module; each sample environment image is input into the backbone network to obtain a multi-scale feature map, and then the multi-scale feature map is input into the feature extraction and fusion module for upsampling and downsampling to obtain an intermediate feature map. Then, based on the intermediate feature map, each label information, and the target detection and semantic segmentation modules, the training loss is determined. This method provides an optional way to quickly determine the training loss; the model to be trained includes a backbone model and a recognition head network, the recognition head network including a feature extraction and fusion module, a target detection module, and a semantic segmentation module. Each sample environment image is input into the backbone network for feature extraction to obtain a multi-scale feature map, and then the multi-scale feature map is input into the feature extraction and fusion module for further feature extraction and fusion processing to obtain an intermediate feature map. The intermediate feature map is input into the target detection module and the semantic segmentation module respectively, and the information obtained is used to calculate the training loss, providing data support for the training of a multi-task target recognition model.

[0038] In one embodiment, the sample environment image is an environment image in a traffic scene; the semantic segmentation module includes a semantic segmentation unit and an optimized semantic segmentation unit; the training loss is determined based on the intermediate feature map, each label information, and the object detection module and the semantic segmentation module, including:

[0039] The intermediate feature map is input into the target detection module to obtain dynamic target prediction information, and into the semantic segmentation unit to obtain static target prediction information. The intermediate feature map is also input into the optimized semantic segmentation unit to obtain lane marking prediction information.

[0040] The training loss is determined based on dynamic target prediction information, static target prediction information, lane marking prediction information, and label information.

[0041] In the target recognition method provided in this application embodiment, the sample environment image is an environment image under traffic scene; the semantic segmentation module includes a semantic segmentation unit and an optimized semantic segmentation unit; the intermediate feature map is input into the target detection module to obtain dynamic target prediction information, and the intermediate feature map is input into the semantic segmentation unit to obtain static target prediction information, and the intermediate feature map is input into the optimized semantic segmentation unit to obtain lane marking prediction information. Then, the training loss is determined based on the dynamic target prediction information, static target prediction information, lane marking prediction information, and each label information. In this method, another optional way to quickly determine the training loss is provided; by inputting the intermediate feature map into the semantic segmentation unit and the optimized semantic segmentation unit of the target detection module and the semantic segmentation module respectively to perform different types of target recognition tasks, dynamic target prediction information, static target prediction information, and lane marking prediction information are obtained respectively. Based on the combination of dynamic target prediction information, static target prediction information, lane marking prediction information, and label information, the training loss can be calculated, laying the foundation for adjusting the parameters of the model to be trained.

[0042] In one embodiment, the training loss is determined based on dynamic target prediction information, static target prediction information, lane marking prediction information, and various label information, including:

[0043] The dynamic target loss value is determined based on the dynamic target prediction information, each label information, and the preset loss function corresponding to the dynamic target prediction information; the static target loss value is determined based on the static target prediction information, each label information, and the preset loss function corresponding to the static target prediction information; and the lane marking loss value is determined based on the lane marking prediction information, each label information, and the preset loss function corresponding to the lane marking prediction information.

[0044] The training loss is determined based on the dynamic target loss value, the static target loss value, and the lane marking loss value.

[0045] In the target recognition method provided in this application embodiment, the sample environment image is an environment image under traffic scene; the semantic segmentation module includes a semantic segmentation unit and an optimized semantic segmentation unit; the intermediate feature map is input into the target detection module to obtain dynamic target prediction information, and the intermediate feature map is input into the semantic segmentation unit to obtain static target prediction information, and the intermediate feature map is input into the optimized semantic segmentation unit to obtain lane marking prediction information. Then, the training loss is determined based on the dynamic target prediction information, static target prediction information, lane marking prediction information, and each label information. In this method, another optional way to quickly determine the training loss is provided; by inputting the intermediate feature map into the semantic segmentation unit and the optimized semantic segmentation unit of the target detection module and the semantic segmentation module respectively to perform different types of target recognition tasks, dynamic target prediction information, static target prediction information, and lane marking prediction information are obtained respectively. Based on the combination of dynamic target prediction information, static target prediction information, lane marking prediction information, and label information, the training loss can be calculated, laying the foundation for adjusting the parameters of the model to be trained.

[0046] In one embodiment, the training loss is determined based on the dynamic target loss value, the static target loss value, and the lane marking loss value, including:

[0047] Obtain the detection weights of the object detection module, the segmentation weights of the semantic segmentation unit, and the optimized segmentation weights of the optimized semantic segmentation unit;

[0048] The training loss is determined based on the detection weight, segmentation weight, optimized segmentation weight, dynamic target loss value, static target loss value, and lane marking loss value.

[0049] In the target recognition method provided in this application, a dynamic target loss value is determined based on dynamic target prediction information, each label information, and a preset loss function corresponding to the dynamic target prediction information; a static target loss value is determined based on static target prediction information, each label information, and a preset loss function corresponding to the static target prediction information; a lane marking loss value is determined based on lane marking prediction information, each label information, and a preset loss function corresponding to the lane marking prediction information; and then, the training loss is determined based on the dynamic target loss value, the static target loss value, and the lane marking loss value. In this method, by using multiple preset loss functions and combining the corresponding prediction information and the corresponding label information, the corresponding loss value can be calculated. Furthermore, by combining multiple loss values, the training loss can be obtained accurately and quickly, providing another optional method for rapidly determining the training loss.

[0050] In one embodiment, the method further includes:

[0051] Obtain the test dataset; the test dataset includes multiple test environment images and test label information for each test environment image;

[0052] The test environment images are input into the trained model to obtain the dynamic target prediction results, static target prediction results and lane marking prediction results output by the trained model.

[0053] Based on the dynamic target prediction results, static target prediction results, lane marking prediction results, and various test label information, the trained model is tested to obtain a multi-task target recognition model.

[0054] The target recognition method provided in this application obtains the detection weights of the target detection module, the segmentation weights of the semantic segmentation unit, and the optimized segmentation weights of the optimized semantic segmentation unit. Then, based on the detection weights, segmentation weights, optimized segmentation weights, dynamic target loss values, static target loss values, and lane marking loss values, a training loss is determined. This method, by introducing the detection weights of the target detection module, the segmentation weights of the semantic segmentation unit, and the optimized segmentation weights of the optimized semantic segmentation unit, comprehensively calculates each weight with its corresponding loss value to obtain the training loss. Training the model under test using this training loss allows the model to converge faster, improving the effect and accuracy of target recognition.

[0055] Secondly, embodiments of this application also provide a target recognition device, comprising:

[0056] The image acquisition module is used to acquire environmental images of the scene to be identified; the environmental images include at least one type of target recognition task;

[0057] The target recognition module is used to input environmental images into a pre-trained multi-task target recognition model to obtain the recognition results of various types of targets in the scene to be recognized. The recognition head network of the multi-task target recognition model includes a target detection module and a semantic segmentation module, which are used to handle different types of target recognition tasks.

[0058] Thirdly, embodiments of this application also provide a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in any of the embodiments of the first aspect described above.

[0059] Fourthly, embodiments of this application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps in any of the embodiments of the first aspect described above.

[0060] Fifthly, embodiments of this application also provide a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps in any of the embodiments of the first aspect described above.

[0061] The aforementioned target recognition method acquires an environmental image of the scene to be recognized, and then inputs the environmental image into a pre-trained multi-task target recognition model to obtain the recognition results of various types of targets in the scene. The environmental image includes at least one type of target recognition task. The recognition head network of the multi-task target recognition model includes a target detection module and a semantic segmentation module, which are used to handle different types of target recognition tasks, respectively. In this method, by setting up target detection and semantic segmentation modules in the recognition head network of the multi-task target recognition model, and having these modules handle different types of target recognition tasks respectively, multiple types of target recognition tasks can be completed with a single model during multi-task target recognition. That is, the multi-task target recognition model processes the environmental image of the scene to be recognized and directly outputs the recognition results of various types of targets in the scene. Essentially, in addition to the target detection function implemented in the recognition head network, semantic segmentation is also implemented, allowing one model to complete the recognition of multiple tasks and reducing the consumption of computational resources. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a diagram illustrating the application environment of the target recognition method in one embodiment;

[0064] Figure 2 This is a flowchart illustrating a target recognition method in one embodiment;

[0065] Figure 3 This is a schematic diagram of the structure of the recognition head network in one embodiment;

[0066] Figure 4 This is a schematic diagram of the structure of a convolutional layer in one embodiment;

[0067] Figure 5 This is a schematic diagram of the feature fusion layer in one embodiment;

[0068] Figure 6 This is a schematic diagram of the structure of a semantic segmentation unit in one embodiment;

[0069] Figure 7 This is a schematic diagram of the structure of an optimized semantic segmentation unit in one embodiment;

[0070] Figure 8 This is a flowchart illustrating the process of constructing a multi-task target recognition model in one embodiment;

[0071] Figure 9 This is a flowchart illustrating the process of training a multi-task target recognition model in one embodiment;

[0072] Figure 10 This is a schematic diagram of the process for determining the training loss in one embodiment;

[0073] Figure 11 This is a flowchart illustrating the process of determining the training loss in another embodiment;

[0074] Figure 12 This is a flowchart illustrating the process of determining the training loss in another embodiment;

[0075] Figure 13 This is a flowchart illustrating the process of determining the training loss in another embodiment;

[0076] Figure 14 This is a flowchart illustrating the testing model in one embodiment;

[0077] Figure 15 This is a flowchart illustrating the target recognition method in another embodiment;

[0078] Figure 16 This is a schematic diagram of the target recognition device in one embodiment;

[0079] Figure 17 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0081] The target recognition method provided in this application embodiment can be applied to, for example, Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Optionally, terminal 102 uploads environmental images of the scene to be identified to server 104 for target recognition. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0082] The technical background of this application is described below.

[0083] With the continuous development of autonomous driving technology, market demand is also increasing. In addition to the driver assistance functions installed in passenger cars, there are also unmanned logistics delivery vehicles, unmanned vehicles in mining areas, and other projects.

[0084] In autonomous driving technology, target recognition is a crucial step. It involves using various sensors to collect relevant information about the surrounding environment, including roads, pedestrians, and traffic signs, providing valuable information for planning the vehicle's movement path. Currently, commonly used sensors include LiDAR and cameras. Cameras, due to their superior ability to distinguish the color and shape of target objects, are primarily responsible for recognizing road elements such as traffic lights, traffic signs, and lane markings.

[0085] The camera primarily uses deep learning-based object detection / semantic segmentation methods to identify different targets. Object detection, represented by the YOLO series, directly and densely predicts the target's category and bounding box on the input image, commonly used to identify targets such as vehicles and pedestrians. Semantic segmentation, represented by the DeepLab series, provides a category for each pixel in the input image, commonly used to identify targets such as roads, green belts, and lane lines. For different recognition tasks, related technologies often use different models for recognition, and these models are loaded into the system to achieve the purpose of object recognition.

[0086] However, with the development of autonomous driving technology, the requirements for target recognition are also increasing. Following existing solutions, target detection models and semantic segmentation modules are needed, and more recognition models, such as keypoint detection, may be introduced later. As the number of models increases, the computational resources required for the entire system increase, thus affecting the system's response speed. Furthermore, each new model requires data collection, annotation, and training, which are cumbersome and time-consuming. Therefore, how to lightweight the models or constrain the number of models without affecting the final recognition performance is of great significance for target recognition in autonomous driving technology.

[0087] Furthermore, using a semantic segmentation module for lane line recognition results in unsatisfactory performance. This is because the target category occupies a relatively small proportion of pixels, leading to class imbalance and affecting recognition accuracy. Therefore, to address this issue, the semantic segmentation module needs improvement and optimization to enhance recognition performance.

[0088] Based on this, this application provides a target recognition method. By setting a target detection module and a semantic segmentation module in the recognition head network of a multi-task target recognition model, the target detection model and the semantic segmentation module respectively handle different types of target recognition tasks. In this way, when performing multi-task target recognition, multiple types of target recognition tasks can be completed by a single model. That is, the multi-task target recognition model processes the environmental image of the scene to be recognized and directly outputs the recognition results of each type of target in the scene to be recognized. It is equivalent to realizing the semantic segmentation function on the basis of the target detection function of the recognition head network, so that a single model can complete the recognition of multiple tasks and reduce the consumption of computing resources.

[0089] In one exemplary embodiment, such as Figure 2 As shown, a target recognition method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 202. Wherein:

[0090] S201, acquire an environmental image of the scene to be identified; the environmental image includes at least one type of target recognition task.

[0091] Environmental images refer to images of the surrounding environment captured by autonomous vehicles in the scene to be identified. These scenes can include urban driving scenarios, closed-park driving scenarios, and rural road driving scenarios. Target recognition in urban driving scenarios is complex, including identifying pedestrians, bicycles, cars, buses, taxis, traffic lights, traffic signs, and handling emergencies such as pedestrians suddenly entering the lane. In closed-park driving scenarios, autonomous vehicles need to identify buildings, road signs, pedestrians, and other vehicles within the park to ensure safe navigation within a confined space. In rural road driving scenarios, autonomous vehicles need to identify small animals, pedestrians, agricultural vehicles, and be aware of potential potholes and obstacles.

[0092] The environmental image includes at least one type of target recognition task. Taking an urban driving scene as an example, different types of targets include motor vehicles, pedestrians, traffic lights, traffic signs, roads, crosswalks, green belts, lane lines, etc. The recognition of targets such as motor vehicles, pedestrians, traffic lights, and traffic signs can be one type of recognition task; the recognition of targets such as roads, crosswalks, and green belts can be another type of recognition task; and the recognition of targets such as lane lines can be yet another type of recognition task. Alternatively, the recognition of targets such as motor vehicles and pedestrians can be one type of recognition task, for example, it could refer to the recognition of dynamic targets; the recognition of targets such as traffic lights, traffic signs, roads, crosswalks, and green belts can be one type of recognition task, for example, it could refer to the recognition of static targets; and the recognition of targets such as lane lines can be one type of recognition task, for example, it could refer to the recognition of lane markings. This application does not limit this aspect.

[0093] Various types of target recognition tasks include the recognition of motor vehicles, pedestrians, traffic lights, traffic signs, roads, pedestrian crossings, green belts, lane lines, etc.

[0094] In one embodiment, the environmental image is acquired in real time by the autonomous vehicle. For example, image data of the environment surrounding the vehicle can be captured by an onboard camera as the environmental image. Typically, to obtain high-quality environmental images, autonomous driving systems are equipped with high-resolution cameras capable of capturing visual information about the vehicle's surroundings. Through these cameras, the system can obtain image data of different road types, traffic scenarios, and lighting conditions.

[0095] In another embodiment, the environmental image is obtained from a database. For example, the image data of the surrounding environment collected by the autonomous vehicle in different scenarios are stored in the database. When an environmental image is needed, the corresponding image data can be matched from the database based on the scene identifier information of the scene to be identified, and the image data can be identified as the environmental image of the scene to be identified.

[0096] S202, the environmental image is input into the pre-trained multi-task target recognition model to obtain the recognition results of various types of targets in the scene to be recognized; the recognition head network of the multi-task target recognition model includes a target detection module and a semantic segmentation module, which are used to process different types of target recognition tasks respectively.

[0097] The multi-task target recognition model is obtained by training the model to be trained using the training dataset. The model to be trained also includes a recognition head network, which comprises an target detection module and a semantic segmentation module. The target detection module and the semantic segmentation module are used to handle the recognition tasks of different types of targets, respectively.

[0098] A head network typically consists of a series of convolutional layers used to extract useful information from the feature maps processed by the backbone network. This information includes the object's category, location, and possibly other attributes. The recognition head network in this embodiment is the head network of the multi-task object recognition model, which includes an object detection module and a semantic segmentation module. The object detection module detects objects in the image and determines the location (usually represented by bounding boxes) and category of each object. The semantic segmentation module classifies each pixel in the image into a predefined category, such as people, vehicles, and buildings.

[0099] In this embodiment, the object detection module and the semantic segmentation module are used to handle different types of object recognition tasks. For example, the object detection module is used to handle objects such as motor vehicles, pedestrians, traffic lights, and traffic signs; the semantic segmentation module is used to handle objects such as roads, pedestrian crossings, green belts, and lane lines. Alternatively, the object detection module is used to handle objects such as motor vehicles and pedestrians; the semantic segmentation module is used to handle objects such as traffic lights, traffic signs, roads, pedestrian crossings, green belts, and lane lines.

[0100] Typically, object recognition networks include a head network and a backbone network, which work in conjunction with the head network. Thus, multi-task object recognition models and the models to be trained both include a recognition head network and a backbone network. The backbone network is used to extract features from the raw input data to obtain feature maps.

[0101] Training a multi-task target recognition model can begin by acquiring a training dataset, which consists of multiple historical environment images of the scene to be recognized and their annotation information. The training dataset is then input into a backbone network, which extracts features from each historical environment image to obtain feature maps. These feature maps are then input into a recognition head network, which is responsible for specific target detection and recognition tasks and outputs the category information for each target. A loss value is then calculated based on the category information of each target. The model parameters of the model under training are continuously adjusted based on the loss value until the loss value is less than a preset threshold, at which point training stops, resulting in the multi-task target recognition model.

[0102] After training the multi-task target recognition model, when it is necessary to identify different types of targets in the environmental image of the scene to be identified, the environmental image of the scene to be identified can be input into the trained multi-task target recognition model. The multi-task target recognition model performs feature extraction and target detection on the environmental image and obtains the recognition results of each type of target output by the multi-task target recognition model, that is, the recognition results of each type of target in the scene to be identified.

[0103] The target recognition method provided in this application acquires an environmental image of the scene to be recognized, and then inputs the environmental image into a pre-trained multi-task target recognition model to obtain the recognition results of various types of targets in the scene to be recognized. The environmental image includes at least one type of target recognition task. The recognition head network of the multi-task target recognition model includes a target detection module and a semantic segmentation module, which are used to process different types of target recognition tasks, respectively. In this method, by setting a target detection module and a semantic segmentation module in the recognition head network of the multi-task target recognition model, and having the target detection model and semantic segmentation module process different types of target recognition tasks respectively, multiple types of target recognition tasks can be completed by a single model during multi-task target recognition. That is, the multi-task target recognition model processes the environmental image of the scene to be recognized and directly outputs the recognition results of various types of targets in the scene to be recognized. Essentially, on the basis of the target detection function implemented in the recognition head network, semantic segmentation function is also implemented, enabling one model to complete the recognition of multiple tasks and reducing the consumption of computing resources.

[0104] Based on the above embodiments, an embodiment of the structure of the above multi-task recognition model is provided for illustration.

[0105] In an exemplary embodiment, the multi-task target recognition model further includes a backbone network, which is connected to the feature extraction and fusion module in the recognition head network. The feature extraction and fusion module is connected to the target detection module and the semantic segmentation module, respectively.

[0106] The backbone network is used to extract features from environmental images to obtain multi-scale feature maps;

[0107] The feature extraction and fusion module is used to perform feature extraction and fusion processing on multi-scale feature maps to obtain a fused feature map.

[0108] The object detection module and the semantic segmentation module are used to process corresponding types of object recognition tasks based on fused feature maps, respectively.

[0109] The backbone network transforms input data into high-dimensional feature vectors through multi-layer convolution and pooling operations, providing effective feature representations for subsequent task modules. When performing target recognition on environmental images in the scene to be recognized using a multi-task target recognition model, the environmental image is input into the model, and the backbone network first extracts features from the environmental image to obtain feature maps at different scales, i.e., multi-scale feature maps.

[0110] In this embodiment, the recognition head network also includes a feature extraction and fusion module, which is connected to both the object detection module and the semantic segmentation module. The feature extraction and fusion module is used to perform feature extraction and fusion processing on multi-scale feature maps to obtain a fused feature map.

[0111] After the environmental image undergoes feature extraction operations in the backbone network to obtain multi-scale feature maps, these maps are input into the recognition head network. First, the feature extraction and fusion module in the recognition head network performs feature extraction and fusion processing on the multi-scale feature maps to obtain a fused feature map. This feature fusion processing can include upsampling and downsampling operations. The feature extraction and fusion module can include multiple convolutional layers, activation function layers, and feature fusion layers. The main purpose of feature extraction and fusion processing is to integrate feature information from different scales or levels, which may contain location, detail, semantic, and other information. This approach improves the model's recognition accuracy and robustness.

[0112] The object detection module and the semantic segmentation module are used to process corresponding types of object recognition tasks based on fused feature maps, respectively.

[0113] After the multi-scale feature maps are processed by the feature extraction and fusion module to obtain a fused feature map, they are input into the object detection module and the semantic segmentation module, respectively. The object detection module and the semantic segmentation module then process the corresponding type of object recognition task based on the fused feature map, finally outputting recognition results for various types of objects. Specifically, the object detection module can recognize objects such as motor vehicles, pedestrians, traffic lights, and traffic signs, while the semantic segmentation module can recognize objects such as roads, pedestrian crossings, green belts, and lane lines.

[0114] In this embodiment of the application, a schematic diagram of the structure of a recognition head network is provided, such as... Figure 3 As shown in the diagram; where ① is the feature extraction and fusion module, ② is the object detection module, and ③ is the semantic segmentation module. A schematic diagram of the convolutional layer structure in the feature extraction and fusion module is shown below. Figure 4 As shown; the batch normalization layer performs batch normalization on the feature maps output by the 2D convolutional layer, making them conform to a normalized distribution with a mean of 0 and a variance of 1, thereby accelerating the convergence speed of the network; the activation function layer has a smoother curve when close to zero, and because it uses the sigmoid function, it can make the network output range between 0 and 1. The formula for the activation function is: A schematic diagram of the feature fusion layer in the feature extraction and fusion module is shown below. Figure 5 As shown, the feature fusion layer is composed of multiple stacked bottleneck layers. These bottleneck layers borrow the idea of ​​residual structures, using 3x3 convolutional kernels for feature dimensionality reduction to extract redundant information. Then, 3x3 convolutional kernels are used for feature dimensionality upscaling to restore the original dimensions. The stacking of multiple bottleneck layers, along with skip connections and data segmentation operations, allows the feature fusion layer to obtain richer gradient flow information.

[0115] The target recognition method provided in this application includes a multi-task target recognition model that further comprises a backbone network. The backbone network is connected to a feature extraction and fusion module in the recognition head network. This feature extraction and fusion module is connected to both a target detection module and a semantic segmentation module. The backbone network is used to extract features from the environmental image to obtain multi-scale feature maps. The feature extraction and fusion module is used to perform feature extraction and fusion processing on the multi-scale feature maps to obtain fused feature maps. The target detection module and the semantic segmentation module are used to process corresponding types of target recognition tasks based on the fused feature maps. In this method, by setting a backbone network in the multi-task target recognition model, feature maps can be obtained by extracting features from the environmental image. By setting a feature extraction and fusion module in the recognition head network, multi-level feature extraction and fusion of the feature maps can be performed, which helps improve the recognition accuracy of the multi-task target recognition model.

[0116] Based on the above embodiments, an embodiment of the structure of the semantic segmentation module is provided for illustration.

[0117] In an exemplary embodiment, the semantic segmentation module includes a semantic segmentation unit; the semantic segmentation unit includes a convolutional layer and a confidence output layer and a category output layer connected to the convolutional layer;

[0118] Convolutional layers are used to extract features from the fused feature map to obtain an enhanced feature map;

[0119] The confidence output layer is used to predict whether each pixel in the enhanced feature map belongs to the foreground or the background.

[0120] The category output layer is used to predict the category of foreground pixels in the enhanced feature map.

[0121] In this embodiment of the application, the semantic segmentation module includes a semantic segmentation unit, which includes a convolutional layer and a confidence output layer and a category output layer connected to the convolutional layer, such as... Figure 6 As shown, the convolutional layer includes two cascaded standard convolutional layers and two two-dimensional convolutional layers connected to the second standard convolutional layer.

[0122] When the fused feature map is input into the semantic segmentation module, it first passes through the convolutional layer of the semantic segmentation unit, where features are extracted by the standard convolutional layer. Then, it is input into two two-dimensional convolutional layers for further feature extraction, ultimately yielding an enhanced feature map. This enhanced feature map contains richer and deeper feature information.

[0123] The enhanced feature maps obtained after two 2D convolutional layers are input into the confidence output layer and the category output layer, respectively. The confidence output layer predicts whether each pixel in the enhanced feature map belongs to the foreground or the background, and the category output layer predicts the category of the pixels in the enhanced feature map that belong to the foreground.

[0124] In the target recognition method provided in this application embodiment, the semantic segmentation module includes a semantic segmentation unit; the semantic segmentation unit includes a convolutional layer and a confidence output layer and a category output layer connected to the convolutional layer; the convolutional layer is used to extract features from the fused feature map to obtain an enhanced feature map; the confidence output layer is used to predict whether each pixel in the enhanced feature map belongs to the foreground or background; the category output layer is used to predict the category of the foreground pixels in the enhanced feature map. In this method, by setting a semantic segmentation unit in the semantic segmentation module, the convolutional layer in the semantic segmentation unit can continue to extract features from the fused feature map to obtain an enhanced feature map, the confidence output layer can predict whether each pixel in the enhanced feature map belongs to the foreground or background, and the category output layer can predict the category of the foreground pixels in the enhanced feature map, thus realizing a type of target recognition task.

[0125] Based on the above embodiments, another embodiment of the structure of the semantic segmentation module is described below.

[0126] In an exemplary embodiment, the semantic segmentation module further includes an optimized semantic segmentation unit; the optimized semantic segmentation unit includes a convolutional layer and a confidence output layer, an offset output layer, and a category output layer connected to the convolutional layer;

[0127] Convolutional layers are used to extract features from the fused feature map to obtain an enhanced feature map;

[0128] The confidence output layer is used to predict whether each pixel in the enhanced feature map belongs to the foreground or the background.

[0129] The offset output layer is used to predict the offset when the enhanced feature map is reconstructed into the environment image;

[0130] The category output layer is used to predict the category of foreground pixels in the enhanced feature map.

[0131] In this embodiment, the semantic segmentation module includes not only a semantic segmentation unit but also an optimized semantic segmentation unit. The optimized semantic segmentation unit includes a convolutional layer and a confidence output layer, an offset output layer, and a category output layer connected to the convolutional layer, such as... Figure 7 As shown, the convolutional layer includes two cascaded standard convolutional layers and three two-dimensional convolutional layers connected to the second standard convolutional layer.

[0132] When the fused feature map is input into the semantic segmentation module, it enters the semantic segmentation unit and the optimized semantic segmentation unit respectively. In the optimized semantic segmentation unit, it first passes through the convolutional layer of the optimized semantic segmentation unit, where the standard convolutional layer performs feature extraction. Then, it is input into three two-dimensional convolutional layers for another feature extraction, finally obtaining the enhanced feature map. The enhanced feature map contains richer and deeper feature information.

[0133] The enhanced feature maps obtained after three two-dimensional convolutional layers are input into the confidence output layer and the category output layer, respectively. The confidence output layer determines whether each pixel in the semantic enhanced feature map belongs to the foreground or the background. The offset output layer predicts the offset when the enhanced feature map is restored to the environment image. The category output layer determines the category of the foreground pixels in the semantic enhanced feature map.

[0134] In this embodiment, the optimized semantic segmentation unit adds an offset output layer on the basis of the semantic segmentation unit. This is because the feature map used for calculation is downsampled, and there will be an accuracy loss when restoring it to the original image. This loss will have a significant impact on the recognition of lane line targets. The offset output layer is responsible for predicting the offset when the feature map is restored to the original image, so as to improve the accuracy of recognition.

[0135] In this embodiment, an optimized semantic segmentation unit is designed in the recognition head network, which can improve the target recognition effect. This unit is also integrated into the recognition head network without additional resource consumption.

[0136] In the target recognition method provided in this application embodiment, the semantic segmentation module further includes an optimized semantic segmentation unit. The optimized semantic segmentation unit includes a convolutional layer and a confidence output layer, an offset output layer, and a category output layer connected to the convolutional layer. The convolutional layer is used to extract features from the fused feature map to obtain an enhanced feature map. The confidence output layer is used to predict whether each pixel in the enhanced feature map belongs to the foreground or background. The offset output layer is used to predict the offset when the enhanced feature map is restored to the environmental image. The category output layer is used to predict the category of the foreground pixels in the enhanced feature map. In this method, by setting an optimized semantic segmentation unit in the semantic segmentation module, the enhanced feature map can be obtained by further extracting features from the fused feature map through the convolutional layer in the optimized semantic segmentation unit. The confidence output layer predicts whether each pixel in the enhanced feature map belongs to the foreground or background, the offset output layer predicts the offset when the enhanced feature map is restored to the environmental image, and the category output layer predicts the category of the foreground pixels in the enhanced feature map, thus achieving another type of target recognition task.

[0137] Based on any of the above embodiments, an embodiment is provided to illustrate the construction process of the above multi-task target recognition model.

[0138] In one exemplary embodiment, such as Figure 8 As shown, the construction process of the multi-task target recognition model includes:

[0139] S301, acquire multiple sample environment images and the label information of each sample environment image.

[0140] In this embodiment, the sample environment images are environmental images collected during the driving process by autonomous vehicles at historical times. For example, they can be images taken from a forward-looking perspective during autonomous driving by an unmanned vehicle. The label information is data obtained by labeling multiple sample environment images according to preset data labeling rules.

[0141] The data annotation rules mainly involve annotating a single frame image for multiple tasks. The object detection task mainly targets targets such as motor vehicles, pedestrians, traffic lights, and traffic signs, requiring the bounding box to fit the target object. The semantic segmentation task mainly targets targets such as roads, pedestrian crossings, and green belts, requiring the boundaries between different categories to be clearly labeled. The improved semantic segmentation task mainly targets lane line targets, with the same requirements as the semantic segmentation task.

[0142] For example, the collected sample environment images and the label information of each sample environment image are stored in a database. When needed, the sample environment images and the label information of each sample environment image can be retrieved from the database.

[0143] S302 uses the environmental images and label information of each sample to train the model to be trained, thus obtaining a multi-task target recognition model.

[0144] After obtaining multiple sample environment images and their label information, a dataset can be constructed. The dataset can then be divided into a training set and a test set. Based on this, image enhancement techniques can be used to preprocess the images and expand the dataset.

[0145] The training set was then used to train the model to be trained, and the test set was used to test the trained model to obtain a multi-task target recognition model.

[0146] In this process, when training the model using a training set, the environmental images and their labels from each sample in the training set can be input into the model. The model then analyzes and processes these images to obtain prediction information. This prediction information is then combined with the label information to calculate a loss value. Based on this loss value, the model parameters are adjusted until training is complete, resulting in the trained model. Finally, the trained model is tested using a test set to obtain the multi-task object recognition model.

[0147] The target recognition method provided in this application acquires multiple sample environment images and their label information, and then uses these images and label information to train a training model to obtain a multi-task target recognition model. This method provides an optional approach for rapidly constructing a multi-task target recognition model; by acquiring multiple sample environment images and their label information as training data, and then using this training data to train the training model, a multi-task target recognition model is obtained, laying the foundation for target recognition of environment images in the scene to be recognized.

[0148] Based on the above embodiments, an embodiment is provided to illustrate the process of training the model to be trained using each sample environmental image and each label information to obtain a multi-task target recognition model.

[0149] In one exemplary embodiment, such as Figure 9 As shown, the model to be trained includes a recognition head network; using environmental images of each sample and label information, the model to be trained is trained to obtain a multi-task target recognition model, including:

[0150] S401, determine the training loss based on the environmental images of each sample, the label information, and the recognition head network.

[0151] In this embodiment, the model to be trained has the same structure as the multi-task target recognition model, both consisting of a backbone network and a recognition head network.

[0152] When training the model, the environmental images of each sample and their label information can be input into the backbone network of the model to be trained to obtain feature maps. Then, the feature maps are input into the recognition head network to obtain prediction information. After that, the loss value is calculated based on the prediction information output by the recognition head network and the label information of each environmental image. This loss value is the training loss.

[0153] S402, based on the training loss, adjust the model parameters of the model to be trained until training is complete, and obtain the multi-task target recognition model.

[0154] After calculating the training loss, the model parameters of the model to be trained are adjusted based on the training loss. Then, the sample environment image is input into the adjusted model to be trained, and the prediction information at this time is output. The loss value is calculated again based on the label information and the prediction information. If the current loss value is less than the preset loss value, training is stopped and a multi-task target recognition model is obtained. If the current loss value is not less than the preset loss value, the model parameters of the model to be trained are adjusted again until the calculated training loss is less than the preset loss value, training is stopped and a multi-task target recognition model is obtained.

[0155] In the target recognition method provided in this application, the model to be trained includes a recognition head network. Based on the environmental images of each sample, the label information, and the recognition head network, a training loss is determined. Then, based on the training loss, the model parameters of the model to be trained are adjusted until training is complete, resulting in a multi-task target recognition model. In this method, a loss value is first calculated based on the environmental images of each sample, the label information, and the recognition head network as the training loss. Then, the model parameters of the model to be trained are continuously adjusted based on the training loss until training is completed, resulting in a multi-task target recognition model. The training loss is used to optimize the parameters of the model to be trained, thereby improving the model's predictive performance and accuracy.

[0156] Based on the above embodiments, an embodiment is provided to illustrate the process of determining the training loss based on each sample environmental image, each label information, and the recognition head network.

[0157] In one exemplary embodiment, such as Figure 10 As shown, the model to be trained also includes a backbone network; the recognition head network includes a feature extraction and fusion module, an object detection module, and a semantic segmentation module; the training loss is determined based on the environmental images of each sample, the label information, and the recognition head network, including:

[0158] S501 inputs the environmental images of each sample into the backbone network to obtain multi-scale feature maps.

[0159] In this embodiment of the application, the model to be trained further includes a backbone network, and the recognition head network includes a feature extraction and fusion module, an object detection module, and a semantic segmentation module.

[0160] During training, the environmental images of each sample are first input into the backbone network of the model to be trained for feature extraction, and feature maps of different scales are obtained, that is, multi-scale feature maps are obtained.

[0161] S502, input the multi-scale feature map into the feature extraction and fusion module for upsampling and downsampling to obtain the intermediate feature map.

[0162] After obtaining the multi-scale feature map, the multi-scale feature map is input into the feature extraction and fusion module of the recognition head network to perform multi-level feature extraction and fusion to obtain the intermediate feature map.

[0163] See also Figure 3 First, the small-scale feature map (input 1) undergoes an upsampling operation via an upsampling module. Then, it is merged with the large-scale feature map (input 2) via a channel concatenation module. After passing through a feature fusion module and another upsampling module, it is merged with an even larger-scale feature map (input 3) to obtain the large-scale output feature map (output 1). During this process, the high resolution of the large-scale features and the semantic information of the small-scale features are fused. Next, a reverse downward flow (downsampling) is performed for further fusion, resulting in small-scale output feature maps (output 2, output 3). These output feature maps are the intermediate feature maps obtained after the upsampling and downsampling operations.

[0164] S503 determines the training loss based on the intermediate feature map, label information, and the object detection and semantic segmentation modules.

[0165] After obtaining the intermediate feature map, the intermediate feature map is input into the object detection module and the semantic segmentation module respectively to obtain the prediction information output by the object detection module and the semantic segmentation module respectively. Then, the two prediction information are used to calculate the loss with each label information to obtain two loss values. The two loss values ​​are combined to obtain the comprehensive loss value, which is the training loss.

[0166] In the target recognition method provided in this application embodiment, the model to be trained further includes a backbone network; the recognition head network includes a feature extraction and fusion module, a target detection module, and a semantic segmentation module; each sample environment image is input into the backbone network to obtain a multi-scale feature map, and then the multi-scale feature map is input into the feature extraction and fusion module for upsampling and downsampling to obtain an intermediate feature map. Then, based on the intermediate feature map, each label information, and the target detection and semantic segmentation modules, the training loss is determined. This method provides an optional way to quickly determine the training loss; the model to be trained includes a backbone model and a recognition head network, the recognition head network including a feature extraction and fusion module, a target detection module, and a semantic segmentation module. Each sample environment image is input into the backbone network for feature extraction to obtain a multi-scale feature map, and then the multi-scale feature map is input into the feature extraction and fusion module for further feature extraction and fusion processing to obtain an intermediate feature map. The intermediate feature map is input into the target detection module and the semantic segmentation module respectively, and the information obtained is used to calculate the training loss, providing data support for the training of a multi-task target recognition model.

[0167] Based on the above embodiments, an embodiment is provided to illustrate the process of determining the training loss based on the intermediate feature map, each label information, and the object detection module and semantic segmentation module.

[0168] In one exemplary embodiment, such as Figure 11 As shown, the sample environment image is an environment image in a traffic scene; the semantic segmentation module includes a semantic segmentation unit and an optimized semantic segmentation unit; based on the intermediate feature map, each label information, and the object detection module and semantic segmentation module, the training loss is determined, including:

[0169] S601, the intermediate feature map is input into the target detection module to obtain dynamic target prediction information, and the intermediate feature map is input into the semantic segmentation unit to obtain static target prediction information, and the intermediate feature map is input into the optimized semantic segmentation unit to obtain lane marking prediction information.

[0170] In this embodiment, the sample environment image is an environment image under traffic scene. The environment image may contain targets such as motor vehicles, pedestrians, traffic lights, traffic signs, roads, crosswalks, green belts, and lane lines. Motor vehicles and pedestrians can be dynamic targets, while traffic lights, traffic signs, roads, crosswalks, and green belts can be static targets. Lane lines can be lane marking targets. Dynamic target prediction information refers to the prediction results for motor vehicles and pedestrians in the sample environment image; static target prediction information refers to the prediction results for traffic lights, traffic signs, roads, crosswalks, and green belts in the sample environment image; and lane marking prediction information refers to the prediction results for lane lines in the sample environment image.

[0171] After obtaining the intermediate feature map, it is input into the target detection module, which can then identify targets such as motor vehicles and pedestrians to obtain dynamic target prediction information. The intermediate feature map is also input into the semantic segmentation unit of the semantic segmentation module, which can then identify targets such as traffic lights, traffic signs, roads, pedestrian crossings, and green belts to obtain static target prediction information. Finally, the intermediate feature map is input into the optimized semantic segmentation unit of the semantic segmentation module, which can then identify targets such as lane lines to obtain lane marking prediction information.

[0172] S602 determines the training loss based on dynamic target prediction information, static target prediction information, lane marking prediction information, and each label information.

[0173] After obtaining dynamic target prediction information, static target prediction information, and lane marking prediction information, the label information is combined with the dynamic target prediction information, static target prediction information, and lane marking prediction information respectively to calculate the loss, resulting in multiple loss values. Then, the multiple loss values ​​are combined to calculate the training loss.

[0174] In the target recognition method provided in this application embodiment, the sample environment image is an environment image under traffic scene; the semantic segmentation module includes a semantic segmentation unit and an optimized semantic segmentation unit; the intermediate feature map is input into the target detection module to obtain dynamic target prediction information, and the intermediate feature map is input into the semantic segmentation unit to obtain static target prediction information, and the intermediate feature map is input into the optimized semantic segmentation unit to obtain lane marking prediction information. Then, the training loss is determined based on the dynamic target prediction information, static target prediction information, lane marking prediction information, and each label information. In this method, another optional way to quickly determine the training loss is provided; by inputting the intermediate feature map into the semantic segmentation unit and the optimized semantic segmentation unit of the target detection module and the semantic segmentation module respectively to perform different types of target recognition tasks, dynamic target prediction information, static target prediction information, and lane marking prediction information are obtained respectively. Based on the combination of dynamic target prediction information, static target prediction information, lane marking prediction information, and label information, the training loss can be calculated, laying the foundation for adjusting the parameters of the model to be trained.

[0175] Based on the above embodiments, an embodiment is provided to illustrate the process of determining the training loss based on dynamic target prediction information, static target prediction information, lane marking prediction information, and each label information.

[0176] In one exemplary embodiment, such as Figure 12As shown, the training loss is determined based on dynamic target prediction information, static target prediction information, lane marking prediction information, and various label information, including:

[0177] S701, determine the dynamic target loss value based on dynamic target prediction information, each label information, and a preset loss function corresponding to the dynamic target prediction information; determine the static target loss value based on static target prediction information, each label information, and a preset loss function corresponding to the static target prediction information; determine the lane marking loss value based on lane marking prediction information, each label information, and a preset loss function corresponding to the lane marking prediction information.

[0178] As mentioned earlier, the semantic segmentation unit has two output layers: a confidence output layer and a category output layer. Both output layers can use the cross-entropy loss function. The cross-entropy loss function directly reflects the difference between the model's prediction and the true label, effectively solving the gradient vanishing problem, and conforms to the maximum likelihood estimation principle, performing well in multi-class classification problems. Taking the category output layer as an example, the formula for the cross-entropy loss function is: Among them, y c p is the true label for category c. c The output is for the prediction of category c.

[0179] The optimized semantic segmentation unit has three output layers: a confidence output layer, a category output layer, and a bias output layer. The confidence and category output layers, consistent with the semantic segmentation unit, use the cross-entropy loss function. The bias output layer can use mean squared error loss. Mean squared error loss measures the difference between the model's predicted and actual values ​​by calculating the average of the squared differences between the predicted and actual values, and it performs well in regression tasks. The formula is: Among them, y i For tag information, For predictive information.

[0180] After obtaining dynamic target prediction information, static target prediction information, and lane marking prediction information, the dynamic target prediction information and the corresponding label information are input into the preset loss function corresponding to the dynamic target prediction information to obtain the dynamic target loss value. The static target prediction information and the corresponding label information are input into the preset loss function corresponding to the static target prediction information to obtain the static target loss value. The lane marking prediction information and the corresponding label information are input into the preset loss function corresponding to the lane marking prediction information to obtain the lane marking loss value.

[0181] S702 determines the training loss based on the dynamic target loss value, the static target loss value, and the lane marking loss value.

[0182] In this embodiment, after obtaining the dynamic target loss value, the static target loss value, and the lane marking loss value, the training loss is calculated by combining the dynamic target loss value, the static target loss value, and the lane marking loss value. For example, the training loss can be determined directly by the sum of the dynamic target loss value, the static target loss value, and the lane marking loss value, or it can be calculated by combining different weights.

[0183] In the target recognition method provided in this application, a dynamic target loss value is determined based on dynamic target prediction information, each label information, and a preset loss function corresponding to the dynamic target prediction information; a static target loss value is determined based on static target prediction information, each label information, and a preset loss function corresponding to the static target prediction information; a lane marking loss value is determined based on lane marking prediction information, each label information, and a preset loss function corresponding to the lane marking prediction information; and then, the training loss is determined based on the dynamic target loss value, the static target loss value, and the lane marking loss value. In this method, by using multiple preset loss functions and combining the corresponding prediction information and the corresponding label information, the corresponding loss value can be calculated. Furthermore, by combining multiple loss values, the training loss can be obtained accurately and quickly, providing another optional method for rapidly determining the training loss.

[0184] Based on the above embodiments, an embodiment is provided to illustrate the process of determining the training loss based on the dynamic target loss value, the static target loss value, and the lane marking loss value.

[0185] In one exemplary embodiment, such as Figure 13 As shown, the training loss is determined based on the dynamic target loss value, the static target loss value, and the lane marking loss value, including:

[0186] S801, obtain the detection weight of the target detection module, the segmentation weight of the semantic segmentation unit, and the optimized segmentation weight of the optimized semantic segmentation unit.

[0187] In this embodiment, since the recognition head network includes three different task recognition modules or units, namely the object detection module, the semantic segmentation unit, and the optimized semantic segmentation unit, the loss functions of different modules or units may be unbalanced. Therefore, a weighting mechanism is introduced to calculate the loss.

[0188] Among them, the detection weight is the loss weight corresponding to the target detection module, the segmentation weight is the loss weight corresponding to the semantic segmentation unit, and the optimized segmentation weight is the loss weight corresponding to the optimized semantic segmentation unit.

[0189] For example, the mapping relationship between different modules or units and different weights is stored in the database, and can be obtained from the above mapping relationship based on the identification information of different modules or units. For example, the identification information of the target detection module, the labeling information of the semantic segmentation unit, and the labeling information of the optimized semantic segmentation unit are obtained respectively. Then, based on the three labeling information, the corresponding weights are matched from the above mapping relationship in the database and used as the detection weight of the target detection module, the segmentation weight of the semantic segmentation unit, and the optimized segmentation weight of the optimized semantic segmentation unit, respectively.

[0190] S802 determines the training loss based on the detection weight, segmentation weight, optimized segmentation weight, dynamic target loss value, static target loss value, and lane marking loss value.

[0191] After obtaining the detection weights, segmentation weights, and optimized segmentation weights, the detection weights are multiplied by the dynamic target loss value, the segmentation weights are multiplied by the static target loss value, and the optimized segmentation weights are multiplied by the lane marking loss value. Then, the results of multiple multiplications are summed to obtain the training loss.

[0192] In this embodiment, a weighting mechanism is introduced when calculating the training loss. The weights of each module or unit are continuously updated with the number of training iterations. Using this method for loss calculation can enable the network to converge faster and improve the overall recognition effect.

[0193] The target recognition method provided in this application obtains the detection weights of the target detection module, the segmentation weights of the semantic segmentation unit, and the optimized segmentation weights of the optimized semantic segmentation unit. Then, based on the detection weights, segmentation weights, optimized segmentation weights, dynamic target loss values, static target loss values, and lane marking loss values, a training loss is determined. This method, by introducing the detection weights of the target detection module, the segmentation weights of the semantic segmentation unit, and the optimized segmentation weights of the optimized semantic segmentation unit, comprehensively calculates each weight with its corresponding loss value to obtain the training loss. Training the model under test using this training loss allows the model to converge faster, improving the effect and accuracy of target recognition.

[0194] Based on the above embodiments, an embodiment of the testing process for the model to be trained is provided for illustration.

[0195] In one exemplary embodiment, such as Figure 14 As shown, the method also includes:

[0196] S901, Obtain the test dataset; the test dataset includes multiple test environment images and test label information for each test environment image.

[0197] The test dataset can be a dataset divided from sample environment images and their label information, or it can be separately collected historical data. The test dataset includes multiple test environment images and their test label information. The annotation rules for the test label information are the same as those for the label information of the sample environment images.

[0198] For example, pre-defined sample environment images and corresponding label information can be obtained from a database as a test dataset. Alternatively, other pre-collected historical environment images and pre-labeled information can also be used as a test dataset.

[0199] S902, input the images of each test environment into the trained model to obtain the dynamic target prediction results, static target prediction results and lane marking prediction results output by the trained model.

[0200] After obtaining the test dataset, the test environment images in the test dataset are input into the trained model. The trained model processes the test environment images and outputs different types of target recognition prediction results, namely dynamic target prediction results, static target prediction results, and lane marking prediction results.

[0201] S903, based on the dynamic target prediction results, static target prediction results, lane marking prediction results and various test label information, the trained model is tested to obtain a multi-task target recognition model.

[0202] After obtaining the dynamic target prediction results, static target prediction results, and lane marking prediction results, different evaluation indicators can be calculated by combining the test label information of each test environment image. Based on the evaluation indicators, the trained model can be tested.

[0203] For example, the average accuracy across all classes can be determined based on the dynamic target prediction results and the test label information; the average intersection-over-union ratio can be determined based on the static target prediction results and the test label information; and the average pixel error can be determined based on the lane marking prediction results and the test label information. Then, the trained model can be tested based on the average accuracy across all classes, the average intersection-over-union ratio, and the average pixel error to obtain a multi-task target recognition model.

[0204] Average precision across all classes is a key performance indicator (API) for object detection in multi-label images, measuring the model's average performance across multiple classes. For each class, a precision-recall curve can be plotted, where precision and recall are performance metrics at different thresholds, representing accuracy and recall, respectively. The area under the precision-recall curve represents the average precision for that class. Calculating the average precision for each class and then averaging them yields the overall average precision. Generally, a higher overall average precision indicates better object detection performance across all classes.

[0205] The average intersection-union ratio (OCR) is used to measure the similarity between predicted results and ground truth labels in semantic segmentation tasks. For each category, the similarity is measured by the ratio between the intersection and union regions of the predicted and ground truth labels. The intersection refers to the set of pixels in both the predicted and ground truth labels that are labeled as belonging to the same category, while the union region refers to the set of pixels in both the predicted and ground truth labels that are labeled as belonging to that category at least once. The OCR is calculated for each category, and then the average OCR is obtained. Generally, a higher OCR value indicates better semantic segmentation performance across all categories.

[0206] Mean pixel error (MPixel Error) is an evaluation metric designed for lane recognition tasks, used to measure the similarity between the predicted result and the ground truth label. For each pixel in the predicted result, the distance to the nearest ground truth label pixel is calculated, and then the average distance is calculated to obtain the mean pixel error. Generally speaking, the lower the mean pixel error, the better the model's performance on the lane recognition task.

[0207] The target recognition method provided in this application involves acquiring a test dataset, which includes multiple test environment images and test label information for each image. These test environment images are then input into a trained model to obtain dynamic target prediction results, static target prediction results, and lane marking prediction results. The trained model is then tested based on these results and the test label information to obtain a multi-task target recognition model. This method, by acquiring a test dataset and inputting it into the trained model, allows for understanding the model's performance in real-world applications, ensuring its generalization performance, and providing a basis for model optimization and improvement.

[0208] In addition, in an exemplary embodiment, this application also provides a target recognition method, such as... Figure 15 As shown, it may include the following steps:

[0209] S1001, acquire multiple sample environment images and the label information of each sample environment image.

[0210] S1002, input the environmental images of each sample into the backbone network of the model to be trained to obtain multi-scale feature maps.

[0211] S1003, input the multi-scale feature map into the feature extraction and fusion module of the recognition head network for upsampling and downsampling to obtain the intermediate feature map.

[0212] The model to be trained includes a recognition head network, which includes a feature extraction and fusion module, an object detection module, and a semantic segmentation module. The semantic segmentation module includes a semantic segmentation unit and an optimized semantic segmentation unit.

[0213] S1004, the intermediate feature map is input into the target detection module to obtain dynamic target prediction information, and the intermediate feature map is input into the semantic segmentation unit to obtain static target prediction information, and the intermediate feature map is input into the optimized semantic segmentation unit to obtain lane marking prediction information.

[0214] S1005, determine the dynamic target loss value based on the dynamic target prediction information, each label information, and the preset loss function corresponding to the dynamic target prediction information; determine the static target loss value based on the static target prediction information, each label information, and the preset loss function corresponding to the static target prediction information; determine the lane marking loss value based on the lane marking prediction information, each label information, and the preset loss function corresponding to the lane marking prediction information.

[0215] S1006, obtain the detection weight of the target detection module, the segmentation weight of the semantic segmentation unit, and the optimized segmentation weight of the optimized semantic segmentation unit.

[0216] S1007. The training loss is determined based on the detection weight, segmentation weight, optimized segmentation weight, dynamic target loss value, static target loss value, and lane marking loss value.

[0217] S1008: Based on the training loss, adjust the model parameters of the model to be trained until training is complete, and obtain the multi-task target recognition model.

[0218] S1009, acquire an environmental image of the scene to be identified; the environmental image includes at least one type of target recognition task.

[0219] S1010: Input the environmental image into the pre-trained multi-task target recognition model to obtain the recognition results of various types of targets in the scene to be recognized.

[0220] The processes S1001-S1010 described above can be found in the description of the above method embodiments, and their implementation principles and technical effects are similar, so they will not be repeated here.

[0221] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0222] Based on the same inventive concept, this application also provides a target recognition device for implementing the target recognition method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more target recognition device embodiments provided below can be found in the limitations of the target recognition method described above, and will not be repeated here.

[0223] In one exemplary embodiment, such as Figure 16 As shown, a target recognition device 1 is provided, comprising: an image acquisition module 10 and a target recognition module 20, wherein:

[0224] Image acquisition module 10 is used to acquire environmental images of the scene to be identified; the environmental images include at least one type of target recognition task;

[0225] The target recognition module 20 is used to input environmental images into a pre-trained multi-task target recognition model to obtain the recognition results of various types of targets in the scene to be recognized. The recognition head network of the multi-task target recognition model includes a target detection module and a semantic segmentation module, which are used to handle different types of target recognition tasks.

[0226] In one embodiment, the target recognition device 1 further includes:

[0227] The model building module is used to acquire multiple sample environment images and label information for each sample environment image; using each sample environment image and each label information, the model to be trained is trained to obtain a multi-task target recognition model.

[0228] In one embodiment, the model building module is further configured to:

[0229] Based on the environmental images of each sample, the label information, and the recognition head network, the training loss is determined; based on the training loss, the model parameters of the model to be trained are adjusted until training is completed, and a multi-task target recognition model is obtained.

[0230] In one embodiment, the model building module is further configured to:

[0231] Each sample environmental image is input into the backbone network to obtain a multi-scale feature map; the multi-scale feature map is input into the feature extraction and fusion module for upsampling and downsampling to obtain an intermediate feature map; the training loss is determined based on the intermediate feature map, each label information, the target detection module, and the semantic segmentation module.

[0232] In one embodiment, the model building module is further configured to:

[0233] The intermediate feature map is input into the object detection module to obtain dynamic object prediction information, and into the semantic segmentation unit to obtain static object prediction information. The intermediate feature map is also input into the optimized semantic segmentation unit to obtain lane marking prediction information. Based on the dynamic object prediction information, static object prediction information, lane marking prediction information, and each label information, the training loss is determined.

[0234] In one embodiment, the model building module is further configured to:

[0235] The dynamic target loss value is determined based on the dynamic target prediction information, each label information, and the preset loss function corresponding to the dynamic target prediction information; the static target loss value is determined based on the static target prediction information, each label information, and the preset loss function corresponding to the static target prediction information; the lane marking loss value is determined based on the lane marking prediction information, each label information, and the preset loss function corresponding to the lane marking prediction information; and the training loss is determined based on the dynamic target loss value, the static target loss value, and the lane marking loss value.

[0236] In one embodiment, the model building module is further configured to:

[0237] Obtain the detection weights of the object detection module, the segmentation weights of the semantic segmentation unit, and the optimized segmentation weights of the optimized semantic segmentation unit; determine the training loss based on the detection weights, segmentation weights, optimized segmentation weights, dynamic object loss values, static object loss values, and lane marking loss values.

[0238] In one embodiment, the target recognition device 1 further includes:

[0239] The dataset acquisition module is used to acquire the test dataset; the test dataset includes multiple test environment images and test label information for each test environment image;

[0240] The result prediction module is used to input the images of each test environment into the trained model to obtain the dynamic target prediction results, static target prediction results and lane marking prediction results output by the trained model.

[0241] The model testing module is used to test the trained model based on the dynamic target prediction results, static target prediction results, lane marking prediction results, and various test label information, so as to obtain a multi-task target recognition model.

[0242] Each module in the aforementioned target recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0243] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 17 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores target identification data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a target identification method.

[0244] Those skilled in the art will understand that Figure 17 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0245] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0246] Acquire environmental images of the scene to be identified; the environmental images include at least one type of target recognition task;

[0247] The environmental image is input into a pre-trained multi-task target recognition model to obtain the recognition results of various types of targets in the scene to be recognized. The recognition head network of the multi-task target recognition model includes a target detection module and a semantic segmentation module, which are used to process different types of target recognition tasks.

[0248] The implementation principles and technical effects of each step in the processor embodiment of this application are similar to those of the target recognition method described above, and will not be repeated here.

[0249] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0250] Acquire environmental images of the scene to be identified; the environmental images include at least one type of target recognition task;

[0251] The environmental image is input into a pre-trained multi-task target recognition model to obtain the recognition results of various types of targets in the scene to be recognized. The recognition head network of the multi-task target recognition model includes a target detection module and a semantic segmentation module, which are used to process different types of target recognition tasks.

[0252] The implementation principles and technical effects of each step in the computer program executed by the processor in this embodiment are similar to those of the target recognition method described above, and will not be repeated here.

[0253] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0254] Acquire environmental images of the scene to be identified; the environmental images include at least one type of target recognition task;

[0255] The environmental image is input into a pre-trained multi-task target recognition model to obtain the recognition results of various types of targets in the scene to be recognized. The recognition head network of the multi-task target recognition model includes a target detection module and a semantic segmentation module, which are used to process different types of target recognition tasks.

[0256] The implementation principles and technical effects of each step in the computer program executed by the processor in this embodiment are similar to those of the target recognition method described above, and will not be repeated here.

[0257] It should be noted that the personnel information (including but not limited to personnel and equipment information, personnel personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the personnel or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0258] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0259] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0260] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A target recognition method, characterized in that: The method comprises: Acquire an environmental image of a scene to be recognized; the environmental image includes at least one type of target recognition task; The environmental image is input into a pre-trained multi-task target recognition model to obtain recognition results of various types of targets in the scene to be recognized; the recognition head network of the multi-task target recognition model includes a target detection module and a semantic segmentation module, and the target detection module and the semantic segmentation module are respectively used to process different types of target recognition tasks.

2. The method according to claim 1, characterized in that The multi-task target recognition model also includes a backbone network, which is connected to a feature extraction and fusion module in the recognition head network, and the feature extraction and fusion module is respectively connected to the target detection module and the semantic segmentation module; The backbone network is used to extract features from the environment image to obtain a multi-scale feature map; The feature extraction and fusion module is used to perform feature extraction and fusion processing on the multi-scale feature map to obtain a fused feature map; The target detection module and the semantic segmentation module are respectively used to process corresponding types of target recognition tasks based on the fused feature map.

3. The method according to claim 2, characterized in that The semantic segmentation module includes a semantic segmentation unit; the semantic segmentation unit includes a convolution layer and a confidence output layer and a category output layer connected to the convolution layer; The convolution layer is used to extract features from the fused feature map to obtain an enhanced feature map; The confidence output layer is used to predict whether each pixel in the enhanced feature map belongs to the foreground or the background; The category output layer is used to predict the category of the foreground pixel points in the enhanced feature map.

4. The method according to claim 2, characterized in that: The semantic segmentation module also includes an optimized semantic segmentation unit; the optimized semantic segmentation unit includes a convolution layer and a confidence output layer, an offset output layer and a category output layer connected to the convolution layer; The convolution layer is used to extract features from the fused feature map to obtain an enhanced feature map; The confidence output layer is used to predict whether each pixel in the enhanced feature map belongs to the foreground or the background; The offset output layer is used to predict the offset when the enhanced feature map is restored to the environment image; The category output layer is used to predict the category of the foreground pixel points in the enhanced feature map.

5. The method according to any one of claims 1 to 4, characterized in that: The construction process of the multi-task target recognition model includes: Acquire multiple sample environment images and label information of each of the sample environment images; The model to be trained is trained using each of the sample environment images and each of the label information to obtain the multi-task target recognition model.

6. The method according to claim 5, characterized in that The model to be trained includes the recognition head network; the method of using each of the sample environment images and each of the label information to train the model to be trained to obtain the multi-task target recognition model includes: Determining a training loss according to each of the sample environment images, each of the label information and the recognition head network; Based on the training loss, the model parameters of the model to be trained are adjusted until the training is completed to obtain the multi-task target recognition model.

7. The method according to claim 6, characterized in that The model to be trained also includes a backbone network; the recognition head network includes a feature extraction and fusion module, a target detection module and a semantic segmentation module; the training loss is determined according to each of the sample environment images, each of the label information and the recognition head network, including: Inputting each of the sample environment images into the backbone network to obtain a multi-scale feature map; Inputting the multi-scale feature map into the feature extraction and fusion module for upsampling and downsampling to obtain an intermediate feature map; The training loss is determined according to the intermediate feature map, each of the label information, the object detection module and the semantic segmentation module.

8. The method according to claim 7, characterized in that The sample environment image is an environment image in a traffic scene; the semantic segmentation module includes a semantic segmentation unit and an optimized semantic segmentation unit; the determining the training loss according to the intermediate feature map, each of the label information and the target detection module and the semantic segmentation module includes: Inputting the intermediate feature map into the target detection module to obtain dynamic target prediction information, inputting the intermediate feature map into the semantic segmentation unit to obtain static target prediction information, and inputting the intermediate feature map into the optimized semantic segmentation unit to obtain lane marking prediction information; The training loss is determined according to the dynamic target prediction information, the static target prediction information, the lane marking prediction information and each of the label information.

9. The method according to claim 8, characterized in that The determining the training loss according to the dynamic target prediction information, the static target prediction information, the lane marking prediction information and each of the label information includes: Determine a dynamic target loss value according to the dynamic target prediction information, each of the label information and a preset loss function corresponding to the dynamic target prediction information; determine a static target loss value according to the static target prediction information, each of the label information and a preset loss function corresponding to the static target prediction information; determine a lane marking loss value according to the lane marking prediction information, each of the label information and a preset loss function corresponding to the lane marking prediction information; The training loss is determined according to the dynamic target loss value, the static target loss value and the lane marking loss value.

10. The method according to claim 9, characterized in that The determining the training loss according to the dynamic target loss value, the static target loss value and the lane marking loss value comprises: Acquire the detection weight of the target detection module, the segmentation weight of the semantic segmentation unit, and the optimized segmentation weight of the optimized semantic segmentation unit; The training loss is determined according to the detection weight, the segmentation weight, the optimized segmentation weight, the dynamic target loss value, the static target loss value and the lane marking loss value.

11. The method according to claim 5, characterized in that The method further comprises: Acquire a test data set; the test data set includes a plurality of test environment images and test label information of each of the test environment images; Inputting each of the test environment images into the trained model to be trained, to obtain a dynamic target prediction result, a static target prediction result, and a lane marking prediction result output by the trained model to be trained; According to the dynamic target prediction result, the static target prediction result, the lane marking prediction result and each of the test label information, the trained model to be trained is tested to obtain the multi-task target recognition model.

Images