Multi-task detection method, system and equipment for travelable area and garbage and medium

Abstract

The invention discloses a multi-task detection method, system and device for a travelable area and garbage and a medium, belongs to the technical field of artificial intelligence, and aims to solve the technical problems of poor positioning precision and segmentation effect of the travelable area in the prior art. The constructed multi-task detection model comprises a backbone network, a neck network, a drivable area segmentation head and a garbage detection head. The FPN sub-network of the neck network comprises two CSPStage modules, two convolution-batch normalization-activation function layers and two up-sampling modules; the PAN sub-network comprises three CSPStage modules and two convolution-batch normalization-activation function layers, and the CSPStage modules are connected with the two convolution-batch normalization-activation function layers; the drivable area segmentation head comprises three convolution-batch normalization-activation function layers, three dynamic up-sampling modules and three CEEM modules which are arranged in sequence. A dynamic up-sampling module is introduced into a segmentation head to realize step-by-step up-sampling and spatial resolution recovery of high-level semantic features, so that the perception ability of a model to details in an image is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to the detection and identification of drivable areas and garbage for sanitation vehicles, and in particular to a multi-task detection method, system, equipment and medium for drivable areas and garbage. Background Art

[0002] In recent years, with the rapid development of artificial intelligence (AI), it has also been widely used in intelligent assisted driving. In intelligent assisted driving, especially for sanitation vehicles, identifying and segmenting drivable road areas and detecting and identifying garbage are the most fundamental and core tasks.

[0003] Driving area identification and segmentation refers to the technology that detects and analyzes the vehicle's surroundings, dividing the road and surrounding scene into drivable and non-drivable areas. This technology utilizes sensors, cameras, lidar, and other devices to obtain real-time road surface information and obstacle locations, and processes and analyzes this information in conjunction with map data to determine safe driving areas. Driving area segmentation technology plays a key role in intelligent assisted driving systems, improving vehicle perception, decision-making capabilities, and driving safety, providing important support for the realization of highly intelligent assisted driving.

[0004] Garbage detection and recognition refers to the technology that detects and analyzes garbage around a vehicle and its surroundings to identify it. This technology primarily captures and processes images to identify the presence and location of garbage. Garbage detection and recognition technology plays a key role in achieving road cleaning in intelligent assisted driving for sanitation vehicles and provides important support for their path planning.

[0005] Currently, whether it is the identification and segmentation of drivable areas or the detection and identification of garbage, the mainstream method still uses deep learning segmentation models to segment road images to obtain drivable areas, non-drivable areas, whether there is garbage, and the location of the garbage.

[0006] The invention patent application with application number 202411501638.3 discloses a method and program for identifying drivable areas on unstructured roads based on deep learning, which includes: obtaining a road image; preprocessing the road image; constructing a drivable area recognition model, and pre-training the drivable area recognition model based on a road image training set; wherein the network structure of the drivable area recognition model includes: an input end, a backbone network, a neck network, and a segmentation detection head; the backbone network is provided with a C2f-iRMB module, the C2f-iRMB module is used for feature extraction, and the position information in the feature image is integrated into the channel attention; the neck network is used to perform semantic segmentation on the road image; the segmentation detection head is used for target detection and target image segmentation; the road image is input into the drivable area recognition model for image recognition to obtain a road recognition result. The drivable area recognition model integrates the position information in the feature image into the channel attention through the C2f-iRMB module, performs semantic segmentation on the road image through the neck network, and performs target detection and target image segmentation through the segmentation detection head.

[0007] As mentioned in the aforementioned invention patent application, existing technologies for unstructured road recognition rely on high-resolution semantic restoration, but these methods lack an enhanced structure specifically for reconstructing spatial details. For example, the upsampling module in the aforementioned invention patent application number 202411501638.3 uses the conventional up → SDI → C2f2_3d algorithm, which fails to restore edge details when recovering spatial structure. The Segment output structure is more suitable for instance segmentation and is less suitable for detection tasks. Summary of the Invention

[0008] The purpose of the present invention is to provide a multi-task detection method, system, device and medium for drivable area and garbage in order to solve the technical problems of poor positioning accuracy and segmentation effect of drivable area in the prior art.

[0009] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions: A multi-task detection method for a drivable area and garbage includes the following steps: Step S1, obtaining sample data; Use the forward-facing camera of a sanitation vehicle to obtain road image sample data, and mark the drivable areas and garbage in the road image sample data to obtain label data; Step S2, building a multi-task detection model; Build a multi-task detection model, which includes a backbone network, a neck network, a drivable area segmentation head, and a garbage detection head; The backbone network includes a first convolution-batch normalization-activation function layer, a first CSPRes module, a second CSPRes module, a third CSPRes module, and a fourth CSPRes module, which are arranged in sequence; The neck network includes an FPN subnetwork and a PAN subnetwork. The FPN subnetwork includes a first CSPStage module, a second convolution-batch normalization-activation function layer, a first upsampling module, a second CSPStage module, a third convolution-batch normalization-activation function layer, and a second upsampling module. The output of the fourth CSPRes module of the backbone network is used as the input of the first CSPStage module, and the output of the third CSPRes module of the backbone network and the output of the first upsampling module are used as the input of the second CSPStage module. The PAN subnetwork includes a third CSPStage module set in sequence. The output of the second CSPRes module of the backbone network and the output of the second upsampling module are used as the input of the third CSPStage module. The output of the fourth convolution-batch normalization-activation function layer and the output of the second CSPStage module are used as the input of the fourth CSPStage module. The output of the first CSPStage module and the output of the fifth convolution-batch normalization-activation function layer are used as the input of the fifth CSPStage module. The drivable area segmentation head includes a sixth convolution-batch normalization-activation function layer, a first dynamic upsampling module, a first CEEM module, a seventh convolution-batch normalization-activation function layer, a second dynamic upsampling module, a second CEEM module, an eighth convolution-batch normalization-activation function layer, a third dynamic upsampling module, and a third CEEM module, which are arranged in sequence; the output of the second upsampling module of the FPN subnetwork serves as the input of the sixth convolution-batch normalization-activation function layer; The garbage detection head includes the first convolutional layer, the second convolutional layer, and the third convolutional layer. The outputs of the third CSPStage module, the fourth CSPStage module, and the fifth CSPStage module serve as the inputs of the first convolutional layer, the second convolutional layer, and the third convolutional layer, respectively. Step S3, training a multi-task detection model; The sample data and label data obtained in step S1 are input into the multi-task detection model constructed in step S2 to train the multi-task detection model; Step S4, multi-task real-time detection; A real-time image of the road is obtained using the forward-facing camera of the sanitation vehicle, and the real-time image of the road is input into the multi-task detection model trained in step S3. The multi-task detection model outputs the drivable area and garbage detection results.

[0010] Furthermore, in step S1, the record bag file of the rosbag tool in the sanitation vehicle ROS system is used to store the data recorded by the forward camera, and the frames are unpacked and extracted into image data; the X-AnyLabeling tool is used to label the drivable area and garbage targets in each image, and the label data is obtained and saved as a JSON format file.

[0011] Furthermore, in step S3, when training the multi-task detection model, the total loss function for: ; Adaptive Weighted Zoom Classification Loss for: ; Regression loss combining IoU and normalized Wasserstein distance for: ; in, 、 、 、 Both represent weights, is the target confidence loss, Split the losses into drivable areas; 、 Respectively represent the length and width of the anchor box mapped back to the original image, 、 Represent the length and width of the original image respectively, represents the adaptive weight factor, represents the prediction score, represents the true label of the sample, represents the target IoU score, represents the equilibrium parameter, Represents the shape-based normalized Wasserstein distance.

[0012] Furthermore, the target confidence loss for: ; Driving area segmentation loss for: ; in, Indicates the number of predicted targets; Represents the true label of the i-th predicted target. If the target exists, then =1, otherwise 0; represents the probability that the i-th predicted target is a positive class, 、 Represent the length and width of the original image respectively; Represents the true label of the jth and kth pixel points. If the true label is drivable, then =1, otherwise 0; Indicates the probability that the jth and kth pixel points are predicted to be "drivable".

[0013] Further, The calculation formula is: ; parameter 、 The calculation formula is:

[0014] The calculation formula is: ; in, 、 Represents the weight coefficients in the horizontal and vertical directions respectively, and the values ​​are related to the shape of the true bounding box; Indicates the average size of the target data set, 、 Represents the center coordinate of the prediction box, 、 represents the center coordinate of the real box, It represents the difference between the measured frame and the real frame after normalization in terms of size w and h. 、 Indicates the width and height of the real box, represents the scale factor; A constant representing the normalized width and height error.

[0015] A multi-task detection system for drivable areas and garbage, comprising: The sample data acquisition module is used to obtain road image sample data using the forward-facing camera of the sanitation vehicle, and mark the drivable areas and garbage in the road image sample data to obtain label data; Multi-task detection model construction module, used to build a multi-task detection model, which includes a backbone network, a neck network, a drivable area segmentation head, and a garbage detection head; The backbone network includes a first convolution-batch normalization-activation function layer, a first CSPRes module, a second CSPRes module, a third CSPRes module, and a fourth CSPRes module, which are arranged in sequence; The neck network includes an FPN subnetwork and a PAN subnetwork. The FPN subnetwork includes a first CSPStage module, a second convolution-batch normalization-activation function layer, a first upsampling module, a second CSPStage module, a third convolution-batch normalization-activation function layer, and a second upsampling module. The output of the fourth CSPRes module of the backbone network is used as the input of the first CSPStage module, and the output of the third CSPRes module of the backbone network and the output of the first upsampling module are used as the input of the second CSPStage module. The PAN subnetwork includes a third CSPStage module set in sequence. The output of the second CSPRes module of the backbone network and the output of the second upsampling module are used as the input of the third CSPStage module. The output of the fourth convolution-batch normalization-activation function layer and the output of the second CSPStage module are used as the input of the fourth CSPStage module. The output of the first CSPStage module and the output of the fifth convolution-batch normalization-activation function layer are used as the input of the fifth CSPStage module. The drivable area segmentation head includes a sixth convolution-batch normalization-activation function layer, a first dynamic upsampling module, a first CEEM module, a seventh convolution-batch normalization-activation function layer, a second dynamic upsampling module, a second CEEM module, an eighth convolution-batch normalization-activation function layer, a third dynamic upsampling module, and a third CEEM module, which are arranged in sequence; the output of the second upsampling module of the FPN subnetwork serves as the input of the sixth convolution-batch normalization-activation function layer; The garbage detection head includes the first convolutional layer, the second convolutional layer, and the third convolutional layer. The outputs of the third CSPStage module, the fourth CSPStage module, and the fifth CSPStage module serve as the inputs of the first convolutional layer, the second convolutional layer, and the third convolutional layer, respectively. The multi-task detection model training module is used to input the sample data and label data obtained by the sample data acquisition module into the multi-task detection model constructed by the multi-task detection model construction module to train the multi-task detection model; The multi-task real-time detection module is used to use the forward-facing camera of the sanitation vehicle to obtain real-time road images, input the real-time road images into the multi-task detection model trained by the multi-task detection model training module, and the multi-task detection model outputs the drivable area and garbage detection results.

[0016] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0017] A computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the steps of the above method.

[0018] The beneficial effects of the present invention are as follows: 1. In this invention, a dynamic upsampling module is introduced into the segmentation head to achieve step-by-step upsampling of high-level semantic features and restore spatial resolution, thereby enhancing the model's ability to perceive details in the image and helping to accurately restore key details such as boundary contours and surface textures in unstructured roads. At the same time, an edge extraction module is integrated to guide the model to focus on the edge information of the target area, further improving the positioning accuracy and segmentation effect of the drivable area boundary. 2. In this invention, the drivable area segmentation head and the garbage detection head are structurally decoupled. Each of them constructs independent task branches, but shares the backbone network and neck feature extraction module to achieve efficient reuse of underlying features. While ensuring feature sharing, it avoids feature competition and information interference between multiple tasks, improves the independence and collaborative stability of tasks, and balances the precision of drivable area segmentation with the accuracy of garbage target detection. 3. In this invention, standard convolution is adopted, CSPRes is used to enhance feature expression capabilities, and FPN and PAN modules are used to achieve multi-scale feature fusion and context modeling, which facilitates rapid deployment in existing industrial frameworks. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic flow diagram of the present invention; Figure 2 It is a structural diagram of the multi-task detection model in the present invention; Figure 3 It is a schematic diagram of the structure of the neck network in the present invention; Figure 4 It is a structural schematic diagram of the drivable area segmentation head in the present invention; Figure 5 It is a schematic diagram of the structure of the CEEM module in the present invention; Figure 6 Schematic diagram of the structure of the CSPRes module in the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0021] Therefore, based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0022] Example 1 This embodiment provides a multi-task detection method for drivable area and garbage, which is used to perform multi-task detection of drivable area detection and garbage identification on road images. Figure 1 As shown, it includes the following steps: Step S1, obtaining sample data; The forward-facing camera of a sanitation vehicle is used to obtain road image sample data, and the drivable areas and garbage in the road image sample data are marked to obtain label data.

[0023] When obtaining road image samples, the forward-facing camera of the sanitation vehicle is used to obtain images, and the data recorded by the forward-facing camera is stored in the record bag file of the rosbag tool in the sanitation vehicle ROS system, and the frames are unpacked and extracted into image data; the X-AnyLabeling tool is used to annotate the drivable area and pull-in in each image, obtain the label data, and save it as a JSON format file.

[0024] When labeling, polygon coordinates are used to delineate the range of the drivable area and generate a mask map; the garbage targets in the image are represented by the target area indicated by a rectangular frame.

[0025] Step S2, building a multi-task detection model; Construct a multi-task detection model, which includes a backbone network, a neck network, a drivable area segmentation head, and a garbage detection head. Figure 2 shown.

[0026] The backbone network includes the first convolution-batch normalization-activation function layer, the first CSPRes module, the second CSPRes module, the third CSPRes module, and the fourth CSPRes module. CSPRepResNet (i.e., CSPRes module) is selected as the backbone network for feature extraction. The CSPRepResNet introduces the cross-stage partial network CSP structure and RepVGG optimization module, and integrates the ESE channel attention module. The CSPRes module is a cross-stage local residual block, and its structure is as follows: Figure 6 As shown in the figure; in the backbone network, the attention mechanism is an Effective Squeeze and Extraction module, which adaptively adjusts the weights of each channel of the feature map.

[0027] The neck network is constructed based on FPN feature pyramid and PAN path aggregation, such as Figure 3As shown, the neck network includes an FPN subnetwork and a PAN subnetwork. The FPN subnetwork includes a first CSPStage module, a second convolution-batch normalization-activation function layer, a first upsampling module, a second CSPStage module, a third convolution-batch normalization-activation function layer, and a second upsampling module, and the output of the fourth CSPRes module of the backbone network is used as the input of the first CSPStage module. The output of the third CSPRes module of the backbone network and the output of the first upsampling module are used together as the input of the second CSPStage module. The PAN subnetwork includes a third CSPStage module set in sequence. The backbone network consists of the following modules: the fourth convolution-batch normalization-activation function layer, the fourth CSPStage module, the fifth convolution-batch normalization-activation function layer, and the fifth CSPStage module. The output of the second CSPRes module and the output of the second upsampling module of the backbone network serve as the input of the third CSPStage module. The output of the fourth convolution-batch normalization-activation function layer and the output of the second CSPStage module serve as the input of the fourth CSPStage module. The output of the first CSPStage module and the output of the fifth convolution-batch normalization-activation function layer serve as the input of the fifth CSPStage module. In the neck network, the attention mechanism uses upper and lower anchor attention (Context Anchor Attention) to capture long-range contextual information and fuse feature information.

[0028] like Figure 4 As shown, the drivable area segmentation head includes, in sequence, a sixth convolution-batch normalization-activation layer, a first dynamic upsampling module, a first CEEM module, a seventh convolution-batch normalization-activation layer, a second dynamic upsampling module, a second CEEM module, an eighth convolution-batch normalization-activation layer, a third dynamic upsampling module, and a third CEEM module. The output of the second upsampling module of the FPN subnetwork serves as the input to the sixth convolution-batch normalization-activation layer. This drivable area segmentation head uses the output of the FPN feature pyramid and restores it to the original image size through a stepwise upsampling module. A dynamic upsampling module is introduced during this process to achieve stepwise upsampling of high-level semantic features and restore spatial resolution, thereby enhancing the model's ability to perceive image details and helping to accurately restore key details such as boundary contours and surface textures in unstructured roads.

[0029] Among them, the CEEM module is the edge enhancement module, and its structure is as follows Figure 5 As shown in the figure, the CEEM module uses a 1*1 convolution layer to adjust the number of channels of the input feature map, and uses a residual structure to connect the edge-enhanced features with the input features. Feature map adjusted by the 1*1 convolution layer Expressed as: , represents the input features, Represents the weight of the 1*1 convolutional layer; the enhanced feature map Expressed as: , represents the weight of the 3*3 convolutional layer, BN represents the batch normalization operation, and ReLU represents the corresponding activation function; the fused features are finally obtained. : = + .

[0030] The garbage detection head consists of the first, second, and third convolutional layers. The outputs of the third, fourth, and fifth CSPStage modules serve as the inputs to the first, second, and third convolutional layers, respectively. The garbage detection head uses the convolutional layers to adjust the channel-matching anchor box predictions, performs target prediction on the neck network output, and merges predictions at multiple scales.

[0031] The above-mentioned convolution-batch normalization-activation function layer is the basic convolution module structure, and the specific structure is the convolution layer, batch normalization layer and activation function set in sequence.

[0032] Step S3, training a multi-task detection model; The sample data and label data obtained in step S1 are input into the multi-task detection model constructed in step S2 for training. The multi-task detection model is trained using a weighted loss function that combines classification loss with adaptive weight zoom, regression loss with a combination of IoU and normalized Wasserstein distance, object confidence loss, and drivable area segmentation loss with a binary cross entropy loss function.

[0033] When training a multi-task detection model, the total loss function for: ; Adaptive Weighted Zoom Classification Loss for: ; Regression loss combining IoU and normalized Wasserstein distance for: ; Target confidence loss for: ; Driving area segmentation loss for: ; in, 、 、 、 Both represent weights, is the target confidence loss, Split the losses into drivable areas; 、 Respectively represent the length and width of the anchor box mapped back to the original image, 、 Represent the length and width of the original image respectively, represents the adaptive weight factor, represents the prediction score, represents the true label of the sample, represents the target IoU score, represents the equilibrium parameter, Represents the shape-based normalized Wasserstein distance; Indicates the number of predicted targets; Represents the true label of the i-th predicted target. If the target exists, then =1, otherwise 0; represents the probability that the i-th predicted target is a positive class, Represents the true label of the jth and kth pixel points. If the true label is drivable, then =1, otherwise 0; Indicates the probability that the jth and kth pixel points are predicted to be "drivable".

[0034] Regression Loss middle, The calculation formula is: ; parameter 、 The calculation formula is:

[0035] The calculation formula is: ; in, 、 Represents the weight coefficients in the horizontal and vertical directions respectively, and the values ​​are related to the shape of the true bounding box; Represents the average size of the target in the data set, which is used to adjust the scale of the loss and control the sensitivity of the loss; 、 Represents the center coordinate of the prediction box, 、 represents the center coordinate of the real box, It represents the difference between the measured frame and the real frame after normalization in terms of size w and h. 、 Indicates the width and height of the real box, represents the scale factor; A constant representing the normalized width and height error.

[0036] When training a multi-task detection model, in addition to the above-mentioned loss function, other loss methods can refer to the existing technology, and those skilled in the art do not need to make any creative efforts.

[0037] Step S4, multi-task real-time detection; The sanitation vehicle's forward-facing camera is used to obtain real-time road images, which are then fed into the multi-task detection model trained in step S3. After forward reasoning, the multi-task detection model outputs the drivable area and garbage detection results.

[0038] The drivable area and garbage detection results obtained through real-time segmentation can be used for assisted driving of sanitation vehicles, or can be used to represent them in the original image (for example, selecting and displaying the drivable area in the original image), among other uses.

[0039] Example 2 This embodiment provides a multi-task detection system for drivable areas and garbage, which is used to perform multi-task detection of drivable areas and garbage identification on road images, and includes: The sample data acquisition module is used to obtain road image sample data using the forward-facing camera of the sanitation vehicle, and mark the drivable areas and garbage in the road image sample data to obtain label data.

[0040] When obtaining road image samples, the forward-facing camera of the sanitation vehicle is used to obtain images, and the data recorded by the forward-facing camera is stored in the record bag file of the rosbag tool in the sanitation vehicle ROS system, and the frames are unpacked and extracted into image data; the X-AnyLabeling tool is used to annotate the drivable area and pull-in in each image, obtain the label data, and save it as a JSON format file.

[0041] When labeling, polygon coordinates are used to delineate the range of the drivable area and generate a mask map; the garbage targets in the image are represented by the target area indicated by a rectangular frame.

[0042] Multi-task detection model building module is used to build a multi-task detection model. The multi-task detection model includes a backbone network, a neck network, a drivable area segmentation head, and a garbage detection head. Figure 2 shown.

[0043] The backbone network consists of the first convolution-batch normalization-activation function layer, the first CSPRes module, the second CSPRes module, the third CSPRes module, and the fourth CSPRes module. CSPRepResNet (i.e., the CSPRes module) was selected as the backbone network for feature extraction. This CSPRepResNet incorporates a cross-stage partial network (CSP) structure and a RepVGG optimization module, while also integrating an ESE channel attention module.

[0044] The neck network is constructed based on FPN feature pyramid and PAN path aggregation, such as Figure 3 As shown, the neck network includes an FPN subnetwork and a PAN subnetwork. The FPN subnetwork includes a first CSPStage module, a second convolution-batch normalization-activation function layer, a first upsampling module, a second CSPStage module, a third convolution-batch normalization-activation function layer, and a second upsampling module, and the output of the fourth CSPRes module of the backbone network is used as the input of the first CSPStage module. The output of the third CSPRes module of the backbone network and the output of the first upsampling module are used together as the input of the second CSPStage module. The PAN subnetwork includes a third CSPStage module set in sequence. Module, the fourth convolution-batch normalization-activation function layer, the fourth CSPStage module, the fifth convolution-batch normalization-activation function layer, the fifth CSPStage module, the output of the second CSPRes module of the backbone network and the output of the second upsampling module are used as the input of the third CSPStage module, the output of the fourth convolution-batch normalization-activation function layer and the output of the second CSPStage module are used as the input of the fourth CSPStage module, and the output of the first CSPStage module and the output of the fifth convolution-batch normalization-activation function layer are used as the input of the fifth CSPStage module.

[0045] like Figure 4 As shown, the drivable area segmentation head includes, in sequence, a sixth convolution-batch normalization-activation layer, a first dynamic upsampling module, a first CEEM module, a seventh convolution-batch normalization-activation layer, a second dynamic upsampling module, a second CEEM module, an eighth convolution-batch normalization-activation layer, a third dynamic upsampling module, and a third CEEM module. The output of the second upsampling module of the FPN subnetwork serves as the input to the sixth convolution-batch normalization-activation layer. This drivable area segmentation head uses the output of the FPN feature pyramid and restores it to the original image size through a stepwise upsampling module. A dynamic upsampling module is introduced during this process to achieve stepwise upsampling of high-level semantic features and restore spatial resolution, thereby enhancing the model's ability to perceive image details and helping to accurately restore key details such as boundary contours and surface textures in unstructured roads.

[0046] The garbage detection head consists of the first, second, and third convolutional layers. The outputs of the third, fourth, and fifth CSPStage modules serve as the inputs to the first, second, and third convolutional layers, respectively. The garbage detection head uses the convolutional layers to adjust the channel-matching anchor box predictions, performs target prediction on the neck network output, and merges predictions at multiple scales.

[0047] The above-mentioned convolution-batch normalization-activation function layer is the basic convolution module structure, and the specific structure is the convolution layer, batch normalization layer and activation function set in sequence.

[0048] The multi-task detection model training module uses the sample data and label data acquired by the sample data acquisition module to input the multi-task detection model built by the multi-task detection model construction module for training. The multi-task detection model is trained using a weighted loss function that combines classification loss with adaptive weight zoom, regression loss combining IoU and normalized Wasserstein distance, object confidence loss, and drivable area segmentation loss with a binary cross entropy loss function.

[0049] When training a multi-task detection model, the total loss function for: ; Adaptive Weighted Zoom Classification Loss for: ; Regression loss combining IoU and normalized Wasserstein distance for: ; Target confidence loss for: ; Driving area segmentation loss for: ; in, 、 、 、 Both represent weights, is the target confidence loss, Split the losses into drivable areas; 、 Respectively represent the length and width of the anchor box mapped back to the original image, 、 Represent the length and width of the original image respectively, represents the adaptive weight factor, represents the prediction score, represents the true label of the sample, represents the target IoU score, represents the equilibrium parameter, Represents the shape-based normalized Wasserstein distance; Indicates the number of predicted targets; Represents the true label of the i-th predicted target. If the target exists, then =1, otherwise 0; represents the probability that the i-th predicted target is a positive class, Represents the true label of the jth and kth pixel points. If the true label is drivable, then =1, otherwise 0; Indicates the probability that the jth and kth pixel points are predicted to be "drivable".

[0050] Regression Loss middle, The calculation formula is: ; parameter 、 The calculation formula is:

[0051] The calculation formula is: ; in, 、 Represents the weight coefficients in the horizontal and vertical directions respectively, and the values ​​are related to the shape of the true bounding box; Represents the average size of the target in the data set, which is used to adjust the scale of the loss and control the sensitivity of the loss; 、 Represents the center coordinate of the prediction box, 、 represents the center coordinate of the real box, It represents the difference between the measured frame and the real frame after normalization in terms of size w and h. 、 Indicates the width and height of the real box, represents the scale factor; A constant representing the normalized width and height error.

[0052] When training a multi-task detection model, in addition to the above-mentioned loss function, other loss methods can refer to the existing technology, and those skilled in the art do not need to make any creative efforts.

[0053] The multi-task real-time detection module is used to use the forward-facing camera of the sanitation vehicle to obtain real-time road images, and input the real-time road images into the multi-task detection model trained by the multi-task detection model training module. After forward inference, the multi-task detection model outputs the drivable area and garbage detection results.

[0054] The drivable area and garbage detection results obtained through real-time segmentation can be used for assisted driving of sanitation vehicles, or can be used to represent them in the original image (for example, selecting and displaying the drivable area in the original image), among other uses.

[0055] Example 3 A computer device includes a memory and a processor, wherein the memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of a multi-task detection method for a drivable area and garbage.

[0056] The computer device may be a desktop computer, a notebook computer, a PDA, a cloud server, etc. The computer device may interact with the user via a keyboard, a mouse, a remote control, a touchpad, or a voice control device.

[0057] The memory includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or D-interface display memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of the computer device, such as the hard disk or internal memory of the computer device. In other embodiments, the memory may also be an external storage device of the computer device, such as a plug-in hard disk equipped with the computer device, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is often used to store the operating system and various application software installed on the computer device, such as the program code of the multi-task detection method for drivable areas and garbage. In addition, the memory may also be used to temporarily store various types of data that have been output or are about to be output.

[0058] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of the computer device. In this embodiment, the processor is used to execute program code stored in the memory or process data, such as executing the program code for the multi-task detection method for the drivable area and garbage.

[0059] Example 4 A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor executes the steps of a multi-task detection method for a drivable area and garbage.

[0060] The computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor to enable the at least one processor to perform the steps of the multi-task detection method for drivable areas and garbage as described above.

[0061] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment method can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the multi-task detection method for drivable areas and garbage described in the embodiment of the present application.

Claims

1. A multi-task detection method for drivable areas and garbage, characterized in that: The following steps are involved: Step S1, obtaining sample data; Use the forward-facing camera of a sanitation vehicle to obtain road image sample data, and mark the drivable areas and garbage in the road image sample data to obtain label data; Step S2, building a multi-task detection model; Build a multi-task detection model, which includes a backbone network, a neck network, a drivable area segmentation head, and a garbage detection head; The backbone network includes a first convolution-batch normalization-activation function layer, a first CSPRes module, a second CSPRes module, a third CSPRes module, and a fourth CSPRes module, which are arranged in sequence; The neck network includes an FPN subnetwork and a PAN subnetwork. The FPN subnetwork includes a first CSPStage module, a second convolution-batch normalization-activation function layer, a first upsampling module, a second CSPStage module, a third convolution-batch normalization-activation function layer, and a second upsampling module. The output of the fourth CSPRes module of the backbone network is used as the input of the first CSPStage module, and the output of the third CSPRes module of the backbone network and the output of the first upsampling module are used as the input of the second CSPStage module. The PAN subnetwork includes a third CSPStage module set in sequence. The output of the second CSPRes module of the backbone network and the output of the second upsampling module are used as the input of the third CSPStage module. The output of the fourth convolution-batch normalization-activation function layer and the output of the second CSPStage module are used as the input of the fourth CSPStage module. The output of the first CSPStage module and the output of the fifth convolution-batch normalization-activation function layer are used as the input of the fifth CSPStage module. The drivable area segmentation head includes a sixth convolution-batch normalization-activation function layer, a first dynamic upsampling module, a first CEEM module, a seventh convolution-batch normalization-activation function layer, a second dynamic upsampling module, a second CEEM module, an eighth convolution-batch normalization-activation function layer, a third dynamic upsampling module, and a third CEEM module, which are arranged in sequence; the output of the second upsampling module of the FPN subnetwork serves as the input of the sixth convolution-batch normalization-activation function layer; The garbage detection head includes the first convolutional layer, the second convolutional layer, and the third convolutional layer. The outputs of the third CSPStage module, the fourth CSPStage module, and the fifth CSPStage module serve as the inputs of the first convolutional layer, the second convolutional layer, and the third convolutional layer, respectively. Step S3, training a multi-task detection model; The sample data and label data obtained in step S1 are input into the multi-task detection model constructed in step S2 to train the multi-task detection model; Step S4, multi-task real-time detection; A real-time image of the road is obtained using the forward-facing camera of the sanitation vehicle, and the real-time image of the road is input into the multi-task detection model trained in step S3. The multi-task detection model outputs the drivable area and garbage detection results.

2. A multi-task detection method for drivable areas and garbage according to claim 1, characterized in that: In step S1, the data recorded by the forward-facing camera is stored in the recording bag file of the rosbag tool in the sanitation vehicle ROS system, and the frames are unpacked and extracted into image data. The drivable area and garbage targets in each image are labeled using the X-AnyLabeling tool to obtain the label data and save it as a JSON format file.

3. The multi-task detection method for drivable areas and garbage according to claim 1, characterized in that: In step S3, when training the multi-task detection model, the total loss function for: ； Adaptive Weighted Zoom Classification Loss for: ； Regression loss combining IoU and normalized Wasserstein distance for: ； in, 、 、 、 Both represent weights, is the target confidence loss, Split the losses into drivable areas; 、 Respectively represent the length and width of the anchor box mapped back to the original image, 、 Represent the length and width of the original image respectively, represents the adaptive weight factor, represents the prediction score, represents the true label of the sample, represents the target IoU score, represents the equilibrium parameter, Represents the shape-based normalized Wasserstein distance.

4. The multi-task detection method for drivable areas and garbage according to claim 3, characterized in that: Target confidence loss for: ； Driving area segmentation loss for: ； in, Indicates the number of predicted targets; Represents the true label of the i-th predicted target. If the target exists, then =1, otherwise 0; represents the probability that the i-th predicted target is a positive class, 、 Represent the length and width of the original image respectively; Represents the true label of the jth and kth pixel points. If the true label is drivable, then =1, otherwise 0; Indicates the probability that the jth and kth pixel points are predicted to be "drivable".

5. The multi-task detection method for drivable areas and garbage according to claim 3, characterized in that: The calculation formula is: ； parameter 、 The calculation formula is: ； in, 、 Represents the weight coefficients in the horizontal and vertical directions respectively, and the values ​​are related to the shape of the true bounding box; Indicates the average size of the target data set, 、 Represents the center coordinate of the prediction box, 、 represents the center coordinate of the real box, It represents the difference between the measured frame and the real frame after normalization in terms of size w and h. 、 Indicates the width and height of the real box, represents the scale factor; A constant representing the normalized width and height error.

6. A multi-task detection system for drivable areas and garbage, characterized by: include: The sample data acquisition module is used to obtain road image sample data using the forward-facing camera of the sanitation vehicle, and mark the drivable areas and garbage in the road image sample data to obtain label data; Multi-task detection model construction module, used to build a multi-task detection model, which includes a backbone network, a neck network, a drivable area segmentation head, and a garbage detection head; The backbone network includes a first convolution-batch normalization-activation function layer, a first CSPRes module, a second CSPRes module, a third CSPRes module, and a fourth CSPRes module, which are arranged in sequence; The neck network includes an FPN subnetwork and a PAN subnetwork. The FPN subnetwork includes a first CSPStage module, a second convolution-batch normalization-activation function layer, a first upsampling module, a second CSPStage module, a third convolution-batch normalization-activation function layer, and a second upsampling module. The output of the fourth CSPRes module of the backbone network is used as the input of the first CSPStage module, and the output of the third CSPRes module of the backbone network and the output of the first upsampling module are used as the input of the second CSPStage module. The PAN subnetwork includes a third CSPStage module set in sequence. The output of the second CSPRes module of the backbone network and the output of the second upsampling module are used as the input of the third CSPStage module. The output of the fourth convolution-batch normalization-activation function layer and the output of the second CSPStage module are used as the input of the fourth CSPStage module. The output of the first CSPStage module and the output of the fifth convolution-batch normalization-activation function layer are used as the input of the fifth CSPStage module. The drivable area segmentation head includes a sixth convolution-batch normalization-activation function layer, a first dynamic upsampling module, a first CEEM module, a seventh convolution-batch normalization-activation function layer, a second dynamic upsampling module, a second CEEM module, an eighth convolution-batch normalization-activation function layer, a third dynamic upsampling module, and a third CEEM module, which are arranged in sequence; the output of the second upsampling module of the FPN subnetwork serves as the input of the sixth convolution-batch normalization-activation function layer; The garbage detection head includes the first convolutional layer, the second convolutional layer, and the third convolutional layer. The outputs of the third CSPStage module, the fourth CSPStage module, and the fifth CSPStage module serve as the inputs of the first convolutional layer, the second convolutional layer, and the third convolutional layer, respectively. The multi-task detection model training module is used to input the sample data and label data obtained by the sample data acquisition module into the multi-task detection model constructed by the multi-task detection model construction module to train the multi-task detection model; The multi-task real-time detection module is used to use the forward-facing camera of the sanitation vehicle to obtain real-time road images, input the real-time road images into the multi-task detection model trained by the multi-task detection model training module, and the multi-task detection model outputs the drivable area and garbage detection results.

7. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 5.

Images