Multi-task traffic panoramic perception method, device and computer equipment

Abstract

The application relates to a multi-task traffic panoramic perception method, device and computer equipment. The method comprises the following steps: performing shared feature extraction on a to-be-processed image through a feature extraction model, acquiring multi-layer image features, multi-layer semantic features and multi-layer positioning features; determining target driving area information, target lane line information and target object positioning information according to the multi-layer image features, the multi-layer semantic features and the multi-layer positioning features through a target detection model; selecting and tracking target objects based on the target object positioning information, and acquiring the motion state of the selected target objects according to the tracking result; and determining traffic panoramic perception information according to the target driving area information, the target lane line information, the target object positioning information and the motion state of the target objects. The method can improve the perception speed and expand the perception dimension.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a multi-task traffic panoramic perception method, apparatus, and computer device. Background Technology

[0002] Traffic panoramic perception, as a crucial component of autonomous driving, is maturing in terms of functional types and performance indicators. Traffic panoramic perception refers to the process by which a vehicle uses a perception system to process vast amounts of road and surface information from panoramic images in real time, assisting the vehicle in making safe and informed decisions while driving. Autonomous driving typically employs neural network models for traffic panoramic perception design, relying on the analysis of visual image information extracted from onboard sensors. By understanding the traffic scene through visual images, it assists the decision-making system in controlling vehicle actions to achieve safe driving objectives.

[0003] Traditional real-time perception models include single-task and multi-task models. Single-task models focus solely on object detection, thus limiting their ability to extract information from panoramic traffic images. Multi-task models often use a series of perception models to perform tasks one after another, but each of these models performs feature extraction independently, and this repetitive feature extraction consumes limited onboard computing resources and reduces device response time.

[0004] Current multi-task traffic panoramic perception still has certain limitations in analyzing complex road conditions, and its performance indicators still need further improvement. Summary of the Invention

[0005] Therefore, it is necessary to provide a multi-task traffic panoramic perception method, device, and computer equipment to improve the performance of perception models in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a multi-task traffic panoramic perception method. The method includes:

[0007] Through a feature extraction model, shared features are extracted from the image to be processed, obtaining multi-layer image features, multi-layer semantic features, and multi-layer localization features;

[0008] The target detection model determines the target driving area information, target lane line information, and target object location information based on multi-layer image features, multi-layer semantic features, and multi-layer localization features.

[0009] The target object is selected and tracked based on its location information, and its motion state is obtained based on the tracking results.

[0010] Based on the target driving area information, target lane line information, target object positioning information, and target object movement status, traffic panoramic perception information is determined.

[0011] In one embodiment, the feature extraction model includes: a feature map extraction network, a feature pyramid network, and a path aggregation network; the feature map extraction network contains multiple sequentially connected first downsampling layers, the feature pyramid network contains multiple sequentially connected upsampling layers, and the path aggregation network contains multiple sequentially connected second downsampling layers; and each upsampling layer has a corresponding first downsampling layer in the feature map extraction network and a corresponding second downsampling layer in the path aggregation network.

[0012] Through a feature extraction model, shared features are extracted from the image to be processed, obtaining multi-layer image features, multi-layer semantic features, and multi-layer localization features, including:

[0013] The first downsampling layer in the network is extracted by feature map extraction. Based on the input data of each first downsampling layer, the image to be processed is downsampled multiple times to obtain the multi-layer image features output by each first downsampling layer. The input data of the first first downsampling layer is the image to be processed, and the input data of each other first downsampling layer is the multi-layer image features output by the previous first downsampling layer.

[0014] Through each upsampling layer in the feature pyramid network, based on the input data of each upsampling layer, the multi-layer image features are upsampled multiple times to obtain the multi-layer semantic features output by each upsampling layer; wherein, the input data of the first upsampling layer is the multi-layer image features output by the first downsampling layer corresponding to the first upsampling layer, and the input data of each other upsampling layer is the multi-layer image features output by the first downsampling layer corresponding to that upsampling layer and the multi-layer semantic features output by the previous upsampling layer;

[0015] Through each second downsampling layer in the path aggregation network, the multi-layer semantic features are downsampled multiple times based on the input data of each second downsampling layer to obtain the multi-layer localization features output by each second downsampling layer. The input data of the first second downsampling layer is the multi-layer semantic features output by the upsampling layer corresponding to the first second downsampling layer, and the input data of each other second downsampling layer is the multi-layer semantic features output by the upsampling layer corresponding to the second downsampling layer and the multi-layer localization features output by the previous second downsampling layer.

[0016] In one embodiment, the feature map extraction network and the feature pyramid network have at least one first downsampling layer and at least one upsampling layer that are multi-channel fusion layers for multi-channel feature fusion processing.

[0017] In one embodiment, the target detection model includes: a region segmentation network, a lane line detection network, and an object detection network;

[0018] Using a target detection model, based on multi-layer image features, multi-layer semantic features, and multi-layer localization features, the target driving area information, target lane line information, and target object localization information are determined, including:

[0019] The target driving area information is detected by using a region segmentation network based on multi-layer image features;

[0020] The target lane line information is detected by using a lane line detection network based on multi-layer image features and multi-layer semantic features.

[0021] The target object detection network detects target object location information based on multi-layered localization features.

[0022] In one embodiment, the region segmentation network includes: a channel attention layer, a multi-channel fusion layer, and an upsampling network;

[0023] Using a region segmentation network based on multi-layer image features, target driving area information is detected, including:

[0024] Through the channel attention layer, adaptive feature optimization processing based on the attention mechanism is performed on multi-layer image features to obtain optimized image features;

[0025] The optimized image features are fused using a multi-channel fusion layer to obtain fused image features.

[0026] The features of the fused image are upsampled using an upsampling network to output a dual-channel grayscale image of the driving area.

[0027] Obtain the target driving area information based on the grayscale image of the driving area.

[0028] In one embodiment, the lane detection network includes: a channel attention layer, a multi-channel fusion layer, and a deconvolutional sampling network;

[0029] The lane line detection network detects target lane line information based on multi-layer image features and multi-layer semantic features, including:

[0030] Through the channel attention layer, adaptive feature optimization based on the attention mechanism is performed on multi-layer image features and multi-layer semantic features to obtain optimized semantic features;

[0031] The optimized semantic features are fused using a multi-channel fusion layer to obtain fused semantic features.

[0032] By using a deconvolution sampling network, the fused semantic features are deconvolutionally upsampled to output a single-channel lane line grayscale image.

[0033] Target lane line information is obtained from the grayscale image of lane lines in a single channel.

[0034] In one embodiment, the multi-channel fusion layer is used to perform bi-channel segmentation on the input data to obtain first bi-channel data and second bi-channel data; to perform bi-channel segmentation on the second bi-channel data to obtain first quartic channel data and second quartic channel data; to perform bi-channel segmentation on the second quartic channel data to obtain first octet channel data and second octet channel data; and to perform feature aggregation on the first bi-channel data, first quartic channel data, first octet channel data and second octet channel data to obtain the output data of the multi-channel fusion layer.

[0035] Wherein, if the multi-channel fusion layer is the first downsampling layer, then the input data of the multi-channel fusion layer is the input data of the first downsampling layer, and the output data of the multi-channel fusion layer is the multi-layer image features output by the first downsampling layer;

[0036] If the multi-channel fusion layer is an upsampling layer, then the input data of the multi-channel fusion layer is the input data of the upsampling layer, and the output data of the multi-channel fusion layer is the multi-layer semantic features output by the upsampling layer.

[0037] If the multi-channel fusion layer is located in the region segmentation network, the input data of the multi-channel fusion layer is the optimized image features, and the output data is the fused image features;

[0038] If the multi-channel fusion layer is located in the lane detection network, the input data of the multi-channel fusion layer is the optimized semantic features, and the output data is the fused semantic features.

[0039] In one embodiment, the channel attention layer is used to perform feature binary classification on the input data to obtain refined feature channel information and channel weight allocation information; based on the refined feature channel information and channel weight allocation information, adaptive feature optimization based on the attention mechanism is performed to obtain the output data;

[0040] If the channel attention layer is located in the region segmentation network, the input data of the channel attention layer is multi-layer image features, and the output data is optimized image features.

[0041] If the channel attention layer is located in the lane detection network, the input data of the channel attention layer consists of multi-layer image features and multi-layer semantic features, and the output data consists of optimized semantic features.

[0042] In one embodiment, the target detection network includes: a weight allocation layer, a target localization layer, and a target filtering layer;

[0043] Using a target detection network, based on multi-layered localization features, target location information is detected, including:

[0044] The multi-layered localization features are weighted by a weight allocation layer to obtain multi-layered weighted features;

[0045] Through the target localization layer, the initial localization information of the target object is locked based on multi-layer weighted features, where the target object includes pedestrians, non-motorized vehicles and motorized vehicles;

[0046] The target filtering layer uses a non-maximum suppression algorithm to filter the initial positioning information of the target and obtain the final positioning information of the target.

[0047] In one embodiment, the region segmentation network is optimized based on binary classification cross-entropy loss.

[0048] In one embodiment, the lane detection network is optimized based on a combination of focus loss, crossover ratio loss, and Tversky loss.

[0049] In one embodiment, the target detection network is optimized based on a combination of confidence loss, classification loss, and crossover ratio loss.

[0050] Secondly, this application also provides a multi-task traffic panoramic perception device. The device includes:

[0051] The feature extraction module is used to extract shared features from the image to be processed through a feature extraction model, thereby obtaining multi-layer image features, multi-layer semantic features, and multi-layer localization features.

[0052] The target detection module is used to determine the target driving area information, target lane line information, and target object location information by using a target detection model based on multi-layer image features, multi-layer semantic features, and multi-layer localization features.

[0053] The tracking module is used to select and track targets based on their location information, and to obtain the motion state of the selected targets based on the tracking results.

[0054] The perception module is used to determine the panoramic traffic perception information based on the target driving area information, target lane line information, target object positioning information, and target object motion state.

[0055] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0056] Through a feature extraction model, shared features are extracted from the image to be processed, obtaining multi-layer image features, multi-layer semantic features, and multi-layer localization features;

[0057] The target detection model determines the target driving area information, target lane line information, and target object location information based on multi-layer image features, multi-layer semantic features, and multi-layer localization features.

[0058] The target object is selected and tracked based on its location information, and its motion state is obtained based on the tracking results.

[0059] Based on the target driving area information, target lane line information, target object positioning information, and target object movement status, traffic panoramic perception information is determined.

[0060] Fourthly, this application also provides a computer-readable storage medium. This computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0061] Through a feature extraction model, shared features are extracted from the image to be processed, obtaining multi-layer image features, multi-layer semantic features, and multi-layer localization features;

[0062] The target detection model determines the target driving area information, target lane line information, and target object location information based on multi-layer image features, multi-layer semantic features, and multi-layer localization features.

[0063] The target object is selected and tracked based on its location information, and its motion state is obtained based on the tracking results.

[0064] Based on the target driving area information, target lane line information, target object positioning information, and target object movement status, traffic panoramic perception information is determined.

[0065] The aforementioned multi-task traffic panoramic perception method, device, and computer equipment acquire multi-layer image features, multi-layer semantic features, and multi-layer localization features through shared feature extraction. Based on the extracted features, target driving area information, target lane line information, and target object localization information are obtained, corresponding to the outputs of the driving area segmentation task, lane line detection task, and target object detection task, respectively, so that autonomous vehicles can regulate their driving area and avoid obstacles accordingly. Target objects are selected and tracked based on the target object localization information, and the motion state of the selected target objects is obtained, corresponding to the output of the target object tracking task, so that autonomous vehicles can avoid moving obstacles accordingly. Finally, based on the target driving area information, target lane line information, target object localization information, and target object motion state, traffic panoramic perception information is determined, thereby assisting the decision-making body of the autonomous vehicle in making vehicle control actions. The driving area segmentation, lane detection, and object detection tasks in this application can obtain their respective required features through a single shared feature extraction. Compared to traditional single-task perception models, this provides a more comprehensive perception of road traffic conditions, which is beneficial for autonomous vehicles to make accurate decisions. Compared to traditional multi-task perception models, it reduces the onboard computing resources and equipment response time required, thereby improving the reaction speed of autonomous vehicles and ultimately enhancing their safety. Furthermore, this application serializes the object tracking task based on the object detection task, eliminating the need for an additional feature extraction step. While maintaining real-time perception, it further expands the perception dimensions, thereby improving the overall traffic perception performance. This allows autonomous vehicles to combine the motion state of objects to make more accurate obstacle avoidance route planning, thus ensuring driving safety. Attached Figure Description

[0066] Figure 1 This is an application environment diagram of a multi-task traffic panoramic perception method in one embodiment;

[0067] Figure 2 This is a flowchart illustrating a multi-task traffic panoramic perception method in one embodiment;

[0068] Figure 3 This is a schematic diagram of the shared feature extraction process in one embodiment;

[0069] Figure 4 This is a schematic diagram of the data processing flow of a multi-channel fusion layer in one embodiment;

[0070] Figure 5 This is a schematic diagram of the data processing flow of a region segmentation network in one embodiment;

[0071] Figure 6 This is a schematic diagram of the data processing flow of a lane detection network in one embodiment;

[0072] Figure 7This is a schematic diagram of the data processing flow of a target detection network in one embodiment;

[0073] Figure 8 This is a flowchart illustrating the process of outputting the motion state of the selected target object in one embodiment;

[0074] Figure 9 This is a flowchart illustrating a multi-task traffic panoramic perception method in one embodiment;

[0075] Figure 10 This is a structural block diagram of a multi-task traffic panoramic perception device in one embodiment;

[0076] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0078] The multi-task traffic panoramic perception method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various intelligent vehicles. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0079] In one embodiment, such as Figure 2 As shown, a multi-task traffic panoramic perception method is provided, which is then applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps:

[0080] Step 202: Through the feature extraction model, shared features are extracted from the image to be processed to obtain multi-layer image features, multi-layer semantic features, and multi-layer localization features.

[0081] The image to be processed is a high-definition panoramic traffic image acquired in real time by the vehicle-mounted sensor. The image is converted into tensor format using methods such as torchvision.transforms function, which contains a large amount of traffic information about target objects such as motor vehicles, non-motor vehicles, pedestrians and traffic road surface.

[0082] The image to be processed is input into a feature extraction model, which performs shared feature extraction to obtain multi-layer image features, multi-layer semantic features, and multi-layer localization features. Shared feature extraction includes feature extraction steps and feature combination and mixing steps. Multi-layer image features are shallow feature information, containing more pixel information and coarse-grained information such as color, texture, edges, and corners. Multi-layer semantic features and multi-layer localization features are deep feature information, containing fine-grained information and more abstract information such as semantic information. Deep feature information can be used for effective identification, detection, and classification of traffic targets. Deep feature information supplements shallow feature information, and shallow feature information is also the basis for deep feature recognition. Multi-layer semantic features carry stronger semantics and weaker localization information. Multi-layer localization features carry stronger localization information and weaker semantic features. The multi-layer image features, multi-layer semantic features, and multi-layer localization features are obtained through non-linear transformations of a convolutional network.

[0083] In this embodiment, in order to achieve a real-time prediction effect of 28fps (frames per second), at least 28 images to be processed are input into the perception model every second.

[0084] Step 204: Using the target detection model, based on multi-layer image features, multi-layer semantic features, and multi-layer localization features, determine the target driving area information, target lane line information, and target object localization information.

[0085] The target detection model acts as the detection head for the panoramic perception task. Based on the multi-layer image features, multi-layer semantic features, and multi-layer localization features in the input target detection model, it outputs prediction results, namely target driving area information, target lane line information, and target object localization information.

[0086] In this embodiment, the target driving area information includes road area and road boundaries, which can refer to information such as the location, range, and direction of the target object's driving area in the traffic panoramic image. For example, it can be information about pedestrian walkway areas or lane driving areas. Target lane line information can refer to various ground marking lines, including lane line color, solid / dash type, and direction, such as solid yellow lines, dashed white lines, and straight lane lines. Target object location information can refer to the location information of the target object in the traffic panoramic image; the target object can be a pedestrian, motor vehicle, or non-motor vehicle.

[0087] In this embodiment, the target detection model can be divided into three detection heads according to their functions: drivable area segmentation, lane line detection, and target object detection, representing three tasks. The feature extraction model inputs multi-layer image features, multi-layer semantic features, and multi-layer localization features into the target detection model. Based on these multi-layer image features, multi-layer semantic features, and multi-layer localization features, the three detection heads ultimately output target drivable area information, target lane line information, and target object localization information, respectively.

[0088] Step 206: Select and track the target object based on the target object's location information, and obtain the motion state of the selected target object based on the tracking results.

[0089] In this embodiment, the target object can be any type of target object in the traffic panoramic image, and the motion state of the target object can refer to the motion trajectory, pose, and direction of movement of the target object.

[0090] Traffic panoramic images consist of continuous image frames. Target location information is obtained from individual frames through an object detection task. Based on this location information, targets are selected, with selection criteria including target type, such as selecting motor vehicles or pedestrians. After target selection, continuous image frames are used to track the target, thereby improving the obstacle avoidance accuracy of autonomous vehicles.

[0091] Step 208: Determine the traffic panoramic perception information based on the target driving area information, target lane line information, target object positioning information, and target object motion state.

[0092] By integrating target driving area information, target lane line information, target object location information, and target object motion status, a panoramic traffic perception information is obtained, including the current drivable area of ​​the vehicle, the location of road lane lines, road obstacles and other targets, and the motion status of motor vehicles and other targets, thereby assisting the decision-making body in autonomous vehicles to make safe and reasonable driving plans.

[0093] Specifically, by utilizing drivable areas and road lane markings, vehicles can be constrained in their direction of travel and driving regulations, enabling them to comply with laws and drive safely. By analyzing the position of road obstacles and the movement of vehicles, traffic congestion or smooth flow can be assessed, helping vehicles to effectively avoid obstacles.

[0094] In this embodiment, the driving area segmentation task, lane detection task, and object detection task can obtain their respective required features through a single shared feature extraction. Compared with traditional single-task perception models, this provides a more comprehensive perception of road traffic conditions, which is beneficial for autonomous vehicles to make accurate decisions. Compared with traditional multi-task perception models, it reduces the onboard computing resources and equipment response time, thereby improving the reaction speed of autonomous vehicles and thus enhancing autonomous driving safety. Furthermore, this application serializes the object tracking task based on the object detection task, eliminating the need for an additional feature extraction step. While maintaining real-time perception, it further expands the perception dimension, improving the perception performance of the overall traffic panorama. This allows autonomous vehicles to make more accurate obstacle avoidance driving plans based on the motion state of the objects, thereby ensuring driving safety.

[0095] Traditional single-task perception models typically perform feature extraction and then directly connect to the detection head. Therefore, when multiple single tasks are performed sequentially, feature extraction of the traffic panorama image needs to be repeated multiple times, consuming significant inference time. Based on this, this application designs a feature extraction model that enables feature sharing among multiple tasks. This ensures real-time prediction while also adapting to detection heads for drivable area segmentation, lane detection, and object detection.

[0096] Different task detection heads are adapted to different feature requirements. In one embodiment, to address the problem of numerous task features, in step 202, such as... Figure 3 As shown, the feature extraction model includes: a feature map extraction network, a feature pyramid network (FPN), and a path aggregation network (PAN). The feature map extraction network contains multiple sequentially connected first downsampling layers, the feature pyramid network contains multiple sequentially connected upsampling layers, and the path aggregation network contains multiple sequentially connected second downsampling layers. Each upsampling layer has a corresponding first downsampling layer in the feature map extraction network and a corresponding second downsampling layer in the path aggregation network.

[0097] In step 202, a shared feature extraction model is used to extract features from the image to be processed, obtaining multi-layer image features, multi-layer semantic features, and multi-layer localization features, including:

[0098] The first downsampling layer in the feature map extraction network is used to perform multiple downsampling processes on the image to be processed, thereby obtaining the multi-layer image features output by each first downsampling layer. The input data of the first first downsampling layer is the image to be processed, and the input data of each other first downsampling layer is the multi-layer image features output by the previous first downsampling layer.

[0099] By upsampling multiple times through each upsampling layer in the feature pyramid network, the multi-layer image features are obtained by upsampling multiple times to obtain the multi-layer semantic features output by each upsampling layer. The input data of the first upsampling layer is the multi-layer image features output by the first downsampling layer corresponding to the first upsampling layer, and the input data of each other upsampling layer is the multi-layer image features output by the first downsampling layer corresponding to the upsampling layer and the multi-layer semantic features output by the previous upsampling layer.

[0100] By using each second downsampling layer in the path aggregation network, the multi-layer semantic features are downsampled multiple times to obtain the multi-layer localization features output by each second downsampling layer. The input data of the first second downsampling layer is the multi-layer semantic features output by the upsampling layer corresponding to the first second downsampling layer, and the input data of each other second downsampling layer is the multi-layer semantic features output by the upsampling layer corresponding to the second downsampling layer and the multi-layer localization features output by the previous second downsampling layer.

[0101] Specifically, the feature map extraction network includes multiple first downsampling layers on the image to be processed, performing multiple downsampling operations, for example, six times the 2x downsampling here. The original image to be processed is a large image, which needs to be compressed by a convolutional network; downsampling is a method of image compression. The first downsampling layer can effectively increase the receptive field of the convolutional kernel, improve the model's global recognition ability, and thus extract shallow feature information of color, contour, and edges. Each layer performs downsampling processing through the first downsampling layer. The six times the 2x downsampling is performed iteratively, and each subsequent 2x downsampling is based on the result of the previous 2x downsampling. That is, the input data of the first downsampling layer is the output of the previous first downsampling layer, and the 2x downsampling feature M1, 4x downsampling feature M2, 8x downsampling feature M3, 16x downsampling feature M4, 32x downsampling feature M5, and 64x downsampling feature M6 are obtained sequentially. The above downsampling features M1, M2, M3, M4, M5, and M6 constitute the multi-layer image features.

[0102] Here, "upper" and "lower" refer to the order of data transmission. The output of one first downsampling layer is used as the input of another first downsampling layer. The output first downsampling layer is the "previous first downsampling layer" of the input first downsampling layer, and the input first downsampling layer is the "next first downsampling layer" of the output first downsampling layer.

[0103] By performing six 2x downsampling operations on the image information, the resulting multi-layer image features are known as shallow feature information.

[0104] The feature pyramid network comprises multiple upsampling layers, performing upsampling multiple times, for example, three times a 2x upsampling in this case. Upsampling is a feature aggregation technique that can both concatenate features of different scales obtained from upsampling to achieve feature fusion and facilitate the analysis of heatmaps. Heatmaps are used to identify points that have a significant impact on the final result. Each upsampling layer corresponds to a first downsampling layer. The 64x downsampling features M6, 32x downsampling features M5, 16x downsampling features M4, and 8x downsampling features M3 output from the feature map extraction network are sequentially input into the corresponding upsampling layers in the feature pyramid network. Lateral connections and three times a 2x upsampling are used to obtain the initial features P1, 2x upsampling features P2, 4x upsampling features P3, and 8x upsampling features P4. These upsampling features P1, P2, P3, and P4 constitute the multi-layer semantic features.

[0105] Specifically, the feature pyramid network receives a 64x downsampled feature M6 to obtain the initial feature P1. The upsampling layer receives the initial feature P1 and performs a 2x upsampling, concatenating the upsampled result with the 32x downsampled feature M5 through channels to obtain the 2x upsampled feature P2. Similarly, the next upsampling layer receives the 2x upsampled feature P2 and performs a 2x upsampling, concatenating the sampled result with the 16x downsampled feature M4 through channels to obtain the 4x upsampled feature P3. The next upsampling layer receives the 4x upsampled feature P3 and performs a 2x upsampling, concatenating the sampled result with the 8x downsampled feature M3 through channels to obtain the 8x upsampled feature P4.

[0106] Each upsampling operation yields a feature map at a different scale than before upsampling. By concatenating channels, features at different scales are fused, allowing for better utilization of the complementarity between features with different characteristics, combining the advantages of different features, and improving model performance.

[0107] Here, "up" and "down" refer to the order of data transmission. The output of one upsampling layer is used as the input of another upsampling layer. The output upsampling layer is the "upper upsampling layer" of the input upsampling layer, and the input upsampling layer is the "next upsampling layer" of the output upsampling layer.

[0108] Since the feature maps of 2x downsampled feature M1 and 4x downsampled feature M2 are large in size and lack semantic information, this embodiment does not connect them laterally to the feature pyramid network.

[0109] Multi-layer image features are upsampled three times to obtain multi-layer semantic features, which are deep feature information.

[0110] The path aggregation network includes multiple second downsampling layers, performing downsampling multiple times, for example, three times the 2x downsampling here. Feature compression through the second downsampling layers effectively increases the receptive field of the convolutional kernels, improving the model's global recognition ability and facilitating the extraction of deep feature information such as vehicle type. Each second downsampling layer corresponds to an upsampling layer. The 8x upsampling feature P4, 4x upsampling feature P3, 2x upsampling feature P2, and initial feature P1 are sequentially input into the corresponding second downsampling layer in the feature path aggregation network for processing. Lateral connections and three times the 2x downsampling are used to obtain the initial feature N1, 2x downsampling feature N2, 4x downsampling feature N3, and 8x downsampling feature N4. These downsampling features N1, N2, N3, and N4 constitute the multi-layer localization features.

[0111] Specifically, the path aggregation network receives an 8x upsampled feature P4 to obtain the initial feature N1. The second downsampling layer receives the initial feature N1 and performs a 2x downsampling. The downsampled result is concatenated with the 4x upsampled feature P3 to obtain the 2x downsampled feature N2. The next second downsampling layer receives the 2x downsampled feature N2, performs a 2x downsampling, and concatenates it with the 2x upsampled feature P2 to obtain the 4x downsampled feature N3. The next second downsampling layer receives the 4x downsampled feature N3, performs a 2x downsampling, and concatenates it with the initial feature P1 to obtain the 8x downsampled feature N4.

[0112] Each downsampling operation yields a feature map at a different scale than before downsampling. By concatenating channels, features at different scales are fused, allowing for better utilization of the complementarity between features with different characteristics, combining the advantages of different features, and improving model performance.

[0113] Here, "upper" and "lower" refer to the sequential relationship in the data transmission order. The output of a second downsampling layer is used as the input of another second downsampling layer. The output second downsampling layer is the "previous second downsampling layer" of the input second downsampling layer, and the input second downsampling layer is the "next second downsampling layer" of the output second downsampling layer.

[0114] By performing three rounds of downsampling on the multi-layer semantic features, the obtained multi-layer localization features are also deep feature information.

[0115] The feature extraction model disclosed in this embodiment extracts shallow feature information through a feature map extraction network, and then uses a feature pyramid network to fuse the shallow feature information to obtain semantic feature information. The path aggregation network extracts localization feature information based on the semantic feature information, which is then adapted to different task detection heads to achieve feature sharing. This lays the foundation for the design of a multi-task connected model, avoids repeated feature extraction, and improves the model's perception efficiency.

[0116] In one embodiment, the feature map extraction network and the feature pyramid network have at least one first downsampling layer and at least one upsampling layer, which employ a multi-channel fusion layer for feature fusion.

[0117] A multi-channel fusion layer is used to fuse feature maps of different scales obtained from different sampling layers to achieve fine semantic segmentation. Whether upsampling or downsampling, features from different levels (i.e., different scales) must be fused. In feature fusion, lower-level features have higher resolution and contain more location and detail information, but due to fewer convolutions, their semantics are lower and they contain more noise. Higher-level features have stronger semantic information, but their resolution is low and their ability to perceive details is poor. Therefore, feature fusion bridges the gap between low-level and high-level features, thereby significantly improving segmentation performance.

[0118] In one embodiment, such as Figure 4 The diagram illustrates the data processing flow of the multi-channel fusion layer. The multi-channel fusion layer performs bi-channel segmentation on the input data to obtain first bi-channel data and second bi-channel data; it then performs bi-channel segmentation on the second bi-channel data to obtain first quartic channel data and second quartic channel data; finally, it performs bi-channel segmentation on the second quartic channel data to obtain first octet channel data and second octet channel data; and finally, it performs feature aggregation on the first bi-channel data, first quartic channel data, first octet channel data, and second octet channel data to obtain the output data of the multi-channel fusion layer.

[0119] Binary channel segmentation uses two convolutional kernels to perform a convolution operation on the input data, resulting in two output channels. For example... Figure 4 As shown, assuming C is the input data, after one bi-channel division, two C / 2 values ​​are obtained, namely the first and second bi-channel data. Further bi-channel division of the second bi-channel data yields two C / 4 values, namely the first and second quarter-channel data. Finally, bi-channel division of the second quarter-channel data yields two C / 8 values, namely the first and second octave-channel data. In computer vision processing, image data is generally a single-channel grayscale image or an RGB (three primary colors) three-channel color image. After performing a convolution operation on the image data, more channels can be generated depending on the number of convolution kernels. Therefore, bi-channel division can be achieved by performing convolution with two convolution kernels. In image processing, given an input image, the pixels in a small region of the input image are weighted and averaged to output each corresponding pixel in the output image. The weights are defined by a function, which is the convolution kernel. Therefore, bi-channel division using two different convolution kernels can obtain richer feature information.

[0120] The two C / 8 values ​​obtained from the third bi-channel splitting process, the C / 4 value obtained from the second bi-channel splitting process, and the C / 2 value obtained from the first bi-channel splitting process are aggregated to obtain channel C. Finally, channel compression is performed to obtain channel C / 2. The multi-channel fusion layer performs three bi-channel splitting processes on the input data to fully extract feature information and perform more detailed hierarchical processing of channel information to maintain the accuracy of subsequent inference. At the same time, channel feature aggregation and channel compression effectively control the number of feature channels to maintain the speed of subsequent inference.

[0121] In one embodiment, if the multi-channel fusion layer is a first downsampling layer, then the input data of the multi-channel fusion layer is the input data of the first downsampling layer, and the output data of the multi-channel fusion layer is the multi-layer image features output by the first downsampling layer. If the multi-channel fusion layer is an upsampling layer, then the input data of the multi-channel fusion layer is the input data of the upsampling layer, and the output data of the multi-channel fusion layer is the multi-layer semantic features output by the upsampling layer.

[0122] In one embodiment, such as Figure 3 As shown, the first downsampling layer corresponding to the 64x downsampling feature M6, 32x downsampling feature M5, 16x downsampling feature M4 and 8x downsampling feature M3 in the feature map extraction network adopts a multi-channel fusion layer, and the three upsampling layers in the feature pyramid network all adopt a multi-channel fusion layer.

[0123] The feature map extraction network first uses conventional feature extraction techniques to initially obtain 2x downsampled features M1 and 4x downsampled features M2, and then uses a multi-channel fusion layer for subsequent downsampling processing. Conventional feature extraction techniques can be used to extract features based on specific needs, such as color or texture, extracting the most relevant features from the samples, reducing data dimensionality, and eliminating irrelevant and redundant features to improve the efficiency of subsequent data processing.

[0124] A single megapixel panoramic traffic image typically contains a wealth of information about vehicles, road conditions, and other traffic conditions. Traditional multi-task perception models often use a 640×640 resolution as the input size to achieve real-time prediction, requiring at least 28 images to be predicted per second. This embodiment utilizes a multi-channel fusion layer, achieving an inference speed of 91fps, far exceeding the required 28fps. Therefore, while maintaining an inference speed exceeding 28fps, the input size is expanded to 1280×1280 by sacrificing some inference speed. Simultaneously, with appropriate settings for each task's detection head, the inference accuracy for each task can be guaranteed.

[0125] Unlike traditional multi-task perception models, the feature extraction model in this embodiment increases the input size to 1280×1280 resolution to enhance information content and employs a multi-channel fusion layer to maintain inference speed. The feature map extraction network extracts features through six layers of 2x downsampling, with four of these layers using multi-channel fusion layers to further expand the model structure and improve shared feature information. In the feature pyramid network, three upsampling operations are performed using channel fusion layers to fuse the four shallow feature layers output by the feature map extraction network. Simultaneously, the channel fusion layer promotes efficient acquisition of semantic information from different feature layers, passing down deep, strong semantic information. In the path aggregation network, a feature pyramid is built using three downsampling operations to extract multi-layer semantic features of different sizes for detecting targets of various sizes, passing down deep, strong localization feature information.

[0126] This embodiment can be used with other model structures, not limited to a fixed network model structure. The innovative multi-task connection design proposed in this embodiment can be implemented based on various neural networks. For example, it is not limited to the number of upsampling and downsampling operations, nor to the specific methods of upsampling or downsampling.

[0127] In one embodiment, the target detection model includes: a region segmentation network, a lane detection network, and an object detection network. The target detection model determines target driving area information, target lane information, and target object location information based on multi-layer image features, multi-layer semantic features, and multi-layer localization features. This includes: detecting target driving area information based on multi-layer image features using the region segmentation network; detecting target lane information based on multi-layer image features and multi-layer semantic features using the lane detection network; and detecting target object location information based on multi-layer localization features using the object detection network.

[0128] In this embodiment, as Figure 3 As shown, the target driving area information is obtained based on multi-layer image features, specifically a 64x downsampling feature M6. The 64x downsampling feature M6 is input into a region segmentation network, which detects and obtains the target driving area information. The target lane line information is obtained based on multi-layer image features and multi-layer semantic features, i.e., a combination of deep and shallow feature information, specifically a 32x downsampling feature M6 and a 3x upsampling feature P2. The 32x downsampling feature M6 and the 3x upsampling feature P2 are input together into a lane line detection network to detect and obtain the target lane line information. The target object localization information is obtained through multi-layer localization features, specifically initial feature N1, 2x downsampling feature N2, 4x downsampling feature N3, and 8x downsampling feature N4. The initial feature N1, 2x downsampling feature N2, 4x downsampling feature N3, and 8x downsampling feature N4 are input into a target detection network to detect and obtain the target object localization information.

[0129] In one embodiment, such as Figure 5 As shown, the region segmentation network includes: a channel attention layer, a multi-channel fusion layer, and an upsampling network. Based on multi-layer image features, the network detects target driving area information, including: performing adaptive feature optimization processing on multi-layer image features using the channel attention layer to obtain optimized image features; performing feature fusion processing on the optimized image features using the multi-channel fusion layer to obtain fused image features; performing upsampling processing on the fused image features using the upsampling network to output a dual-channel driving area grayscale image; and obtaining target driving area information based on the driving area grayscale image.

[0130] In this network, the input data of the multi-channel fusion layer is the optimized image features, and the output data is the fused image features.

[0131] Multi-layer image features are input into the region segmentation network. For example, a 1280×1280 resolution 3-channel RGB image, after processing by a feature map extraction network, yields multi-layer image features of N×1024×20×20 dimensions, where N is the number of input images. In the design of the region segmentation network, feature analysis with adaptive weight allocation is first performed through a channel attention layer, and the channels are compressed to 512 dimensions to obtain optimized image features. Then, a multi-channel fusion layer is used to aggregate the optimized image features, performing main feature analysis and processing for the drivable region segmentation task to obtain fused image features. Finally, an upsampling network transforms the fused image features to N×2×320×320 dimensions to obtain a dual-channel drivable region grayscale image. Based on the different grayscale levels on the dual-channel drivable region grayscale image, the target drivable region information can be obtained.

[0132] This embodiment performs region segmentation on multi-layer image features in shallow feature maps, quickly and effectively delineating road areas, effectively simplifying the model structure and reducing inference time, improving model speed, and providing a reference for planning the driving area and direction of vehicles on roads.

[0133] In one embodiment, such as Figure 6 As shown, the lane detection network includes: a channel attention layer, a multi-channel fusion layer, and a deconvolution sampling network. Based on multi-layer image features and multi-layer semantic features, the network detects target lane information, including: using the channel attention layer to perform adaptive feature optimization based on an attention mechanism to obtain optimized semantic features; using the multi-channel fusion layer to perform feature fusion processing on the optimized semantic features to obtain fused semantic features; using the deconvolution sampling network to perform deconvolution upsampling processing on the fused semantic features to output a single-channel lane line grayscale image; and obtaining target lane information based on the single-channel lane line grayscale image.

[0134] In the lane detection network, the input data of the multi-channel fusion layer is optimized semantic features, and the output data is fused semantic features.

[0135] Multi-layer image features and multi-layer semantic features are input into the lane detection network. For example, a 1280×1280 resolution 3-channel RGB image, after processing by a feature extraction model, yields multi-layer image features and multi-layer semantic features of N×512×40×40 dimensions, where N is the number of input images. A channel attention layer performs adaptive weight allocation feature analysis on the input features and compresses the channels to 256 dimensions to obtain optimized semantic features. A multi-channel fusion layer then aggregates the optimized semantic features to obtain fused semantic features, refining the features for subsequent processing. Finally, a deconvolutional sampling network transforms the fused semantic features to N×1×320×320 dimensions to obtain a single-channel lane line grayscale image. Based on the different grayscale levels on the lane line grayscale image, the target lane line information can be obtained.

[0136] This embodiment combines lane line feature textures in shallow feature information with implicit scene information in deep feature information to accurately detect and segment lane lines. At the same time, it uses an adaptive weight allocation through a channel attention layer to identify accurate lane line features, thus assisting the decision-making system in complying with lane line laws and regulations and driving safely.

[0137] In one embodiment, the channel attention layer is used to perform feature binary classification on the input data to obtain refined feature channel information and channel weight allocation information; based on the refined feature channel information and channel weight allocation information, adaptive feature optimization based on the attention mechanism is performed to obtain the output data.

[0138] Specifically, if the channel attention layer is located in the region segmentation network, the input data of the channel attention layer is multi-layer image features, and the output data is optimized image features; if the channel attention layer is located in the lane detection network, the input data of the channel attention layer is multi-layer image features and multi-layer semantic features, and the output data is optimized semantic features.

[0139] The channel attention layer extracts features using two different convolutional kernels to achieve binary classification. One kernel is used to obtain refined feature channel information, while the other is used to obtain feature channel weight allocation information. The channel weight allocation information is then processed through a flattening layer and a fully connected layer to obtain one-dimensional data, which is then multiplied with the refined feature channel information to output adaptive feature optimization data based on the attention mechanism.

[0140] The channel attention layer maintains the independence between features, and at the same time, by refining the number of channels, it reduces the computational parameters, thereby improving the computational speed.

[0141] In one embodiment, such as Figure 7 As shown, the target detection network includes a weight allocation layer, a target localization layer, and a target filtering layer. Based on multi-layer localization features, the network detects target location information, including: weighting the multi-layer localization features using the weight allocation layer to obtain multi-layer weighted features; locking the initial location information of the target based on the multi-layer weighted features using the target localization layer, where the target includes pedestrians, non-motorized vehicles, and motorized vehicles; and filtering the initial location information of the target using a non-maximum suppression algorithm using the target filtering layer to obtain the final location information of the target.

[0142] Non-maximum suppression (NMS) algorithms find local maxima and eliminate the remaining values ​​in their neighborhood. During target detection, anchor boxes are used to predict target locations. A large number of anchor boxes are generated at the same target location, and these anchor boxes may overlap. The NMS algorithm sorts all anchor boxes according to their confidence scores and then removes other anchor boxes whose Intersection over Union (IOU) exceeds a predetermined threshold, thus finding the target anchor box and eliminating redundant anchor boxes.

[0143] The target object detection network uses multi-layer localization features, specifically: Figure 3 The initial feature N1, 2x downsampled feature N2, 4x downsampled feature N3, and 8x downsampled feature N4 are used to enrich the feature information of target detection, corresponding to target information of various sizes, and thus having better feature extraction capabilities. In this embodiment, the feature map size of the input target detection network corresponds to the multi-layer localization features N1, N2, N3, and N4, which are N×128×160×160, N×256×80×80, N×512×40×40, and N×1024×20×20, respectively, where N is the number of input images.

[0144] In this embodiment, the weight allocation layer assigns weights to the multi-layer localization features of the input target object detection network according to their size, obtaining multi-layer weighted features. In this embodiment, the weights of N1, N2, N3, and N4 are 4.0, 1.0, 0.6, and 0.2, respectively. By allocating weights, the target object detection network prioritizes locating the most obvious target objects, then enriches the perception details, and locates smaller target objects, thereby improving the target object localization accuracy. The target localization layer uses the multi-layer weighted features to lock the position of the target object. Specifically, in this embodiment, anchor frames are used in the target localization layer to locate the target position, thereby obtaining the initial localization information of the target object, which consists of multiple anchor frame-defined areas. The target selection layer selects the target anchor frames using a non-maximum suppression algorithm, thereby determining the final localization information of the target object.

[0145] The target detection task also includes classifying the located targets. In traditional multi-task models, target detection only considers motor vehicles. However, in reality, traffic targets are not only diverse in type, but pedestrians and non-motorized vehicles are also densely distributed in scenarios such as intersections, making them crucial traffic detection targets. Ignoring the detection of pedestrians and non-motorized vehicles poses a significant safety threat to autonomous driving. Therefore, in this embodiment, motor vehicles, pedestrians, and non-motorized vehicles are classified through a target localization layer, improving the perception dimension. Furthermore, anchor frames are used to detect target information of various sizes, achieving high-precision target detection and helping vehicles avoid pedestrians and maintain safe distances.

[0146] In one embodiment, the region segmentation network is optimized based on binary classification cross-entropy loss.

[0147] Specifically, in the drivable region segmentation task, binary cross-entropy loss (BCEloss) is used to distinguish drivable regions from other regions. The formula for calculating binary cross-entropy loss is L(gt,pred)=-(gt×ln(pred)+(1-gt)×ln(1-pred)), where gt is the true distribution value of drivable regions and pred is the predicted distribution of drivable regions. The region segmentation network is trained until the binary cross-entropy loss converges to a certain threshold condition, thereby improving the prediction accuracy of the region segmentation network for drivable regions.

[0148] In one embodiment, the lane detection network is optimized based on a combination of focus loss, crossover ratio loss, and Tversky loss.

[0149] In lane detection tasks, the loss function of a lane detection network is expressed as: L = a × L focalloss +b×L iouloss +×L diceloss , where L focalloss For focal loss, L iouloss For intersection-union loss (IOUloss), L dicelossLet a, b, and c represent the weights of the focus loss, intersection-over-union (IoU) loss, and Tversky loss, respectively. In this embodiment, the weights of a, b, and c are 1.0, 0.2, and 0.2, respectively. Adjusting the weights of the focus loss, IoU loss, and Tversky loss controls the different scaling effects of positive and negative samples on the loss. For example, with 100 positive samples and 10,000 negative samples, setting the weight of the positive samples to 10,000 / 10,100 multiplies the contribution of positive samples to the loss by a coefficient of 10,000 / 10,100, while multiplying the contribution of negative samples by a smaller coefficient of 100 / 10,100. This effectively controls the model to focus more on the impact of positive samples on the loss function. By using weighting, the loss value can be controlled from the perspective of the number of positive and negative samples, thus solving the problem of imbalanced positive and negative samples to some extent.

[0150] In one embodiment, the target detection network is optimized based on a combination of confidence loss, classification loss, and crossover ratio loss.

[0151] In the target detection network, after locating the target object, the target localization layer also needs to classify the target object. The specific classification method is as follows: the classification label is encoded with a one-hot code, and the target object is classified into three types of labels: motor vehicle, pedestrian and non-motor vehicle using binary cross-entropy loss.

[0152] The nonmaximum suppression algorithm in the target selection layer sorts the initial positioning information of each target with confidence, then deletes anchor frames with IOU greater than 0.7, removes duplicate anchor frames, and finally obtains the target anchor frames, thus determining the final positioning information of the target.

[0153] Furthermore, the target localization layer needs to be trained to optimize the anchor box locking range. During training, in addition to requiring the anchor box to have an IOU greater than a threshold of 0.5, the fitness must also be less than a threshold of 4. That is, the length and width of the anchor box must meet certain similarity requirements with the real target object to be considered a correct match for the predicted target. The formula for calculating the fitness r is... Among them, gt (w,h) representing the actual length and width of the target object, pred (w,h) This represents the length and width of the model's anchor frame.

[0154] The target detection network is optimized based on confidence loss, classification loss, and crossover ratio loss to achieve the purpose of classifying target objects and improve the accuracy of target object localization.

[0155] In one embodiment, considering that the target detection network also includes the detection of non-motorized vehicles and pedestrians, multi-target category detection and binary cross-entropy loss are added to the target detection network. Furthermore, a self-collected training dataset is added to the original training dataset BDD100K, greatly enriching the detection categories for non-motorized vehicles and pedestrians, filling the gap in the lack of non-motorized vehicle targets in the BDD100K dataset, and meeting the target detection needs of more complex traffic situations. Simultaneously, data augmentation is performed on the training data, increasing sample quantity balancing and improving the usage strategy of the data augmentation techniques Mosaic and Mixup. This strategy uses the `random.random` function to control that in each data augmentation process, 60% of the samples use Mosaic and 40% use Mixup. For example, the `random.random` function defines the probability of random data "1" as 60% and the probability of random data "0" as 40%. When a sample's data corresponds to a random data "1", Mosaic is used; when a sample's data corresponds to a random data "0", Mixup is used. This strategy can improve the accuracy of lane detection, object detection, and drivable region segmentation tasks. At the same time, randomly adding a simpler mixup can speed up the training process compared to using Mosaci exclusively.

[0156] In one embodiment, such as Figure 8 As shown, step 204, which involves selecting and tracking the target object based on its location information and obtaining its motion state according to the tracking results, is achieved through a target tracking network connected to the target detection network. The target tracking network selects the motor vehicle as the tracking object from the target object location information obtained from the target object detection network and extracts its motion state. By tracking the target object through the target tracking network, and leveraging the excellent performance of target object detection, the network further locates the target object, eliminating a large number of false positives and false negatives.

[0157] The object tracking network is an improvement on the YOLOv7_tiny model. Specifically, it utilizes a multi-channel fusion layer for feature extraction, accelerating the model's inference speed. During training, vehicle images are resized to 64×64 pixels before being fed into the network. Cross-entropy loss (CELOSS) is used to classify over 700 vehicle types in the training dataset. The cross-entropy loss function is expressed as L(gt,pred) = -gt × ln(pred), where gt is the true vehicle label and pred is the model's predicted vehicle label. After training, feature information is extracted from the object tracking network for subsequent feature comparison.

[0158] Feature comparison uses cosine similarity. If the cosine similarity between the vehicle features in two images is less than 0.2, the two vehicles are considered similar; otherwise, they are not. The cosine similarity is calculated as follows: Here, x1 and x2 represent the vectors of the two images for feature comparison, and dist(x1,x2) represents the cosine distance between the two images. The cosine similarity is represented by the cosine distance.

[0159] In this embodiment, after processing the 1280×1280 image through a feature extraction model and a target detection model, the target location information corresponding to the motor vehicle in the target location information output by the target detection network is selected. Based on this, the motor vehicle area is selected from the image, the image is extracted, and the image size is adjusted to obtain an image of size 64×64, which is then sent to the target tracking network.

[0160] In the target object tracking network, detected vehicle target bounding boxes are recorded. Kalman filtering is used to record and analyze the vehicle's sequence number. Based on the target object's location information, cosine similarity is used for sequence number matching, helping to record the features of the same vehicle across different frames. Even if a vehicle temporarily disappears from the view, its information is still recorded. Simultaneously, Kalman filtering is used for motion state prediction to achieve vehicle tracking. Kalman filtering predicts motion state through a state equation, where the state equation is x... k = k x k-1 + k u k Among them, x k Indicates the current state of the target object; x k-1 Indicates the state of the target object at the previous time; F k The state transition matrix is ​​determined based on the motion law of the target object; B k For the system control matrix, u k B is the system control vector. k u k This term represents the impact of external forces on the motion state. If there are no external forces, this term can be ignored. The motion of the motor vehicle can be estimated and its motion state obtained based on the state equation.

[0161] In traffic panoramic perception, vehicle tracking not only helps maintain safe vehicle distances and understand road conditions, but also provides a good real-time statistical overview of road traffic conditions, enabling detailed tracking of traffic congestion.

[0162] like Figure 9As shown, in one embodiment, a panoramic traffic image is input into the feature extraction model. The target detection model mainly handles two tasks: detecting traffic targets and detecting road surface conditions. Traffic target detection includes motor vehicle detection, non-motor vehicle detection, and pedestrian detection, with motor vehicle tracking added to the motor vehicle detection task. Road surface condition detection includes drivable area segmentation and lane detection tasks. All the above tasks output corresponding detection information, which constitutes traffic perception information, used to assist the autonomous driving decision-making system in making safe driving decisions.

[0163] Compared to traditional multi-task network models, this application innovatively adds a motor vehicle tracking task while maintaining real-time inference, and improves the detection capabilities for non-motorized vehicles and pedestrians. It optimizes data augmentation strategies, improves feature extraction methods, and proposes a multi-channel fusion layer, effectively enhancing the model's accuracy. This application better meets the detection needs of real-world traffic conditions, providing more comprehensive information on traffic targets and road surface conditions, helping vehicles maintain safety, avoid pedestrians and non-motorized vehicles, plan driving areas, comply with road regulations, and achieve safe driving.

[0164] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0165] Based on the same inventive concept, this application also provides a multi-task traffic panoramic perception device for implementing the multi-task traffic panoramic perception method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the multi-task traffic panoramic perception device provided below can be found in the limitations of the multi-task traffic panoramic perception method described above, and will not be repeated here.

[0166] In one embodiment, such as Figure 10 As shown, a multi-task traffic panoramic perception device is provided, including: a feature extraction module 1002, a target detection module 1004, a tracking module 1006, and a perception module 1008, wherein:

[0167] The feature extraction module 1002 is used to extract shared features from the image to be processed through a feature extraction model, thereby obtaining multi-layer image features, multi-layer semantic features, and multi-layer localization features.

[0168] The target detection module 1004 is used to determine the target driving area information, target lane line information and target object location information by using a target detection model based on multi-layer image features, multi-layer semantic features and multi-layer positioning features.

[0169] The tracking module 1006 is used to select and track targets based on their location information, and to obtain the motion state of the selected targets based on the tracking results.

[0170] The perception module 1008 is used to determine the traffic panoramic perception information based on the target driving area information, target lane line information, target object positioning information and target object motion state.

[0171] The feature extraction model includes a feature map extraction network, a feature pyramid network, and a path aggregation network. The feature map extraction network contains multiple sequentially connected first downsampling layers, the feature pyramid network contains multiple sequentially connected upsampling layers, and the path aggregation network contains multiple sequentially connected second downsampling layers. Each upsampling layer has a corresponding first downsampling layer in the feature map extraction network and a corresponding second downsampling layer in the path aggregation network.

[0172] The feature extraction model performs shared feature extraction on the image to be processed, obtaining multi-layer image features, multi-layer semantic features, and multi-layer localization features. This includes: extracting each first downsampling layer in the feature map extraction network, and performing multiple downsampling processes on the image to be processed based on the input data of each first downsampling layer to obtain the multi-layer image features output by each first downsampling layer; wherein, the input data of the first first downsampling layer is the image to be processed, and the input data of each other first downsampling layer is the multi-layer image features output by the previous first downsampling layer.

[0173] Through each upsampling layer in the feature pyramid network, based on the input data of each upsampling layer, the multi-layer image features are upsampled multiple times to obtain the multi-layer semantic features output by each upsampling layer. The input data of the first upsampling layer is the multi-layer image features output by the first downsampling layer corresponding to the first upsampling layer, and the input data of each other upsampling layer is the multi-layer image features output by the first downsampling layer corresponding to that upsampling layer and the multi-layer semantic features output by the previous upsampling layer.

[0174] Through each second downsampling layer in the path aggregation network, the multi-layer semantic features are downsampled multiple times based on the input data of each second downsampling layer to obtain the multi-layer localization features output by each second downsampling layer. The input data of the first second downsampling layer is the multi-layer semantic features output by the upsampling layer corresponding to the first second downsampling layer, and the input data of each other second downsampling layer is the multi-layer semantic features output by the upsampling layer corresponding to the second downsampling layer and the multi-layer localization features output by the previous second downsampling layer.

[0175] The feature map extraction network and the feature pyramid network each have at least one first downsampling layer and at least one upsampling layer that are multi-channel fusion layers for multi-channel feature fusion processing.

[0176] The target detection model includes a region segmentation network, a lane detection network, and an object detection network. Based on multi-layer image features, multi-layer semantic features, and multi-layer localization features, the model determines target driving area information, target lane line information, and target object localization information. Specifically, the region segmentation network detects target driving area information based on multi-layer image features; the lane detection network detects target lane line information based on multi-layer image features and multi-layer semantic features; and the object detection network detects target object localization information based on multi-layer localization features.

[0177] The region segmentation network comprises a channel attention layer, a multi-channel fusion layer, and an upsampling network. Based on multi-layer image features, the network detects target driving region information, including: performing adaptive feature optimization based on an attention mechanism on the multi-layer image features through the channel attention layer to obtain optimized image features; performing feature fusion processing on the optimized image features through the multi-channel fusion layer to obtain fused image features; performing upsampling processing on the fused image features through the upsampling network to output a dual-channel grayscale image of the driving region; and obtaining target driving region information based on the grayscale image.

[0178] The lane detection network comprises a channel attention layer, a multi-channel fusion layer, and a deconvolutional sampling network. Based on multi-layer image features and multi-layer semantic features, the network detects target lane information, including: using the channel attention layer to perform adaptive feature optimization based on an attention mechanism to obtain optimized semantic features; using the multi-channel fusion layer to perform feature fusion processing on the optimized semantic features to obtain fused semantic features; and using the deconvolutional sampling network to perform deconvolutional upsampling processing on the fused semantic features to output a single-channel lane line grayscale image; finally, the target lane information is obtained based on the single-channel lane line grayscale image.

[0179] The multi-channel fusion layer is used to perform bi-channel segmentation on the input data to obtain first bi-channel data and second bi-channel data; to perform bi-channel segmentation on the second bi-channel data to obtain first quartic data and second quartic data; to perform bi-channel segmentation on the second quartic data to obtain first octet data and second octet data; and to perform feature aggregation on the first bi-channel data, first quartic data, first octet data and second octet data to obtain the output data of the multi-channel fusion layer.

[0180] In this configuration, if the multi-channel fusion layer is a first downsampling layer, its input data is the same as that of the first downsampling layer, and its output data is the multi-layer image features output by the first downsampling layer. If the multi-channel fusion layer is an upsampling layer, its input data is the same as that of the upsampling layer, and its output data is the multi-layer semantic features output by the upsampling layer. If the multi-channel fusion layer is located in a region segmentation network, its input data is optimized image features, and its output data is fused image features. If the multi-channel fusion layer is located in a lane detection network, its input data is optimized semantic features, and its output data is fused semantic features.

[0181] The channel attention layer is used to perform feature binary classification on the input data to obtain refined feature channel information and channel weight allocation information; based on the refined feature channel information and channel weight allocation information, adaptive feature optimization based on the attention mechanism is performed to obtain the output data.

[0182] Specifically, if the channel attention layer is located in the region segmentation network, the input data of the channel attention layer is multi-layer image features, and the output data is optimized image features; if the channel attention layer is located in the lane detection network, the input data of the channel attention layer is multi-layer image features and multi-layer semantic features, and the output data is optimized semantic features.

[0183] The target detection network comprises a weight allocation layer, a target localization layer, and a target filtering layer. Based on multi-layered localization features, the network detects target location information, including: weighting the multi-layered localization features using the weight allocation layer to obtain multi-layered weighted features; using the target localization layer to lock in the initial location information of the target based on the multi-layered weighted features, where the target includes pedestrians, non-motorized vehicles, and motorized vehicles; and using the target filtering layer to filter the initial location information of the target using a non-maximum suppression algorithm to obtain the final location information of the target.

[0184] The region segmentation network is optimized based on binary classification cross-entropy loss.

[0185] The lane detection network is obtained by combining and optimizing the focus loss, intersection-over-union ratio loss and Tversky loss.

[0186] The target detection network is obtained by combining and optimizing confidence loss, classification loss and crossover ratio loss.

[0187] Each module in the aforementioned multi-task traffic panoramic perception device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0188] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores image data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-task traffic panoramic perception method.

[0189] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0190] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method embodiment described above.

[0191] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method embodiment described above.

[0192] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method embodiments described above.

[0193] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0194] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0195] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0196] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-task traffic panoramic perception method, characterized in that, The method includes: A feature extraction model is used to extract shared features from the image to be processed, obtaining multi-layer image features, multi-layer semantic features, and multi-layer localization features. The multi-layer image features are shallow feature information, while the multi-layer semantic features and multi-layer localization features are deep feature information. The feature extraction model includes a feature map extraction network, a feature pyramid network, and a path aggregation network. Each of the feature map extraction network, feature pyramid network, and path aggregation network includes a multi-channel fusion layer. The feature map extraction network contains multiple sequentially connected first downsampling layers, the feature pyramid network contains multiple sequentially connected upsampling layers, and the path aggregation network contains multiple sequentially connected second downsampling layers. Each upsampling layer has a corresponding first downsampling layer in the feature map extraction network and a corresponding second downsampling layer in the path aggregation network. At least one first downsampling layer and at least one upsampling layer in the feature map extraction network and the feature pyramid network are implemented using the multi-channel fusion layer. The target detection model determines target driving area information, target lane line information, and target object location information based on the multi-layer image features, the multi-layer semantic features, and the multi-layer localization features. The target detection model includes a region segmentation network, a lane line detection network, and a target object detection network. The region segmentation network is configured to detect the target driving area information based on the multi-layer image features. The lane line detection network is configured to detect the target lane line information based on the multi-layer image features and the multi-layer semantic features. The target object detection network is configured to detect the target object location information based on the multi-layer localization features. Each of the region segmentation network, lane line detection network, and target object detection network includes a multi-channel fusion layer. The multi-channel fusion layer of the lane detection network is used to perform binary channel segmentation on the input data to obtain first binary channel data and second binary channel data; to perform binary channel segmentation on the second binary channel data to obtain first quartic channel data and second quartic channel data; to perform binary channel segmentation on the second quartic channel data to obtain first octet channel data and second octet channel data; and to perform feature aggregation on the first binary channel data, the first quartic channel data, the first octet channel data, and the second octet channel data to obtain the output data of the multi-channel fusion layer of the lane detection network. Based on the target object's location information, the target object is selected and tracked, and based on the tracking results, the motion state of the selected target object is obtained. Traffic panoramic perception information is determined based on the target driving area information, the target lane line information, the target object positioning information, and the target object's motion state.

2. The method according to claim 1, characterized in that, The process involves using a feature extraction model to extract shared features from the image to be processed, obtaining multi-layer image features, multi-layer semantic features, and multi-layer localization features, including: The feature map is used to extract each of the first downsampling layers in the network. Based on the input data of each of the first downsampling layers, the image to be processed is downsampled multiple times to obtain the multi-layer image features output by each of the first downsampling layers. The input data of the first first downsampling layer is the image to be processed, and the input data of each of the other first downsampling layers is the multi-layer image features output by the previous first downsampling layer. Through each upsampling layer in the feature pyramid network, based on the input data of each upsampling layer, the multi-layer image features are upsampled multiple times to obtain the multi-layer semantic features output by each upsampling layer; wherein, the input data of the first upsampling layer is the multi-layer image features output by the first downsampling layer corresponding to the first upsampling layer, and the input data of each other upsampling layer is the multi-layer image features output by the first downsampling layer corresponding to the upsampling layer and the multi-layer semantic features output by the previous upsampling layer; Through each of the second downsampling layers in the path aggregation network, the multi-layer semantic features are downsampled multiple times based on the input data of each second downsampling layer to obtain the multi-layer localization features output by each second downsampling layer; wherein, the input data of the first second downsampling layer is the multi-layer semantic features output by the upsampling layer corresponding to the first second downsampling layer, and the input data of each other second downsampling layer is the multi-layer semantic features output by the upsampling layer corresponding to the second downsampling layer and the multi-layer localization features output by the previous second downsampling layer.

3. The method according to claim 1, characterized in that, The region segmentation network also includes a channel attention layer and an upsampling network; The target driving area information is detected using the region segmentation network based on the multi-layer image features, including: Through the channel attention layer, the multi-layer image features are subjected to adaptive feature optimization processing based on the attention mechanism to obtain optimized image features; The optimized image features are fused using the multi-channel fusion layer to obtain fused image features; The fused image features are upsampled using the upsampling network to output a dual-channel grayscale image of the driving area. The target driving area information is obtained based on the grayscale image of the driving area.

4. The method according to claim 3, characterized in that, The lane detection network also includes a channel attention layer and a deconvolutional sampling network; The lane detection network detects the target lane information based on the multi-layer image features and the multi-layer semantic features, including: Through the channel attention layer, adaptive feature optimization based on the attention mechanism is performed on the multi-layer image features and multi-layer semantic features to obtain optimized semantic features; The optimized semantic features are fused using the multi-channel fusion layer to obtain fused semantic features. The deconvolutional sampling network is used to perform deconvolutional upsampling on the fused semantic features to output a single-channel lane line grayscale image. The target lane line information is obtained based on the grayscale image of the lane lines in the single channel.

5. The method according to claim 4, characterized in that, If the multi-channel fusion layer is the first downsampling layer, then the input data of the multi-channel fusion layer is the input data of the first downsampling layer, and the output data of the multi-channel fusion layer is the multi-layer image features output by the first downsampling layer; If the multi-channel fusion layer is the upsampling layer, then the input data of the multi-channel fusion layer is the input data of the upsampling layer, and the output data of the multi-channel fusion layer is the multi-layer semantic features output by the upsampling layer; If the multi-channel fusion layer is located in the region segmentation network, then the input data of the multi-channel fusion layer is the optimized image features, and the output data is the fused image features; If the multi-channel fusion layer is located in the lane detection network, then the input data of the multi-channel fusion layer is the optimized semantic features, and the output data is the fused semantic features.

6. The method according to claim 4, characterized in that, The channel attention layer is used to perform feature binary classification on the input data to obtain refined feature channel information and channel weight allocation information; based on the refined feature channel information and the channel weight allocation information, adaptive feature optimization based on the attention mechanism is performed to obtain output data; Wherein, if the channel attention layer is located in the region segmentation network, the input data of the channel attention layer is multi-layer image features, and the output data is optimized image features; If the channel attention layer is located in the lane detection network, then the input data of the channel attention layer consists of multi-layer image features and multi-layer semantic features, and the output data consists of optimized semantic features.

7. The method according to claim 1, characterized in that, The target detection network includes: a weight allocation layer, a target localization layer, and a target filtering layer; The target object detection network detects the target object location information based on the multi-layer localization features, including: The weight allocation layer performs weighted allocation on the multi-layer localization features to obtain multi-layer weighted features; The target positioning layer is used to lock the initial positioning information of the target object based on the multi-layer weighted features, wherein the target object includes pedestrians, non-motorized vehicles and motorized vehicles; The target filtering layer uses a non-maximum suppression algorithm to filter the initial positioning information of the target object, thereby obtaining the final positioning information of the target object.

8. The method according to claim 1, characterized in that, The region segmentation network is optimized based on binary cross-entropy loss.

9. The method according to claim 1, characterized in that, The lane detection network is obtained by optimizing a combination of focus loss, crossover ratio loss, and Tversky loss.

10. The method according to claim 1, characterized in that, The target detection network is obtained by optimizing a combination of confidence loss, classification loss, and cross-union ratio loss.

11. A multi-task traffic panoramic perception device, characterized in that, The device includes: The feature extraction module is used to extract shared features from the image to be processed through a feature extraction model, obtaining multi-layer image features, multi-layer semantic features, and multi-layer localization features. The multi-layer image features are shallow feature information, while the multi-layer semantic features and multi-layer localization features are deep feature information. The feature extraction model includes a feature map extraction network, a feature pyramid network, and a path aggregation network. Each of the feature map extraction network, feature pyramid network, and path aggregation network includes a multi-channel fusion layer. The feature map extraction network contains multiple sequentially connected first downsampling layers, the feature pyramid network contains multiple sequentially connected upsampling layers, and the path aggregation network contains multiple sequentially connected second downsampling layers. Each upsampling layer has a corresponding first downsampling layer in the feature map extraction network and a corresponding second downsampling layer in the path aggregation network. At least one first downsampling layer and at least one upsampling layer in the feature map extraction network and the feature pyramid network are implemented using the multi-channel fusion layer. The target detection module is used to determine target driving area information, target lane line information, and target object location information based on the multi-layer image features, the multi-layer semantic features, and the multi-layer localization features using a target detection model. The target detection model includes a region segmentation network, a lane line detection network, and a target object detection network. The region segmentation network is configured to detect the target driving area information based on the multi-layer image features; the lane line detection network is configured to detect the target lane line information based on the multi-layer image features and the multi-layer semantic features; and the target object detection network is configured to detect the target object location information based on the multi-layer localization features. The region segmentation network... Both the lane detection network and the target object detection network include the multi-channel fusion layer. The multi-channel fusion layer of the lane detection network is used to perform bi-channel segmentation on the input data to obtain first bi-channel data and second bi-channel data; to perform bi-channel segmentation on the second bi-channel data to obtain first quartic data and second quartic data; to perform bi-channel segmentation on the second quartic data to obtain first octet data and second octet data; and to perform feature aggregation on the first bi-channel data, the first quartic data, the first octet data, and the second octet data to obtain the output data of the multi-channel fusion layer of the lane detection network. The tracking module is used to select and track the target object based on the target object's positioning information, and to obtain the motion state of the selected target object based on the tracking results. The perception module is used to determine traffic panoramic perception information based on the target driving area information, the target lane line information, the target object positioning information, and the target object's motion state.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images