A vehicle-road network-connected 3D target detection method and device that integrates roadside information

Abstract

The present invention discloses a vehicle-road network 3D target detection method and device that integrates roadside information, and belongs to the field of autonomous driving perception technology. The method includes: obtaining monitoring data of corresponding monitoring areas sent by multiple monitoring points at the current moment; performing feature fusion on the monitoring data corresponding to the multiple monitoring points according to the regional position, and obtaining the fusion features of the union area corresponding to the multiple monitoring areas at the current moment; determining the intersection area of ​​the union area corresponding to the multiple monitoring areas and the service area, and performing feature fusion on the fusion features of the intersection area at the current moment with the maintenance features of the intersection area at the previous moment, and obtaining the maintenance features in the service area at the current moment; using the maintenance features in the service area at the current moment to provide feature feedback to the target vehicle, so that the target vehicle uses the feedback features to perform 3D target detection. The present invention can ensure the real-time performance of the vehicle when performing 3D target detection and improve the accuracy of the 3D target detection results.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving perception technology, and in particular to a vehicle-road network 3D target detection method and device that integrates roadside information. Background Art

[0002] In the field of autonomous driving perception, 3D object detection technology is the key to enabling vehicles to accurately perceive their surroundings. However, in the process of 3D object detection, in the face of complex traffic scenes, the target is blocked due to the limited field of view of the sensor, which affects the accuracy and reliability of 3D object detection.

[0003] In related technologies, the problem of target obstruction is solved by using vehicle-road collaborative detection methods, and the problem of single-vehicle limited perception is effectively solved through information interaction between vehicles and between vehicles and roads. However, the amount of data that needs to be processed and integrated during the implementation of vehicle-road collaboration is very large, and the real-time and accuracy of target detection results cannot be guaranteed. Summary of the invention

[0004] The present invention provides a vehicle-road network 3D target detection method and device integrating roadside information. The technical solution is as follows:

[0005] On the one hand, a vehicle-road network 3D target detection method integrating roadside information is provided, wherein the server is used to maintain three-dimensional voxel features in a service area, and the method includes:

[0006] Acquire monitoring data of the corresponding monitoring area sent by multiple monitoring points at the current moment; the monitoring points include vehicles and roadside facilities, and the monitoring points are located in the service area or in the edge area of ​​the service area; the monitoring data includes laser radar point cloud data and multi-view two-dimensional image data;

[0007] The monitoring data corresponding to the multiple monitoring points are subjected to feature fusion according to the regional positions to obtain the fusion features of the union area corresponding to the multiple monitoring areas at the current moment; the multiple monitoring areas correspond to the multiple monitoring points one by one;

[0008] Determine the intersection area of ​​the union area corresponding to the multiple monitoring areas and the service area, and fuse the fusion features of the intersection area at the current moment with the maintenance features of the intersection area at the previous moment to obtain the maintenance features of the service area at the current moment;

[0009] The maintenance features at the current moment in the service area are used to provide feature feedback to the target vehicle, so that the target vehicle performs 3D target detection using the feedback features.

[0010] On the other hand, a vehicle-road network 3D target detection method integrating roadside information is provided, which is applied to a vehicle side, and the method includes:

[0011] Acquire monitoring data at the current moment in the monitoring area, and send the monitoring data at the current moment to the server, so that the server uses any of the above-mentioned vehicle-road network 3D target detection methods integrating roadside information to perform feature feedback; the monitoring area and the service area of ​​the server have an overlapping area; the monitoring data includes laser radar point cloud data and multi-view two-dimensional image data;

[0012] Receive the feedback features sent by the server, and use the feedback features to perform 3D object detection.

[0013] On the other hand, a vehicle-road network 3D target detection device integrating roadside information is provided, which is applied to a server, and the server is used to maintain three-dimensional voxel features in a service area, and the device includes:

[0014] An acquisition unit, used to acquire monitoring data of a corresponding monitoring area sent by multiple monitoring points at the current moment; the monitoring points include vehicles and roadside facilities, and the monitoring points are located in the service area or in the edge area of ​​the service area; the monitoring data includes laser radar point cloud data and multi-view two-dimensional image data;

[0015] A first fusion unit is used to perform feature fusion on the monitoring data corresponding to the multiple monitoring points according to the regional positions, so as to obtain the fusion features of the union area corresponding to the multiple monitoring areas at the current moment; the multiple monitoring areas correspond to the multiple monitoring points one by one;

[0016] A second fusion unit is used to determine the intersection area of ​​the union area corresponding to the multiple monitoring areas and the service area, and fuse the fusion features of the intersection area at the current moment with the maintenance features of the intersection area at the previous moment to obtain the maintenance features in the service area at the current moment;

[0017] The feedback unit is used to use the maintenance features in the service area at the current moment to provide feature feedback to the target vehicle, so that the target vehicle uses the feedback features to perform 3D target detection.

[0018] On the other hand, a vehicle-road network 3D target detection device integrating roadside information is provided, which is applied to a vehicle side, and the device includes:

[0019] A monitoring unit, used for acquiring monitoring data at the current moment in a monitoring area, and sending the monitoring data at the current moment to a server, so that the server uses any of the above-mentioned vehicle-road network 3D target detection methods integrating roadside information to perform feature feedback; the monitoring area and the service area of ​​the server have an overlapping area; the monitoring data includes laser radar point cloud data and multi-view two-dimensional image data;

[0020] The detection unit is used to receive the feedback features sent by the server and use the feedback features to perform 3D target detection.

[0021] On the other hand, a computer device is provided, which includes a memory and a processor, the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the steps of the above-mentioned vehicle-road network 3D target detection method that integrates roadside information.

[0022] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned vehicle-road network 3D target detection method integrating roadside information are implemented.

[0023] The technical solution provided by the present invention can at least bring the following beneficial effects:

[0024] The server maintains the features in a service area, and the vehicles and roadside facilities located in the service area or in the edge area of ​​the service area are used as monitoring points. The monitoring data in the monitoring area is sent to the server in real time. The server with strong computing power and high processing efficiency fuses the monitoring data corresponding to multiple monitoring points, thereby obtaining the fusion features of the union area corresponding to multiple monitoring areas at the current moment. By determining the intersection area of ​​the union area and the service area, and fusing the fusion features of the intersection area at the current moment with the maintenance features at the previous moment, the maintenance features of the service area at the current moment can be obtained, wherein the maintenance features are three-dimensional voxel features. It can be seen that the server maintains the three-dimensional voxel features in a service area, which can provide the maintenance features required in the service area for multiple vehicles in the service area, and can ensure the real-time performance of multiple vehicles when performing 3D target detection; and by fusing the fusion features at the current moment with the maintenance features at the previous moment, the temporal features can be maintained, thereby improving the accuracy of the 3D target detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 This is a flow chart of a vehicle-road network 3D target detection method integrating roadside information provided by an embodiment of the present invention;

[0027] Figure 2is a flow chart of another vehicle-road network 3D target detection method integrating roadside information provided by an embodiment of the present invention;

[0028] Figure 3 This is a structural diagram of a vehicle-road network 3D target detection device integrating roadside information provided by an embodiment of the present invention;

[0029] Figure 4 It is a structural diagram of another vehicle-road network 3D target detection device integrating roadside information provided by an embodiment of the present invention;

[0030] Figure 5 It is a hardware architecture diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0032] Please refer to Figure 1 , a vehicle-road network 3D target detection method integrating roadside information provided by an embodiment of the present invention is applied to a server, and the server is used to maintain three-dimensional voxel features in a service area. The method includes:

[0033] Step 100, obtaining monitoring data of the corresponding monitoring area sent by multiple monitoring points at the current moment; the monitoring points include vehicles and roadside facilities, and the monitoring points are located in the service area or in the edge area of ​​the service area; the monitoring data includes laser radar point cloud data and multi-view two-dimensional image data;

[0034] Step 102, feature fusion is performed on the monitoring data corresponding to the multiple monitoring points according to the regional positions, so as to obtain the fusion features of the union area corresponding to the multiple monitoring areas at the current moment; the multiple monitoring areas correspond to the multiple monitoring points one by one;

[0035] Step 104, determining the intersection area of ​​the union area corresponding to the plurality of monitoring areas and the service area, and fusing the fusion feature of the intersection area at the current moment with the maintenance feature of the intersection area at the previous moment to obtain the maintenance feature of the service area at the current moment;

[0036] Step 106 , using the maintenance features at the current moment in the service area to provide feature feedback to the target vehicle, so that the target vehicle performs 3D target detection using the feedback features.

[0037] In the embodiment of the present invention, the server maintains the features in a service area, and the vehicles and roadside facilities located in the service area or the edge area of ​​the service area are used as monitoring points. The monitoring data in the monitoring area is sent to the server in real time. The server with strong computing power and high processing efficiency fuses the monitoring data corresponding to multiple monitoring points, thereby obtaining the fusion features of the union area corresponding to the multiple monitoring areas at the current moment. By determining the intersection area of ​​the union area and the service area, and fusing the fusion features of the intersection area at the current moment with the maintenance features at the previous moment, the maintenance features of the service area at the current moment can be obtained, wherein the maintenance features are three-dimensional voxel features. It can be seen that the server maintains the three-dimensional voxel features in a service area, which can provide the maintenance features required in the service area for multiple vehicles in the service area, and can ensure the real-time performance of multiple vehicles when performing 3D target detection; and by fusing the fusion features at the current moment with the maintenance features at the previous moment, the temporal features can be maintained, thereby improving the accuracy of the 3D target detection results.

[0038] Furthermore, compared with the traditional method in which the vehicle itself saves the features at different times separately, which wastes storage space, in the embodiment of the present invention, the server maintains the three-dimensional voxel features in the service area, and each time the features are fused, the fused features at the current moment are fused with the maintained features at the previous moment. At each moment, only the maintenance features at the current moment in the service area need to be stored, which greatly saves storage space.

[0039] Described below Figure 1 How the various steps are performed.

[0040] First, with respect to step 100, the monitoring data of the corresponding monitoring area sent by multiple monitoring points at the current moment is obtained.

[0041] In the embodiment of the present invention, the service area of ​​the server refers to a geographical area, and the size of the service area can be set according to actual needs.

[0042] In the embodiment of the present invention, the monitoring points include vehicles and roadside facilities. If the monitoring point is located in the edge area outside the service area, the monitoring area of ​​the monitoring point will also have an area intersection with the service area. Therefore, the monitoring points located in the service area and in the edge area outside the service area can send monitoring data to the server.

[0043] Among them, the roadside facilities can be monitoring equipment on fixed signal lights, street lights, road signs, etc. The vehicle can be in a driving state or in a non-driving state. Similarly, the vehicle is also equipped with monitoring equipment.

[0044] In an embodiment of the present invention, the monitoring device may include a laser radar and a multi-view camera. The laser radar is used to obtain point cloud data, and the camera is used to obtain two-dimensional image data. Thus, each monitoring point can use the monitoring device to obtain monitoring data in real time, and the monitoring data includes: laser radar point cloud data and multi-view two-dimensional image data. In one implementation, the number of the multi-view cameras can be 6, deployed in 6 different directions of the monitoring point.

[0045] In the embodiment of the present invention, after the monitoring point obtains the monitoring data at the current moment by using the monitoring device, the monitoring data can be sent to the server in at least the following two ways:

[0046] The first method: directly send monitoring data to the server;

[0047] The second method: Process and compress the monitoring data and send it to the server.

[0048] The above two methods are described below respectively.

[0049] In the first method, the laser radar point cloud data and multi-view 2D image data are sent directly to the server. In this method, the data sent to the server is the original collected data, and the data content is more accurate.

[0050] In the second method, the monitoring point can first process the monitoring data according to the following steps A1-A4 to obtain voxel fusion features:

[0051] A1: Extract features from the LiDAR point cloud data to obtain three-dimensional point cloud features, and convert the three-dimensional point cloud features into voxel expressions to obtain the first voxel features;

[0052] In an embodiment of the present invention, a backbone network can be used to extract features from laser radar point cloud data to obtain three-dimensional point cloud features. The backbone network can at least be a PointPillars network. The three-dimensional point cloud features are converted into voxelized expressions to obtain first voxel features.

[0053] A2: For each 2D image data under each viewing angle, the following steps are performed: using a discrete depth set to associate with each pixel in the 2D image corresponding to the 2D image data, forming associated points corresponding to multiple pixels one by one, and obtaining pseudo point cloud data; extracting features from the pseudo point cloud data to obtain 3D pseudo point cloud features;

[0054] The discrete depth set includes multiple depth values, which gradually increase according to the interval depth. The multiple depth values ​​are associated with each pixel in the two-dimensional image, that is, each pixel is associated with the depth value in the discrete depth set to form an associated point, thereby creating three-dimensional pseudo point cloud data. When extracting features from pseudo point cloud data, the backbone network can also be used to achieve this.

[0055] A3: Fuse the three-dimensional pseudo point cloud features corresponding to multiple perspectives, and convert the fused three-dimensional pseudo point cloud features into voxel expressions to obtain the second voxel features;

[0056] In the embodiment of the present invention, when fusing the three-dimensional pseudo point cloud features corresponding to multiple viewing angles, this can be achieved by combining the internal and external parameters of multiple cameras.

[0057] A4: Fusing the first voxel feature and the second voxel feature to obtain a voxel fusion feature.

[0058] In one embodiment of the present invention, when fusing the first voxel feature and the second voxel feature, the first voxel feature and the second voxel feature can be stacked and input into a pooling layer to obtain a preliminary fused voxel feature, and then the preliminary fused voxel feature is input into a neural network model composed of a convolutional layer and an activation layer for voxel feature refinement to enhance the semantic understanding ability of the neural network model for data of different modalities and obtain the final fused voxel fusion feature.

[0059] In the second mode, after the monitoring point processes the monitoring data to obtain the voxel fusion feature, in order to reduce the amount of data required for transmission, reduce the transmission cost and transmission delay, in one embodiment of the present invention, the monitoring point can also use a compression module to compress the voxel fusion feature. Specifically, the compression module may include two parts: a channel compressor and a spatial compressor. The channel compressor may be composed of multiple convolutional layers, using a 1×1×1 convolution kernel with an output channel number of C / 2 to reduce the number of channels of the feature, so as to output the channel-compressed voxel fusion feature, which is used as the input of the spatial compressor. Wherein, C is the number of channels before compression. The convolution kernel size in the spatial compressor may be 2×2×2. By controlling the moving stride of the convolution kernel to 2, the size of the voxel fusion feature in different dimensions can be halved to obtain the spatially compressed voxel fusion feature. Finally, the channel-compressed voxel fusion feature and the spatially compressed voxel fusion feature are sent to the server as compressed voxel fusion features.

[0060] Regardless of the above method, the server will receive the monitoring data sent by the monitoring point at the current moment.

[0061] In an embodiment of the present invention, if the monitoring point directly sends the laser radar point cloud data and multi-view two-dimensional image data using the first method mentioned above, then after the server receives the monitoring data, it is necessary to obtain the voxel fusion features according to the above steps A1-A4. If the monitoring point processes the monitoring data using the second method mentioned above to obtain the voxel fusion features, and the voxel fusion features are sent uncompressed, then after the server receives the monitoring data, it is not necessary to process the voxel fusion features; if the monitoring point processes the monitoring data using the second method mentioned above to obtain the voxel fusion features, and the monitoring point sends the voxel fusion features after compressing them, then after receiving the monitoring data, the server needs to decompress the monitoring data sent by each monitoring point to restore the voxel fusion features. The decompression process is the inverse operation of the compression process of the monitoring point to complete the restoration of the voxel fusion features.

[0062] Then, for step 102, the monitoring data corresponding to the multiple monitoring points are feature fused according to the regional positions to obtain the fusion features of the union area corresponding to the multiple monitoring areas at the current moment; the multiple monitoring areas correspond one-to-one to the multiple monitoring points.

[0063] In one embodiment of the present invention, this step can be implemented by the following steps:

[0064] B1: Take two monitoring points from multiple monitoring points without replacement, and perform feature fusion on the monitoring data of the two currently taken monitoring points according to the regional positions to obtain the initial fusion features of the union area corresponding to the two monitoring points;

[0065] B2: Determine whether there are any remaining monitoring points that have not been feature fused. If so, take out a monitoring point from the remaining monitoring points without replacement, and fuse the monitoring data of the monitoring point with the initial fusion feature obtained after the monitoring point was taken out last time according to the regional position to obtain the initial fusion feature of the updated union area, and continue to execute step B2; if not, take the initial fusion feature of the updated union area as the fusion feature of the union area corresponding to multiple monitoring areas.

[0066] Since the monitoring areas of different monitoring points are located in different locations, feature fusion needs to be performed according to the regional location to ensure the effectiveness of data fusion.

[0067] In one embodiment of the present invention, the feature fusion according to the regional position may specifically include:

[0068] The overlapping areas of the two areas that need to be fused are fused according to the self-attention mechanism to obtain the fusion features of the overlapping areas; the three-dimensional voxel features of the non-overlapping areas are directly used as the fusion features of the non-overlapping areas.

[0069] Since there may be overlapping areas in the monitoring areas of different monitoring points, when the monitoring data is subjected to feature fusion according to the regional position, feature fusion of the overlapping areas can enhance the feature expression of the overlapping areas and increase the accuracy of 3D target detection.

[0070] In one embodiment of the present invention, feature fusion according to the self-attention mechanism may include the following steps C1-C3:

[0071] C1: Perform linear transformation on the two regions where feature fusion is required;

[0072] Among them, the linear transformations of the first area and the second area are:

[0073] , , ;

[0074] , , ;

[0075] Among them, Q is the query vector, K is the key vector, V is the value vector, and W Q , W K , W V is the weight matrix corresponding to Q, K, V, f 1 , f 2 These are the characteristics of the first and second regions respectively.

[0076] C2: Use the linear transformation results corresponding to the two regions to calculate the corresponding attention weights respectively;

[0077] Attention weights for the first and second regions , The calculation formula is as follows:

[0078]

[0079]

[0080] in, is the scaling factor.

[0081] C3: Calculate the fusion feature using the attention weight and linear transformation result.

[0082] Among them, the fusion feature for:

[0083]

[0084]

[0085]

[0086] Among them, α and β are weight parameters, which represent the contribution of each output feature to the fusion result, and can be learned during the neural network training process.

[0087] Thus, the fusion features of the overlapping areas can be obtained.

[0088] In an embodiment of the present invention, overlapping areas are fused according to a self-attention mechanism rather than by position points. After acquiring the features of interest in the overlapping area, the same features of interest in the two areas are fused, thereby further improving the effectiveness of the fused features.

[0089] In the embodiment of the present invention, by continuously fusing the features of the data monitored by different monitoring points in the overlapping area, the feature expression of the overlapping area can be enhanced and the accuracy of 3D target detection can be improved.

[0090] Next, for step 104, the intersection area of ​​the union area corresponding to the multiple monitoring areas and the service area is determined, and the fusion features of the intersection area at the current moment are fused with the maintenance features of the intersection area at the previous moment to obtain the maintenance features of the service area at the current moment.

[0091] In the embodiment of the present invention, considering that there may be defects at a certain position point in the monitoring data of each frame, which affects the 3D target detection result, the position points where the defects may exist in the monitoring data of adjacent frames are different. Therefore, in order to eliminate the possible defects, it is necessary to fuse the fusion features at the current moment with the maintenance features at the previous moment to eliminate the defects through feature complementation and improve the accuracy of the 3D target detection results.

[0092] In the embodiment of the present invention, since the server is used to maintain the three-dimensional voxel features in the service area, and the union area corresponding to the multiple monitoring areas may have a partial area that does not cover the service area, in order to achieve feature maintenance of the service area, this step may include:

[0093] D1: The fusion features of the intersection area at the current moment are fused with the maintenance features of the intersection area at the previous moment according to the self-attention mechanism to obtain the maintenance features of the intersection area at the current moment;

[0094] D2: taking the maintenance features of the area in the service area other than the intersection area at the previous moment as the maintenance features at the current moment.

[0095] Among them, the method of performing feature fusion according to the self-attention mechanism in step D1 is the same as that in steps C1-C3, and will not be repeated here.

[0096] For areas other than the intersection area in the service area, there is no fusion feature at the current moment, so the maintenance feature at the previous moment can be used directly to ensure the transmission of data features in time series.

[0097] It can be understood that in order to ensure that new monitoring data is available at every moment to update the maintenance characteristics of the entire service area, multiple roadside facilities can be evenly set as monitoring points so that the union area of ​​the monitoring areas of the monitoring data sent by multiple roadside facilities can cover the entire service area.

[0098] In addition, by continuously fusing the fusion features of the current moment with the maintenance features of the previous moment, the temporal sequence of the features can be maintained and the accuracy of 3D target detection can be improved.

[0099] In the embodiment of the present invention, the server is only responsible for maintaining the maintenance features of the current moment in the service area. Compared with the vehicle end separately saving the features at different moments, this embodiment can greatly save feature storage space.

[0100] Finally, for step 106, the maintenance features in the service area at the current moment are used to provide feature feedback to the target vehicle, so that the target vehicle performs 3D target detection using the feedback features.

[0101] In one embodiment of the present invention, when the service end provides feature feedback to the target vehicle, the feedback feature may include at least the following two situations:

[0102] The first case: the maintenance characteristics of the service area at the current moment are used as feedback characteristics;

[0103] The second case: the maintenance feature at the current moment corresponding to the detection range required for 3D target detection of the target vehicle is used as the feedback feature, and the detection range is within the service area.

[0104] For the first case mentioned above, after the maintenance features of the entire service area at the current moment are sent to the target vehicle as feedback features, the target vehicle can perform 3D target detection on the entire service area, or it can realize 3D target detection within its own detection range based on the maintenance features in the entire service area.

[0105] For the second situation mentioned above, since the service area is a relatively large area, the detection range of the vehicle for 3D target detection is a smaller area. In order to improve data transmission efficiency, the server can specifically include: determining the detection area where the target vehicle needs to perform 3D target detection, and the detection area is located in the service area, then obtaining the fusion feature corresponding to the detection area from the maintenance feature at the current moment in the service area, and feeding back the fusion feature corresponding to the detection area to the target vehicle. Among them, the detection area where the target vehicle needs to perform 3D target detection can be determined by the requested feature area of ​​the target vehicle, and can also be determined based on the coordinate position and driving direction of the target vehicle.

[0106] The following processing procedures in the embodiment of the present invention: processing the monitoring data to obtain voxel fusion features, using the voxel fusion features to perform feature fusion of the union area corresponding to multiple monitoring areas, performing feature fusion of the fusion features at the current moment in the intersection area with the maintenance features at the previous moment, and performing 3D target detection on the target vehicle using feedback features, can all be implemented using a neural network, and the neural networks corresponding to the above-mentioned different processing processes can be trained jointly to ensure the accuracy of the 3D target detection results of the final target vehicle.

[0107] Please refer to Figure 2 The embodiment of the present invention further provides a vehicle-road network 3D target detection method integrating roadside information, which is applied to a vehicle side, and the method includes:

[0108] Step 200, obtaining monitoring data at the current moment in the monitoring area, and sending the monitoring data at the current moment to the server, so that the server uses the vehicle-road network 3D target detection method integrating roadside information described in any of the above embodiments to perform feature feedback; the monitoring area and the service area of ​​the server have an overlapping area; the monitoring data includes laser radar point cloud data and multi-view two-dimensional image data;

[0109] Step 202: receiving the feedback features sent by the server, and performing 3D object detection using the feedback features.

[0110] In the embodiment of the present invention, since the computing power and processing power of the server are strong, the features in the union area corresponding to multiple monitoring points can be fused, thereby obtaining the fused features in the union area at the current moment, and the fused features in the intersection area of ​​the union area and the service area at the current moment are fused with the maintenance features at the previous moment to obtain the maintenance features in the service area at the current moment, which can ensure the real-time performance of the vehicle when performing 3D target detection; and the server can maintain the timing characteristics by continuously fusing the features of the overlapping areas and fusing them with the maintenance features at the previous moment, thereby improving the accuracy of the detection results when the vehicle performs 3D target detection.

[0111] In one embodiment of the present invention, after obtaining the monitoring data of the current moment in the monitoring area, the vehicle end can directly send the monitoring data to the server end, or process and compress the monitoring data before sending it to the server end. The specific processing and compression methods can refer to the embodiments of the server end, which will not be repeated here.

[0112] In one embodiment of the present invention, after receiving the feedback features sent by the server, the vehicle side can directly use the feedback features to perform 3D target detection; in addition, considering that the monitoring data is sent from the vehicle side to the server side, the server side processes the data, and the server side sends the processed feedback features to the vehicle side, there is a certain data delay. In order to further improve the accuracy of the 3D target detection results, it can also include: feature fusion of the feedback features with the monitoring data of the vehicle side at the current moment again, so as to use the obtained feature data for 3D target detection.

[0113] Please refer to Figure 3 Based on the same inventive concept as the above method embodiment, an embodiment of the present invention provides a vehicle-road network 3D target detection device integrating roadside information, which is applied to a server, and the server is used to maintain three-dimensional voxel features in a service area. The device includes:

[0114] The acquisition unit 300 is used to acquire the monitoring data of the corresponding monitoring area sent by multiple monitoring points at the current moment; the monitoring points include vehicles and roadside facilities, and the monitoring points are located in the service area or in the edge area of ​​the service area; the monitoring data includes laser radar point cloud data and multi-view two-dimensional image data;

[0115] The first fusion unit 302 is used to perform feature fusion on the monitoring data corresponding to the multiple monitoring points according to the regional positions, so as to obtain the fusion features of the union area corresponding to the multiple monitoring areas at the current moment; the multiple monitoring areas correspond to the multiple monitoring points one by one;

[0116] The second fusion unit 304 is used to determine the intersection area of ​​the union area corresponding to the multiple monitoring areas and the service area, and fuse the fusion features of the intersection area at the current moment with the maintenance features of the intersection area at the previous moment to obtain the maintenance features of the service area at the current moment;

[0117] The feedback unit 306 is used to use the maintenance features in the service area at the current moment to provide feature feedback to the target vehicle, so that the target vehicle uses the feedback features to perform 3D target detection.

[0118] In one embodiment of the present invention, the first fusion unit is specifically used for:

[0119] Take out two monitoring points from multiple monitoring points without replacement, and perform feature fusion on the monitoring data of the two currently taken out monitoring points according to the regional positions to obtain the initial fusion features of the union area corresponding to the two monitoring points;

[0120] Determine whether there are any remaining monitoring points that have not undergone feature fusion. If so, take out a monitoring point from the remaining monitoring points without replacement, and perform feature fusion on the monitoring data of the monitoring point and the initial fusion feature obtained after the monitoring point was taken out last time according to the regional position to obtain the initial fusion feature of the updated union area, and continue to execute the determination of whether there are any remaining monitoring points that have not undergone feature fusion; if not, use the initial fusion feature of the updated union area as the fusion feature of the union area corresponding to the multiple monitoring areas.

[0121] In one embodiment of the present invention, when the first fusion unit performs the feature fusion according to the regional position, the first fusion unit specifically includes:

[0122] The overlapping areas of the two areas that need to be fused are fused according to the self-attention mechanism to obtain the fusion features of the overlapping areas; the three-dimensional voxel features of the non-overlapping areas are directly used as the fusion features of the non-overlapping areas.

[0123] In one embodiment of the present invention, the second fusion unit is specifically used for:

[0124] The fusion feature of the intersection area at the current moment is fused with the maintenance feature of the intersection area at the previous moment according to the self-attention mechanism to obtain the maintenance feature of the intersection area at the current moment;

[0125] The maintenance features of the area in the service area except the intersection area at the previous moment are used as the maintenance features at the current moment.

[0126] In one embodiment of the present invention, the feature fusion according to the self-attention mechanism specifically includes: performing linear transformation on two areas where feature fusion is required; calculating corresponding attention weights using the linear transformation results corresponding to the two areas; and calculating the fused features using the attention weights and the linear transformation results.

[0127] Please refer to Figure 4 Based on the same inventive concept as the above method embodiment, an embodiment of the present invention provides a vehicle-road network 3D target detection device integrating roadside information, which is applied to a vehicle side. The device includes:

[0128] The monitoring unit 400 is used to obtain the monitoring data of the monitoring area at the current moment, and send the monitoring data at the current moment to the server, so that the server uses the vehicle-road network 3D target detection method integrating roadside information described in any of the above embodiments to perform feature feedback; the monitoring area and the service area of ​​the server have an overlapping area; the monitoring data includes laser radar point cloud data and multi-view two-dimensional image data;

[0129] The detection unit 402 is used to receive the feedback features sent by the server and perform 3D object detection using the feedback features.

[0130] It should be noted that the vehicle-road networked 3D target detection device that integrates roadside information provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the vehicle-road networked 3D target detection device that integrates roadside information provided in the above embodiment and the vehicle-road networked 3D target detection method embodiment that integrates roadside information belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0131] The embodiment of the present application also provides a computer device, please refer to Figure 5 The computer device includes a processor and a memory, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the memory, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the vehicle-road network 3D target detection method that integrates roadside information provided by the above-mentioned method embodiments.

[0132] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the vehicle-road network 3D target detection method that integrates roadside information provided in the above-mentioned method embodiments.

[0133] An embodiment of the present application also provides a computer program product, which includes a computer program. A processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the vehicle-road network 3D target detection method that integrates roadside information as described in any of the above embodiments.

[0134] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0135] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0136] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0137] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A vehicle-road network 3D target detection method integrating roadside information, characterized in that: Applied to a server, the server is used to maintain three-dimensional voxel features in a service area, and the method includes: Acquire monitoring data of the corresponding monitoring area sent by multiple monitoring points at the current moment; the monitoring points include vehicles and roadside facilities, and the monitoring points are located in the service area or in the edge area of ​​the service area; the monitoring data includes laser radar point cloud data and multi-view two-dimensional image data; The monitoring data corresponding to the multiple monitoring points are subjected to feature fusion according to the regional positions to obtain the fusion features of the union area corresponding to the multiple monitoring areas at the current moment; the multiple monitoring areas correspond to the multiple monitoring points one by one; Determine the intersection area of ​​the union area corresponding to the multiple monitoring areas and the service area, and fuse the fusion features of the intersection area at the current moment with the maintenance features of the intersection area at the previous moment to obtain the maintenance features of the service area at the current moment; the server is responsible for maintaining the maintenance features of the service area at the current moment; the maintenance features are three-dimensional voxel features; Using the maintenance features at the current moment in the service area to provide feature feedback to the target vehicle, so that the target vehicle performs 3D target detection using the feedback features; The monitoring data corresponding to the multiple monitoring points are subjected to feature fusion according to the regional positions to obtain the fusion features of the union area corresponding to the multiple monitoring areas at the current moment, including: Take two monitoring points from multiple monitoring points without replacement, and perform feature fusion on the monitoring data of the two currently taken monitoring points according to the regional positions to obtain the initial fusion features of the union area corresponding to the two monitoring points; Determine whether there are any remaining monitoring points that have not been feature fused. If so, take out a monitoring point from the remaining monitoring points without replacement, and perform feature fusion on the monitoring data of the monitoring point and the initial fusion feature obtained after the monitoring point was taken out last time according to the regional position to obtain the initial fusion feature of the updated union area, and continue to perform the determination of whether there are any remaining monitoring points that have not been feature fused. If not, use the initial fusion feature of the updated union area as the fusion feature of the union area corresponding to the multiple monitoring areas. The feature fusion according to the region position includes: fusing the overlapping regions of the two regions currently to be feature fused according to the self-attention mechanism to obtain the fusion features of the overlapping regions; and directly using the three-dimensional voxel features of the non-overlapping regions as the fusion features of the non-overlapping regions; The method of fusing the fusion features of the intersection area at the current moment with the maintenance features of the intersection area at the previous moment to obtain the maintenance features of the service area at the current moment includes: fusing the fusion features of the intersection area at the current moment with the maintenance features of the intersection area at the previous moment according to the self-attention mechanism to obtain the maintenance features of the intersection area at the current moment; and using the maintenance features of the area within the service area except the intersection area at the previous moment as the maintenance features of the current moment.

2. The method according to claim 1, characterized in that The feature fusion according to the self-attention mechanism includes: Perform linear transformation on the two regions where feature fusion is required; The corresponding attention weights are calculated using the linear transformation results corresponding to the two regions; The attention weights and the linear transformation results are used to calculate the fusion features.

3. A vehicle-road network 3D target detection method integrating roadside information, characterized in that: Applied to the vehicle side, the method includes: Acquire monitoring data at the current moment in the monitoring area, and send the monitoring data at the current moment to the server, so that the server uses the vehicle-road network 3D target detection method integrating roadside information as described in any one of claims 1-2 to perform feature feedback; the monitoring area and the service area of ​​the server have overlapping areas; the monitoring data include laser radar point cloud data and multi-view two-dimensional image data; Receive the feedback features sent by the server, and use the feedback features to perform 3D object detection.

4. A vehicle-road network 3D target detection device integrating roadside information, characterized in that: Applied to a server, the server is used to maintain the three-dimensional voxel features in the service area, and the device includes: An acquisition unit, used to acquire monitoring data of a corresponding monitoring area sent by multiple monitoring points at the current moment; the monitoring points include vehicles and roadside facilities, and the monitoring points are located in the service area or in the edge area of ​​the service area; the monitoring data includes laser radar point cloud data and multi-view two-dimensional image data; A first fusion unit is used to perform feature fusion on the monitoring data corresponding to the multiple monitoring points according to the regional positions, so as to obtain the fusion features of the union area corresponding to the multiple monitoring areas at the current moment; the multiple monitoring areas correspond to the multiple monitoring points one by one; The second fusion unit is used to determine the intersection area of ​​the union area corresponding to the multiple monitoring areas and the service area, and fuse the fusion features of the intersection area at the current moment with the maintenance features of the intersection area at the previous moment to obtain the maintenance features of the service area at the current moment; the server is responsible for maintaining the maintenance features of the service area at the current moment; the maintenance features are three-dimensional voxel features; A feedback unit, used to use the maintenance features at the current moment in the service area to provide feature feedback to the target vehicle, so that the target vehicle uses the feedback features to perform 3D target detection; The first fusion unit is specifically used for: Take two monitoring points from multiple monitoring points without replacement, and perform feature fusion on the monitoring data of the two currently taken monitoring points according to the regional positions to obtain the initial fusion features of the union area corresponding to the two monitoring points; Determine whether there are any remaining monitoring points that have not been feature fused. If so, take out a monitoring point from the remaining monitoring points without replacement, and perform feature fusion on the monitoring data of the monitoring point and the initial fusion feature obtained after the monitoring point was taken out last time according to the regional position to obtain the initial fusion feature of the updated union area, and continue to perform the determination of whether there are any remaining monitoring points that have not been feature fused. If not, use the initial fusion feature of the updated union area as the fusion feature of the union area corresponding to the multiple monitoring areas. When the first fusion unit performs the feature fusion according to the region position, the first fusion unit specifically includes: performing feature fusion on the overlapping region of the two regions currently to be feature fused according to the self-attention mechanism to obtain the fusion feature of the overlapping region; the three-dimensional voxel feature of the non-overlapping region is directly used as the fusion feature of the non-overlapping region; The second fusion unit is specifically used to: perform feature fusion of the fusion features of the intersection area at the current moment and the maintenance features of the intersection area at the previous moment according to the self-attention mechanism to obtain the maintenance features of the intersection area at the current moment; and use the maintenance features of the area within the service area except the intersection area at the previous moment as the maintenance features of the current moment.

5. The device according to claim 4, characterized in that The feature fusion according to the self-attention mechanism specifically includes: performing linear transformation on two areas where feature fusion is required; calculating corresponding attention weights using the linear transformation results corresponding to the two areas; and calculating the fusion feature using the attention weights and the linear transformation results.

6. A vehicle-road network 3D target detection device integrating roadside information, characterized in that: Applied to a vehicle, the device comprises: A monitoring unit, used for obtaining monitoring data at the current moment in a monitoring area, and sending the monitoring data at the current moment to a server, so that the server uses the vehicle-road network 3D target detection method integrating roadside information as described in any one of claims 1-2 to perform feature feedback; the monitoring area and the service area of ​​the server have an overlapping area; the monitoring data includes laser radar point cloud data and multi-view two-dimensional image data; The detection unit is used to receive the feedback features sent by the server and use the feedback features to perform 3D target detection.

7. A computer device, characterized in that: The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the steps of any of the methods described in claims 1-3.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 3 are implemented.

Images