Anti-noise multi-mode time sequence fusion three-dimensional target detection method for vehicle-road cooperation

Abstract

The invention discloses an anti-noise multi-mode time sequence fusion three-dimensional target detection method for vehicle-road cooperation. The method comprises the following steps: S1, inputting an image with a timestamp, and extracting two-dimensional multi-scale image features; camera image aerial view features are obtained through camera parameter coding; s2, inputting a laser radar point cloud with reflection intensity and a timestamp, and obtaining laser radar point cloud aerial view features; s3, aligning the camera image aerial view features and the laser radar point cloud aerial view features according to timestamps, and feeding the camera image aerial view features and the laser radar point cloud aerial view features to an adaptive alignment fusion module; obtaining a bird's-eye view feature optical flow after multi-modal fusion; s4, a time sequence modeling circulation unit calculates an aerial view feature optical flow after multi-modal fusion along a timestamp dimension, and adjusts and optimizes an aerial view feature sequence; and S5, through a multi-layer perceptron, target detection classification and regression of center point coordinates and bounding box sizes are completed. According to the method, the performance is improved under the condition that a large amount of calculation overhead is not increased, so that a better recognition effect can be obtained.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer vision technology, specifically relating to a noise-resistant multimodal temporal fusion three-dimensional target detection method for vehicle-road cooperative systems. Background Technology

[0002] Vehicle-to-everything (V2X) perception is a technological concept within swarm intelligence. It refers to the information interaction between intelligent connected vehicles and intelligent infrastructure and other intelligent devices in the surrounding environment to enhance the vehicle's perception of its surroundings. In this process, both intelligent connected vehicles and roadside infrastructure are equipped with various sensors, such as LiDAR and cameras. These sensors not only provide detailed environmental information for individual vehicles but also share acquired data through mutual communication, thereby compensating for blind spots in individual perception and expanding the overall perception range.

[0003] In traffic intersection scenarios, these devices have a wide field of view, easily capturing details of targets such as vehicles and pedestrians. However, sensors such as cameras, millimeter-wave radar, and lidar mounted on traffic poles are susceptible to environmental factors such as wind and rain, causing disturbances in the sensor's intrinsic and extrinsic parameter matrices, which in turn affect the depth estimation algorithms for 3D target detection. Furthermore, for targets far from the sensor, the depth difference between vehicles and the ground or grass is small, making localization based solely on depth estimation limited; algorithms struggle to utilize these subtle depth differences for classification and bounding box regression. In addition, existing algorithms often fail to meet basic requirements when dealing with overlapping and occlusion issues among multiple targets and complex, variable weather conditions. Single-camera modalities, unable to provide accurate depth, distance, and velocity information and susceptible to environmental factors such as lighting and humidity, have insufficient perception capabilities and face safety risks.

[0004] For example, under uneven lighting conditions or insufficient light intensity, the images captured by the camera may not be sharp; in extreme and harsh environments such as rain, snow, fog, and haze, camera images may also be affected. All of these situations can lead to missed detections and false detections in the model, and in severe cases, may even result in major traffic accidents. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention aims to provide a noise-resistant multimodal temporal fusion 3D target detection method oriented towards vehicle-road cooperation. This method designs a lightweight adaptive alignment fusion module for dynamically adjusting and fusing bird's-eye view features from cameras and LiDAR. This module combines a deformable attention mechanism and a scale-matching structure for spatial alignment. Furthermore, to reduce occlusion caused by the relative motion of vehicles on the road, a spatiotemporal sequence network is introduced to perform temporal modeling at the bird's-eye view feature level, predicting the target's motion position. This improves performance without significantly increasing computational overhead, thereby achieving better recognition results.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A noise-resistant multimodal temporal fusion 3D target detection method for vehicle-road cooperative systems includes the following steps; S1: Input a timestamped image, extract two-dimensional multi-scale image features through an image encoder; obtain camera image bird's-eye view features through camera parameter encoding; S2: Input a lidar point cloud with reflection intensity and timestamp, encode it using voxels and transform it to the world coordinate system to obtain the lidar point cloud bird's-eye view features. S3: The bird's-eye view features of the camera image and the bird's-eye view features of the LiDAR point cloud are aligned according to the timestamp and then fed to the adaptive alignment and fusion module to obtain the optical flow of the bird's-eye view features after multimodal fusion. S4: The temporal modeling loop unit calculates the optical flow of the multimodal fused bird's-eye view features along the timestamp dimension, aggregates historical frame information along the feature trajectory, and learns the rotation and scaling patterns of the target; a two-dimensional convolutional block is set to decode the candidate hidden states and add them to the input to obtain the final adjusted bird's-eye view feature sequence, and the bird's-eye view feature sequence is adjusted and optimized. S5: After passing through a multilayer perceptron, the object detection classification, center point coordinates, and bounding box size regression are completed.

[0007] Furthermore, in S1, the timestamped image is processed by an image encoder composed of a pre-trained two-dimensional convolutional backbone network and a feature pyramid to extract two-dimensional multi-scale image features; the height estimation network encoded by camera parameters further analyzes the two-dimensional multi-scale image features to obtain refined semantic features and height distribution features; the obtained semantic features and height distribution features are transformed into three-dimensional space by a height-based back projection algorithm to complete the transformation from perspective view to bird's-eye view and obtain the bird's-eye view features of the camera image.

[0008] Furthermore, in S1, the image with the timestamp is , This indicates that the data contains Frame image, Indicates channel dimension, and The height and width of the image are represented; the refined image semantic features are: The corresponding height distribution characteristics are The origin of the world coordinate system is set as the camera's projection point on the horizontal plane. The x-axis points directly in front of the camera's projection point, and the y-axis points to the right of the camera's projection point. The camera's intrinsic parameter matrix is ​​known to be... Projecting image semantic features onto the camera coordinate system: ; Define a virtual coordinate system with its origin coinciding with the camera coordinate system's origin, its y-axis perpendicular to the ground, and its z-axis parallel to the ground, oriented in the same direction as the world coordinate system's x-axis. The transformation matrix from the camera coordinate system to the virtual coordinate system is known. Transform points from the camera coordinate system to the virtual coordinate system: ; at this time, , This represents the height difference of the point relative to the origin of the virtual coordinate system. The vertical distance between the camera and the horizontal ground is known to be... Based on similar triangles, transform to a virtual coordinate system: ; Then, use the transformation matrix from the virtual coordinate system to the world coordinate system to transform it to the world coordinate system: ; The final result is a bird's-eye view of the camera image features.

[0009] Furthermore, in S2, voxel encoding is used to obtain a more simplified and efficient representation, and the position index of the voxel grid in the bird's-eye view plane is recorded. After downsampling, the number of point clouds in the voxel grid is limited. The volumetric encoder compresses the dimension of the point cloud voxels along the Z-axis of the world coordinate system to complete the volumetric representation of the point cloud. According to the pre-recorded position index, the volumetric representation of the point cloud is allocated to the bird's-eye view space to obtain the bird's-eye view features of the LiDAR point cloud in the form of a pseudo-image stack.

[0010] Furthermore, the lidar point cloud with reflection intensity described in S2 is... , This indicates that the data contains Frame point cloud, This indicates that the data contains The point cloud bird's-eye view, stacked in pseudo-image form, is characterized by: [number of points] points; .

[0011] Furthermore, in S3, the adaptive alignment fusion module consists of an alignment module based on deformable cross-attention and a scale matching module based on bilinear sampling; Alignment modules based on deformable cross-attention extract associated local contexts of multimodal features and interactively fuse them; A scale matching module based on bilinear sampling is used to adjust the scale space resolution of multimodal data.

[0012] Furthermore, in S3, the calculation of the alignment module based on deformable cross-attention is shown in the following equation: ; In the formula, This demonstrates how to transform the reference points obtained from linear layer initialization into discrete coordinates on the image feature coordinate system. This involves stitching together the camera image bird's-eye view features and the point cloud bird's-eye view features along the channel dimension. Feature values ​​at location; H and W These represent the height and width of the feature map, respectively. Representative at the reference point The query vector at the location; ; in It refers to the number of attention groups, and different groups have different focuses of attention. This is the index used for indexing group numbers. This is the number of sampling points. Setting it to 4 means that a reference point will focus on the features of 4 points in its neighborhood. The sampling point number is used to index the feature value of the reference point. Indicate query characteristics Reference point for prediction Position offset, This represents the input feature at the final sampling point location. It is the learned weight matrix. For query features The learned attention matrix represents the attention at the reference point. Place, With the The first group The correlation of the sampling points is such that the matrix has the following relationship: ; The calculation of the scale matching module based on bilinear sampling is shown in the following formula: ; in express Convolution kernel operation, and These represent the batch normalization operation and the ReLU activation layer, respectively. This indicates an upsampling operation based on bilinear interpolation, with an upsampling factor of 2. It is Convolutional layers are used to reduce channel dimensions. This indicates a channel connection operation.

[0013] The entire modal alignment module based on deformable cross-attention and the multimodal feature fusion module based on bilinear sampling scale matching are expressed by the following formula: .

[0014] Furthermore, in S4, the timing modeling is shown in the following equation: ; in, It is the input bird's-eye view feature sequence. It is a set of optical flow representing the neighborhood of target pixels. It is a topology learning function, with the current time step information as input. Hidden Units in Historical Timeline Cascade, This represents the size of the neighborhood set of pixels, used to set the number of learnable connections. (Function) Indicates the location specified. and Use bilinear sampling interpolation.

[0015] Furthermore, in s4, the two-dimensional convolutional block decodes the candidate hidden state and adds it to the input, as shown in the following formula: .

[0016] in This represents a two-dimensional convolution operation. This indicates a hidden unit representing a historical time step.

[0017] Furthermore, in S5, the loss for classification is as follows: ; In the formula, This represents the probability that the prediction is the true class. and For hyperparameters; The regression loss is as follows: ; in, This represents the deviation between the predicted attributes of the detection box and the actual values. The loss function is defined as follows: .

[0018] Another objective of this invention is to provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory, used to store computer programs; The processor, when executing a program stored in memory, implements the above method steps.

[0019] Another objective of this invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps.

[0020] A noise-resistant multimodal temporal fusion 3D target detection system for vehicle-road cooperative systems, comprising an image encoder, a point cloud encoder, an adaptive alignment and fusion module, and a temporal modeling loop unit; The image encoder converts the input multi-frame raw images into camera image bird's-eye view features; The point cloud encoder converts the input multi-frame raw point cloud into point cloud bird's-eye view features; The aerial view features of the camera image and the aerial view features of the point cloud are fused by the adaptive alignment and fusion module; The time-series modeling loop unit adjusts and optimizes the feature sequence of the bird's-eye view.

[0021] The beneficial effects of this invention are: (1) Based on the feature fusion of camera and point cloud multimodal features, this embodiment of the invention designs a modality alignment method based on a deformable cross-attention mechanism to adaptively align the heterogeneous modal spaces of the camera and LiDAR, extract the associated local context of multimodal features and interactively fuse them. A scale matching module with bilinear sampling is introduced to compensate for the performance loss caused by the inconsistency in the feature representation fineness of the two modalities, and finally realizes the dynamic fusion of camera images and LiDAR point cloud features.

[0022] (2) The present invention is based on the temporal modeling of the feature sequence of the bird's-eye view. Compared with the prior art, it is more accurate in the spatial analysis of key targets, effectively learns the motion patterns of the target such as rotation and scaling, and reduces the interference of irrelevant background on key targets. In terms of time, it dynamically fuses historical information according to the hidden candidate state of the loop unit. Unlike the existing frame splicing method, this method directly processes information along the timeline, so that the model can better predict the motion state, thereby effectively alleviating the problem of frequent target occlusion. Attached Figure Description

[0023] Figure 1This is the overall framework of the multimodal temporal fusion three-dimensional target detection system method for vehicle-road cooperation according to the embodiments of the present invention.

[0024] Figure 2 This is a schematic diagram of the height-based positioning algorithm used in an embodiment of the present invention.

[0025] Figure 3 This is the detailed structure and overall algorithm framework of the timing modeling module in this embodiment of the invention. Detailed Implementation

[0026] The present invention will now be described in further detail with reference to the accompanying drawings.

[0027] like Figure 1 As shown, the method for a multimodal temporal fusion 3D target detection system oriented towards vehicle-road cooperation includes the following steps: Step 1: Input an image with timestamps and a point cloud with reflection intensity. After appropriate preprocessing, the image is cropped and normalized, and the point cloud is padded with zeros and discarded. The intrinsic parameter matrix, extrinsic parameter matrix, and corresponding transformation matrix are stored in a dictionary. This aims to accelerate the subsequent training process of the neural network and remove the influence of irrelevant features and noise. Step 11: Input an image with a timestamp. The image is processed by an image encoder consisting of a pre-trained two-dimensional convolutional backbone network and a feature pyramid to extract two-dimensional multi-scale image features.

[0028] Image data with timestamps The input is fed into the ResNet-50 feature extraction backbone network. This indicates that the data contains Frame image, Indicates channel dimension, and This represents the height and width of the image; the image scale resolution is progressively reduced in the backbone network, expanding the channel dimension information to obtain multi-channel image features with a 16x downsampling. .

[0029] Step 12, as follows Figure 2 As shown, image features Feeded into the camera intrinsic parameter matrix and extrinsic parameter matrix The feature map is obtained by parsing the encoded neural network. Predicted height distribution value for each pixel and further refined semantic features .

[0030] Step 13: Utilize a height-based back-projection algorithm to analyze the parsed semantic features. and height distribution predictions Elevate to three-dimensional space. For example... Figure 2 As shown, assume the origin of the world coordinate system is the camera's projection point on the horizontal plane, the x-axis points directly forward, and the y-axis points to the right. The camera intrinsic parameter matrix is ​​known to be... Projecting image semantic features onto the camera coordinate system: ; Assume a virtual coordinate system with its origin coinciding with the camera coordinate system's origin, its y-axis perpendicular to the ground, and its z-axis parallel to the ground, oriented in the same direction as the world coordinate system's x-axis. The transformation matrix from the camera coordinate system to the virtual coordinate system is known. It can transform points in the camera coordinate system to the virtual coordinate system: ; at this time, , This represents the height difference of the point relative to the origin of the virtual coordinate system. The vertical distance between the camera and the horizontal ground is known to be... Based on similar triangles, we can transform to a virtual coordinate system: ; Then, use the transformation matrix from the virtual coordinate system to the world coordinate system to transform it to the world coordinate system: ; Step 2, for the input lidar point cloud with reflection intensity , This indicates that the data contains Frame point cloud, This indicates that the data contains Extract volumetric column coding features from the point cloud: Step 21, voxelization of point cloud; let the length, width, and height of the voxel space in the 3D space to be detected be respectively... , and The size of the voxel lattice is The entire space is discretized into Each grid represents a... XYZ The number of discrete intervals in the three directions of the coordinate axes; for any point in a frame of point cloud. The corresponding voxel center index is: ; in , and They are respectively XYZ Minimum values ​​of the coordinate axes.

[0031] Step 22, point cloud voxel encoding; for each point cloud within a voxel, it is represented by four-dimensional attributes, i.e. ,in Represents three-dimensional coordinates. The reflection intensity at that point is indicated; then, the offset of that point from the geometric center of the cylinder is calculated. And the positional offset of this point from the cluster center points of all point clouds within the column. Add this information to the original point cloud to obtain .

[0032] Step 23, volumetric feature extraction; using a multilayer perceptron to map the point cloud features described by 10-dimensional attributes to 64 dimensions, learning feature extraction and representation in different dimensions: ; in, This is represented as the first fully connected layer of a neural network. This maps each point cloud within the volume column to a high-dimensional feature vector.

[0033] Next, max pooling is used to extract and aggregate global features within the point cloud volume pillars: ; in, This represents the first-level max pooling operation. This represents the global feature vector obtained by aggregation, which contains the structural shape of the target boundary within the point cloud column, but loses the spatial information inside the column.

[0034] global feature vectors High-dimensional point clouds within the column By stitching the images together along the channel dimension, an expanded version of the original point cloud is obtained. N Each dimension : ; in, This represents the concatenated vector, with the shape being... It contains both the aggregated global boundary information and retains the independent features of each point in the high-dimensional space. This indicates a channel connection operation.

[0035] Then, a fully connected layer is used to fuse the global and local information of the point cloud, and to further expand the dimensions: ; in, This represents the second-layer perceptron. This is the extracted feature vector.

[0036] Finally, a max pooling operation is used to further enhance the geometric structure of the target within the column: ; in This indicates the second pooling operation. P This represents the final eigenvector of a single volume column.

[0037] After batch processing each column, the extracted feature vectors are remapped onto the bird's-eye view using a pre-calculated indexing method, and finally stacked to obtain a shape of... The pseudo-image form of tensor.

[0038] Step 3: After aligning the multimodal bird's-eye view features of the camera and LiDAR point cloud according to the timestamps, they are handed over to the adaptive alignment and fusion module for processing. Step 31: Based on the deformable cross-attention alignment module, extract the associated local context of multimodal features and interactively fuse them. The calculation is shown in the following formula: ; In the formula, This demonstrates how to transform the reference points obtained from linear layer initialization into discrete coordinates on the image feature coordinate system. This involves stitching together the camera image bird's-eye view features and the point cloud bird's-eye view features along the channel dimension. Feature values ​​at location; H and W These represent the height and width of the feature map, respectively. Representative at the reference point The query vector at the location; ; in It refers to the number of attention groups, and different groups have different focuses of attention. This is the index used for indexing group numbers. This refers to the number of sampling points. In this invention, it is set to 4, indicating that a reference point will focus on the features of 4 points in its neighborhood. It is used to number the sampling points and index the feature values ​​of the reference points. Indicate query characteristics Reference point for prediction Position offset, This represents the input feature at the final sampling point location. It is the learned weight matrix. For query features The learned attention matrix represents the attention at the reference point. Place, With the The first group The correlation of the sampling points is such that the matrix has the following relationship: ; Step 32: The performance loss caused by the inconsistency in fineness and scale of heterogeneous modes is compensated by a scale matching module based on bilinear sampling. The calculation is as follows: ; in express Convolution kernel operation, and These represent the batch normalization operation and the ReLU activation layer, respectively. This indicates an upsampling operation based on bilinear interpolation, with an upsampling factor of 2. It is The convolutional layers are used to reduce the channel dimension. This indicates a channel connection operation.

[0039] Step 33, the entire modal alignment based on deformable cross-attention and multimodal feature fusion module based on bilinear sampling scale matching is expressed by the following formula: ; Step 4, as follows Figure 3 As shown, based on the feature sequence of the bird's-eye view, the temporal modeling module is used for adjustment and optimization: Step 41: Calculate the optical flow along the timestamp dimension on the bird's-eye view features, aggregate historical frame information along the feature trajectory, and learn the target's rotation and scaling patterns. The calculation is as follows: ; in, It is the input bird's-eye view feature sequence. It is a set of optical flow representing the neighborhood of target pixels. It is a topology learning function, with the current time step information as input. Hidden Units in Historical Timeline Cascade. This represents the size of the neighborhood set of each pixel, used to set the number of learnable connections. (Function) Indicates the location specified. and Use bilinear sampling interpolation.

[0040] Step 42: Decode the candidate hidden states using a two-dimensional convolutional block and add it to the input to obtain the final adjusted bird's-eye view feature sequence. Its calculation is as follows: .

[0041] in This represents a two-dimensional convolution operation. This indicates a hidden unit representing a historical time step.

[0042] Step 5, Object Classification and Bounding Box Regression: This involves combining the bird's-eye view feature sequence after multimodal feature fusion and temporal modeling optimization. The data is fed into a classifier and regressor composed of multilayer perceptrons to classify the target and predict the attributes of the bounding box.

[0043] The loss functions for classification and regression networks are as follows: ; In the formula, This represents the probability that the prediction is the true class. and The hyperparameter is denoted as ; the regression loss is as follows: ; in, This represents the deviation between the predicted attributes of the detection box and the actual values. The loss function is defined as follows: ; To address the vulnerability of single sensors in harsh environments, this invention designs a multimodal fusion framework. By employing volumetric columnar encoding of LiDAR point clouds and a deformable cross-attention mechanism, adaptive alignment between camera and point cloud features is achieved. Simultaneously, a scale matching module based on bilinear sampling is introduced to compensate for multimodal differences, extracting the associated local context of multimodal features and interactively fusing them.

[0044] In the temporal modeling stage, this invention applies a spatiotemporal sequence prediction subnetwork to expand the receptive field of the detector in the temporal space. By dynamically fusing historical information through the hidden candidate states of the recurrent unit, compared with the existing frame stitching method, this method directly processes information along the timeline and learns the positional offset and rotational motion of feature pixels within a specific time period, thereby effectively alleviating the problem of frequent target occlusion.

[0045] The aforementioned method for a multimodal temporal fusion 3D target detection system oriented towards vehicle-road cooperation can be implemented as a computer program, stored on a hard disk, and loaded into a processor for execution to implement the method of the present invention.

[0046] This invention also provides a computer-readable medium storing computer program code, which, when executed by a processor, implements the graph convolutional neural network action recognition method based on dynamic time warp as described above.

[0047] When the multimodal temporal fusion 3D target detection system method for vehicle-road cooperative systems is implemented as a computer program, it can also be stored as an article of manufacture in a computer-readable storage medium. For example, computer-readable storage media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical discs (e.g., compact discs (CDs), digital multifunction discs (DVDs)), smart cards, and flash memory devices (e.g., electrically erasable programmable read-only memory (EPROM), cards, sticks, key drives). Furthermore, the various storage media described in the embodiments of this invention can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media (and / or storage media) capable of storing, containing, and / or carrying code and / or instructions and / or data.

[0048] It should be understood that the above embodiments are merely illustrative. The embodiments described in this invention can be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For hardware implementation, the processing unit can be implemented within one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and / or other electronic units designed to perform the functions described in this invention, or combinations thereof.

[0049] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0050] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A noise-resistant multi-modal time-series fusion three-dimensional target detection method for vehicle-road cooperation, characterized in that, Comprise the following steps: S1: input image with timestamp, through image encoder, extract two-dimensional multi-scale image features; Through the camera parameter coding to obtain the camera image bird's eye view feature; S2: input laser radar point cloud with reflection intensity and timestamp, through voxel coding and world coordinate system conversion, obtain laser radar point cloud bird's eye view feature; S3: after the camera image bird's eye view feature and the laser radar point cloud bird's eye view feature are aligned according to the timestamp, they are respectively fed into the adaptive alignment fusion module; obtain the multi-modal fused bird's eye view feature flow; S4: the time sequence modeling loop unit calculates the multi-modal fused bird's eye view feature flow along the timestamp dimension, sets a two-dimensional convolution block to decode the candidate hidden state, and adds the input to obtain the finally adjusted bird's eye view feature sequence, adjusts and optimizes the bird's eye view feature sequence; S5: through the multilayer perception, the classification of target detection and the regression of center point coordinates and boundary box size are completed.

2. The anti-noise multi-modal time series fusion three-dimensional target detection method for car-road cooperation according to claim 1, characterized in that, In S1, the image with timestamp is input into the image encoder composed of pre-trained two-dimensional convolutional backbone network and feature pyramid, and two-dimensional multi-scale image features are extracted; the height estimation network coded by camera parameters further analyzes the two-dimensional multi-scale image features to obtain refined semantic features and height distribution features; the analyzed semantic features and height distribution features are converted to three-dimensional space through height-based back projection algorithm, the conversion from perspective view to bird's eye view is completed, and camera image bird's eye view features are obtained.

3. The anti-noise multi-modal time series fusion three-dimensional target detection method for car-road cooperation according to claim 2, characterized in that, In S1, the image with timestamp is , indicates that there are frame images in the data, indicates the channel dimension, and indicate the height and width of the picture; the refined image semantic feature is , and the corresponding height distribution feature is ; the origin of the world coordinate system is set as the projection point of the camera in the horizontal plane, the x-axis points to the front of the projection point of the camera, the y-axis points to the right of the projection point of the camera, the camera intrinsic matrix is known as , and the image semantic feature is projected into the camera coordinate system: ； A virtual coordinate system is set with an origin coinciding with the camera coordinate system origin, the y-axis being perpendicular to the ground, and the z-axis being parallel to the ground, facing the same direction as the x-axis of the world coordinate system. A conversion matrix from the camera coordinate system to the virtual coordinate system is known Convert the point in the camera coordinate system to the virtual coordinate system: ； At this time, , represents the height difference of this point relative to the origin of the virtual coordinate system, and the vertical distance between the camera and the horizontal ground is known , according to the similar triangle, convert to the virtual coordinate system: ； Then, the virtual coordinate system is converted to the world coordinate system using the conversion matrix, and the conversion to the world coordinate system is completed: ； Finally, the camera image bird's eye view features are obtained.

4. The anti-noise multi-modal time series fusion three-dimensional target detection method for car-road cooperation according to claim 3, characterized in that, In S2, the voxel coding is used to obtain further simplified and efficient representation, and the position index of the voxel grid in the bird's eye view plane is recorded; after the downsampling operation, the number of point clouds in the voxel grid is limited; the body column encoder compresses the dimension of the point cloud voxel along the Z axis of the world coordinate system to complete the point cloud body column representation; according to the pre-recorded position index, the point cloud body column representation is distributed to the bird's eye view space to obtain the laser radar point cloud bird's eye view feature in the form of pseudo picture stacking; The laser radar point cloud with the reflected intensity in S2 is , indicates that there are frame point clouds in the data indicates that there are points in the data; the point cloud bird's eye view feature in the form of the pseudo picture stack is .

5. The anti-noise multi-modal time series fusion three-dimensional target detection method for car-road cooperation according to claim 4, characterized in that, In S3, the adaptive alignment fusion module is composed of an alignment module based on deformable cross attention and a scale matching module based on bilinear sampling; The alignment module based on deformable cross attention extracts the associated local context of multi-modal features and interacts and fuses them; The scale matching module based on bilinear sampling is used to adjust the scale space resolution of multi-modal features; The calculation of the alignment module based on deformable cross attention is shown by the following formula: ； wherein, the reference point obtained by the linear layer initialization is converted to the discrete coordinates on the image feature coordinate system; is the feature value at the position of the reference point . H and W represent the height and width of the feature map, respectively; represents the query vector at the reference point . ； in It refers to the number of attention groups, and different groups have different focuses of attention. This is the index used for indexing group numbers. It is the number of sampling points. The sampling point number is used to index the feature value of the reference point. Indicate query characteristics Reference point for prediction Position offset, This represents the input feature at the final sampling point location. It is the learned weight matrix. For query features The learned attention matrix represents the attention at the reference point. Place, With the The first group The correlation of the sampling points is such that the matrix has the following relationship: ； The calculation of the scale matching module based on bilinear sampling is shown by the following formula: ； wherein denotes denotes and denotes a batch normalization operation and a ReLU activation layer, respectively. denotes an up-sampling operation based on bilinear interpolation with an up-sampling factor of 2. is a denotes a convolutional layer for reducing the channel dimension, denotes a channel concatenation operation; The whole multi-modal feature fusion module based on deformable cross attention modal alignment and bilinear sampling scale matching is expressed by the following formula: 。 6. The anti-noise multi-modal time series fusion three-dimensional target detection method for car-road cooperation of claim 5, wherein, In S4, the time sequence modeling is shown by the following formula: ； in, It is the input bird's-eye view feature sequence. It is a set of optical flow representing the neighborhood of target pixels. It is a topology learning function, with the current time step information as input. Hidden Units in Historical Timeline Cascade, This represents the size of the neighborhood set of pixels, used to set the number of learnable connections. (Function) Indicates the location specified. and Use bilinear sampling interpolation; After the two-dimensional convolution block decodes the candidate hidden state and adds the input, it is shown by the following formula: 。 wherein denotes a two-dimensional convolution operation, denotes a history time step hidden unit.

7. The anti-noise multi-modal time series fusion three-dimensional target detection method for car-road cooperation according to claim 6, characterized in that, In S5, the classification loss is as follows: ； In the formula, denotes the probability of being predicted as a real class; and is a hyperparameter; The regression loss is as follows: ； wherein, represents the deviation between the predicted attributes of the detection frame and the true values, The definition of the loss function is as follows: 。 8. An electronic device, comprising: It comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; a memory for storing a computer program; a processor for executing the program stored on the memory to implement the method steps of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, A computer program is stored in a computer readable storage medium, and the computer program is executed by a processor to implement the method steps.

10. A noise-resistant multi-modal temporal fusion three-dimensional target detection system for cooperative vehicle infrastructure systems, characterized in that, The system comprises an image encoder, a point cloud encoder, an adaptive alignment fusion module and a temporal modeling loop unit. The image encoder converts the inputted multi-frame original images into camera image bird's eye view features. The point cloud encoder converts the inputted multi-frame original point clouds into point cloud bird's eye view features. The camera image bird's eye view features and the point cloud bird's eye view features are fused by the adaptive alignment fusion module. The temporal modeling loop unit adjusts and optimizes the bird's eye view feature sequence.

Images