3D Dynamic Multi-Object Detection Method, System and Storage Medium Based on Cross-View Feature Fusion

Abstract

The present invention provides a 3D dynamic multi-target detection method, system, and storage medium based on cross-view feature fusion. The method includes feature extraction, feature mapping and fusion, and pre-selection box generation and detection box optimization. The present invention combines the advantages of both lidar and camera sensors to implement dynamic multi-target detection technology for autonomous driving. This technology can accurately identify and locate multiple targets, including vehicles, pedestrians, and cyclists, while also ensuring real-time performance and practical application.

Description

Technical Field

[0001] The present invention relates to the field of image processing and autonomous driving technology, and in particular to a 3D dynamic multi-target detection method, system and storage medium based on cross-view feature fusion. Background Art

[0002] Autonomous driving technology has been widely used in various daily scenarios, including automatic road obstacle avoidance, SLAM navigation, and automated parking. Object detection is a crucial component of autonomous vehicles' perception of their surroundings. Current 3D object detection suffers from data degradation and insufficient feature richness, making improving detection performance a key research topic. Multimodal feature fusion combines the strengths and weaknesses of different sensor information, improving feature richness and compensating for data degradation caused by convolution. Object detection algorithms that combine point clouds and images for cross-view feature fusion offer advantages and are becoming a research hotspot.

[0003] While numerous studies have focused on multimodal object detection combining point clouds and images, most employ pseudo-LiDAR mapping, while research on direct feature extraction from point clouds and images remains limited. Solutions for cross-view multimodal object detection can be categorized into two types. One employs sequential fusion, where feature information from the previous stage is concatenated with features from the subsequent stage. For example, F-PointNets employs a frustum method, projecting pre-selected bounding boxes generated from the image onto the point cloud. The resulting frustum region is then used to obtain the final 3D detection bounding box. However, each frustum region only includes a single pre-selected object, making it ineffective in crowded scenes with numerous objects. Another approach employs parallel fusion, directly fusing multimodal features. This unified representation of feature form is fed into the detection module to generate pre-selected bounding boxes. For example, the ContFuse algorithm uses point-by-point feature fusion to address the issue of missing target structural features. However, due to the sparsity of point clouds, point-by-point fusion is computationally expensive in sparse areas and cannot meet the real-time requirements of autonomous driving. Summary of the Invention

[0004] The present invention provides a 3D dynamic multi-target detection method based on cross-view feature fusion, comprising the following steps:

[0005] Step 1, feature extraction step: including point cloud feature extraction network and image feature extraction network, through the parallel cloud feature extraction network and image feature extraction network to obtain complete multimodal features;

[0006] Step 2: Feature mapping and fusion step: Feature mapping adopts point-by-point correspondence to obtain point cloud features and point-by-point corresponding image features; the feature fusion process first calculates the weight of each point in the point cloud feature and image feature map, and sets the point cloud weighted feature as The LiDAR weight map is the weighted value of the point cloud, F L Represents the point cloud feature map, and the image feature map is F C , Represents channel-by-channel cascade, × represents point-by-point product operation, and the image weighted feature is Camera weight map is the image weight value, and then the weighted point cloud features and weighted image features are cascaded to obtain the fusion feature F Ripe ;

[0007] Step 3: Pre-selection box generation and detection box optimization includes the following steps:

[0008] Step 1: Fusion feature F Ripe Perform confidence testing to obtain positioning confidence and classification confidence maps;

[0009] Step 2: For each point in the feature map, generate two angled pre-selected boxes, which contain seven parameters (x, y, z, h, w, l, r), representing the coordinate axis and the length, width, height, and deflection angle of the pre-selected box respectively;

[0010] Step 3: Optimize the detection frame to improve the richness of features and enhance the ability to perceive the contextual structural information of the point cloud and image space.

[0011] As a further improvement of the present invention, the point cloud feature extraction network is composed of a voxel space partition, a voxel feature encoding module and a sub-flow type sparse convolution module. Assume that the range of the point cloud space along the Z, X, and Y coordinate axes is H, D, W, and the sampling step is v. H , v D , v W , the voxel set is The number of divided voxels is represented by h, d, and w respectively; let the set C = (C x , C y , C z ) is the local centroid of each voxel, that is, the coordinate mean of each voxel sampling point, then the encoded voxel feature set V in It can be expressed as T represents the matrix transpose, is a set of real numbers, C x 、C y 、C z is the coordinate of the voxel's center of mass, c i is the spatial coordinate of the i-th point in the voxel (x i,y i , z i ) and the reflectivity of the laser radar at that point, t∈(0, T], represents the number of sampling points in the voxel, V off Indicates the offset of each point relative to the local centroid C; the substream sparse convolution algorithm adopts the method of filling null values with zero and clearing the distorted area to ensure the real-time performance of convolution and avoid distortion during the convolution process. According to the convolution scale calculation formula E i Refers to the point in the input feature map, F i Refers to the point where the output feature map corresponds to the input feature map, and derives the scale information after eight-fold downsampling sparse convolution, where f represents the convolution kernel size, s represents the convolution step, and p is zero-value padding.

[0012] As a further improvement of the present invention, the image feature extraction network consists of ResNet18 and a feature pyramid, wherein ResNet18 consists of an input module and four convolution modules. The input module performs preprocessing operations on the image by combining ordinary convolution kernels with maximum pooling. The convolution module adds a residual network to solve the problem of feature degradation in the neural network; the feature pyramid uses a bottom-up upsampling method to amplify the lowest-level feature map obtained by ResNet18 to the top-level feature map, and retains the image features of each layer by superposition.

[0013] As a further improvement of the present invention, in the step 2, when performing feature mapping, in order to accurately find the correspondence between the point cloud and the image, the voxels of the original size are traversed, and the center point of the voxel is used as the reference coordinate. The corresponding pixel point in the image of the point in the point cloud is found through the correspondence between the internal and external parameters of the camera and the laser radar in the actual situation. If the corresponding pixel cannot be found, the point is skipped; considering that there is an eight-fold scale difference between the original size information and the feature map obtained by convolution, in order to prevent the pixel coordinates in the original image from being not an integer after being reduced by eight times, the present invention adopts a bilinear interpolation method to solve the problem; let I(m k* , n k* ) is the coordinate in the original image, F(m k* , n k* ) is the coordinate in the feature map, then the formula can be obtained in Means round down, i and j are index ranges, m k* and n k* is the horizontal and vertical coordinates corresponding to the kth point in the image coordinate system, b(i, j, n, n) = max(1-|im|, 0) × max(1-|jn|, 0), m represents the horizontal coordinate, which represents the distance from the four coordinate points to the target point (m after normalization. k* , n k* ), Represents the feature vectors of the four coordinate points in the image feature map.

[0014] As a further improvement of the present invention, in step 1, the fusion feature F Ripe Generate positioning confidence and classification confidence maps through 2D convolutional network.

[0015] As a further improvement of the present invention, step 3 includes:

[0016] Step A: Uniformly sample 6×6×6 key points along the coordinate axis of the pre-selected box, find the corresponding pixel points in the image feature map, encode them using PointNet, and obtain the image feature F corresponding to the pre-selected box. Camera ;

[0017] Step B: Then extract the sparse convolution features of each stage, including the features of one, two, four, and eight times downsampling. The sparse convolution fusion features of each stage are expressed as F Raw =∑ m∈M M(P(max(S m ))), where S m Represents the set of voxel feature vectors of the m-layer, P(·) represents the PointNet network, and M(·) represents the multi-layer perceptron network;

[0018] Step C: cascade F Ripe 、F Camera and F Raw Fusion, expressed as F = [F Ripe , F Raw , F Camera ]; for each point in the feature map, six angles of 0°, 30°, 60°, 90°, 120°, and 150° are selected to generate a detection frame, which is then input into the frame optimization network to perform residual calculations on the center, size, and direction of the detection frame; the frame optimization network consists of an MLP network with two branches, which respectively perform classification confidence and location confidence prediction. The classification confidence G of the kth ROI region is k =min(1,max(0,2IoU k -0.5)), where IoU k Represents the interaction ratio between ROI and ground truth.

[0019] As a further improvement of the present invention, in step B, since the scales of the feature maps of each layer are different, an MLP network is used for normalization processing.

[0020] As a further improvement of the present invention, in step 3, to address the inconsistency between the location confidence and the classification confidence, the two confidences are constrained by a mandatory consistency loss function, which is expressed as Where R represents the pre-selected box, T represents the ground truth value, and L(·) represents the position confidence. The position and category information of the final detection box are predicted based on the confidence map.

[0021] The present invention also provides a 3D dynamic multi-target detection system based on cross-view feature fusion, comprising: a memory, a processor, and a computer program stored on the memory, wherein the computer program is configured to implement the steps of the 3D dynamic multi-target detection method of the present invention when called by the processor.

[0022] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program is configured to implement the steps of the 3D dynamic multi-target detection method of the present invention when called by a processor.

[0023] The beneficial effects of the present invention are: the present invention combines the advantages of both lidar and camera sensors to realize dynamic multi-target detection technology in the field of autonomous driving, and can accurately identify and locate multiple types of targets such as vehicles, pedestrians, and cyclists. This technology takes into account real-time performance and can be applied in actual scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a principle block diagram of the present invention. DETAILED DESCRIPTION

[0025] This paper discloses a 3D dynamic multi-target detection method based on cross-view feature fusion. Unlike existing approaches, this method fully considers the information preservation issues of feature extraction methods and feature fusion. By designing a parallel point cloud and image feature extraction network to obtain complete multimodal features, and designing a cross-view feature mapping and fusion module, it achieves effective fusion of point cloud and image features to obtain accurate detection frames.

[0026] The 3D dynamic multi-target detection method of the present invention comprises the following steps:

[0027] Step 1, feature extraction step: including point cloud feature extraction network and image feature extraction network, where the point cloud feature extraction network is composed of voxel space division, voxel feature encoding module and sub-flow type sparse convolution module. Assume that the range of the point cloud space along the Z, X, Y coordinate axis is H, D, W, and the sampling step is v H , v D , v W , the voxel set is Let the set C = (C x , C y , C z ) is the local centroid of each voxel, that is, the coordinate mean of each voxel sampling point, then the encoded voxel feature set Vin It can be expressed as where r i is the reflectivity of the laser radar. Finally, the present invention designs a sub-flow sparse convolution algorithm to avoid the traditional convolution algorithm from sampling every point in the point cloud. It adopts the method of filling null values with zero and clearing the distorted area to ensure the real-time performance of the convolution and avoid distortion during the convolution process. According to the convolution scale calculation formula The scale information after eight-fold downsampling sparse convolution can be derived, where f represents the convolution kernel size. The present invention uses a convolution kernel with a dimension of 3, s represents the convolution step, and p is zero-value padding. The image feature extraction network consists of ResNet18 and a feature pyramid, where ResNet18 consists of an input module and four convolution modules. The input module performs preprocessing operations on the image, using a combination of ordinary convolution kernel and maximum pooling. The convolution module adds a residual network to solve the problem of feature degradation in the neural network; the feature pyramid uses a bottom-up upsampling method to amplify the lowest-level feature map obtained by ResNet18 to the top-level feature map, and retains the image features of each layer by superposition.

[0028] Step 2, feature mapping and fusion step: When using a camera to collect image information, there are usually multiple cameras, and the information representation forms of the image and point cloud are also different. In order to accurately find the correspondence between the point cloud and the image, traverse the voxels of the original size, use the center point of the voxel as the reference coordinate, and find the corresponding pixel point in the image of the point in the point cloud through the correspondence between the internal and external parameters of the camera and the lidar in actual conditions. Since the sensing ranges of the two are inconsistent, if the corresponding pixel cannot be found, the point is skipped. Taking into account the eight-fold scale gap between the original size information and the feature map obtained by convolution, in order to prevent the pixel coordinates in the original image from being not integers after being reduced by eight times, the present invention adopts bilinear interpolation to solve the problem. Let I(m k* , n k* ) is the coordinate in the original image, F(m k* , n k* ) is the coordinate in the feature map, then the formula can be obtained in Meaning round down, b(i, j, m, n) = max(1-|im|, 0) × max(1-|jn|, 0), which means the distance from the four coordinate points to the target point (m k* , n k* ), Represents the feature vectors of the four coordinate points in the image feature map. At this time, the point cloud features and the corresponding image features are obtained. The feature fusion process first calculates the weight of each point in the point cloud features and image feature maps, and sets the point cloud weighted feature as Among them F L Represents the point cloud feature map, Represents channel-by-channel cascade, and × represents point-by-point multiplication operation. The same is true for image weighted features. Then, the weighted point cloud features and weighted image features are cascaded to obtain the fusion feature F Ripe .

[0029] Step 3, pre-selection box generation and detection box optimization step: The obtained fusion features are tested for confidence to obtain the positioning confidence and classification confidence maps. Each point in the feature map generates a pre-selection box at two angles, which contains 7 parameters (x, y, z, h, w, l, r), which represent the coordinate axis coordinates and the length, width, height and deflection angle information of the pre-selection box respectively. The detection box optimization stage further improves the richness of the features and enhances the perception of the contextual structure information of the point cloud and image space. First, 6×6×6 key points are uniformly sampled along the coordinate axis of the pre-selection box, and the corresponding pixel points are found in the image feature map. PointNet is used for encoding to obtain the image feature F corresponding to the pre-selection box. Camera Then the sparse convolution features of each stage are extracted, including the features of one, two, four, and eight times downsampling. The sparse convolution fusion features of each stage can be expressed as F Raw =∑ m∈M M(P(max(S m ))), where S m Represents the set of voxel feature vectors of layer m, N m represents the number of non-empty voxels in the m-level feature map, P(·) represents the PointNet network, and M(·) represents the multi-layer perceptron network. Due to the different scales of the feature maps of each layer, the MLP network is used for normalization. Finally, the F Ripe , F Camera and F Raw Fusion, expressed as F = [F Ripe , F Raw , F Camera ], the feature map can effectively preserve the information of point cloud and image. For each point in the feature map, six angles of 0°, 30°, 60°, 90°, 120°, and 150° are selected to generate a detection frame, which is then input into the frame optimization network to perform residual calculations on the center, size, and direction of the detection frame. The frame optimization network consists of an MLP network with two branches, which respectively perform classification confidence and location confidence prediction. The classification confidence G of the kth ROI region is k =min(1,max(0,2IoU k -0.5)). Where IoU k Represents the interaction ratio between ROI and ground truth. To address the inconsistency between location confidence and classification confidence, this paper proposes a forced consistency loss function to constrain the two confidences, expressed as Where R represents the preselected box, T represents the ground truth value, and L(·) represents the location confidence. The location and category information of the final detection box are predicted based on the confidence map.

[0030] like Figure 1 As shown, the following is a detailed description with reference to specific examples:

[0031] Step 1, feature extraction step: In the point cloud feature extraction network, for the vehicle target, the effective range along the coordinate axis is H = [-3, 1] m, D = [-40, 40] m, W = [0, 70.4] m, and the size of each voxel is (v H , v D , v W )=(0.4, 0.2, 0.2), so the space is divided into 10×400×352 voxels. The voxel feature encoding module first converts the 7-dimensional vector V of each sampling point information into in The fully connected layer maps the image to 32 dimensions. This is then fed into the maximum pooling layer, obtaining 32-dimensional global information for each voxel. This information is then concatenated with the 32-dimensional features of each point, resulting in a 64-dimensional vector. This process is repeated until each point is represented by a 128-dimensional vector. Pooling further yields a 128-dimensional vector for each voxel, resulting in a point cloud representation of (128, 10, 400, 352). Finally, the image is fed into the submanifold sparse convolution module, where multiple convolution downsampling operations yield a feature map of dimensions (32, 50, 44). This means that each point in the feature map of scale (50, 44) is represented by a 32-dimensional vector. In the image feature extraction network, to ensure that the scale of the image features in the subsequent feature fusion stage corresponds to the point cloud features, the image information is set to (1, 3, 400, 352), where 3 represents the number of input channels (RGB channels) and (400, 352) represents the image resolution. After ResNet18 and feature pyramid network, the image feature map of (1, 256, 50, 44) is obtained. Through the maximum pooling operation, 32-dimensional features are obtained, so the final image feature map dimension is (32, 50, 44).

[0032] Step 2: Feature Mapping and Fusion: Feature mapping is performed point-by-point, yielding voxel feature maps and corresponding image feature maps, respectively, with dimensions (32, 50, 44). The feature fusion module then concatenates the voxel and image features. Considering the image is divided into left and right views, channel-by-channel concatenation is performed, resulting in a feature map of (96, 50, 44). For subsequent multi-feature fusion, the 96-dimensional features are mapped to 128 dimensions. The weighting module consists of two MLP convolutional networks. The first convolution uses 128 convolution kernels of size 1×128×1, resulting in n feature vectors of dimension 1×128, where n represents the size of the feature map. The second convolution uses 256 convolution kernels of size 1×1×128, resulting in n feature vectors of dimension 1×256, or n×256. The second MLP network operates similarly, ultimately yielding n feature vectors of dimension 1×1, or n×1, representing the weight information. Finally, the weighted point cloud and image feature maps are obtained by Sigmoid activation function and multiplication with the original input features. The weighted point cloud features and weighted image features after the adaptive feature fusion module are cascaded to obtain a dimension of F Ripe =(128, 50, 44) weighted feature map.

[0033] Step 3: Pre-selection box generation and detection box optimization step: F Ripe Through the 2D convolutional network, the classification confidence map and the positioning confidence map are generated. For each point in the feature map, two pre-selected boxes are generated. Each pre-selected box is represented by a 7-dimensional vector, so the dimension of the positioning confidence map is 14, that is, (14, 50, 44). The pre-selected box optimization stage first sets 6×6×6 key points along the coordinate axis and maps them to the image feature map. The image features are encoded using the PointNet algorithm to obtain a 216×128 feature vector. The input is the maximum pooling layer to obtain a 1×128-dimensional feature. This method is adopted for each region of interest, and the image feature F is finally obtained. Camera =(128, 50, 44). The point cloud dimension of the one-fold downsampling convolution is (128, 10, 400, 352), and the final dimension is (32, 50, 44). Therefore, each point after the eight-fold downsampling convolution represents the features of the surrounding 16 points in the original point cloud feature map, and after the dimension transformation, the size is also compressed along the Z axis, so the max(.) function is used to transform the scale of the feature map to (50, 44). The PointNet algorithm is used to encode the features, and the output dimension is still 128. Then, the features are input into the MLP(·) network to convert the 128-dimensional features into 32 dimensions. Considering that there are four convolution layers of one, two, four, and eight times, the features are finally fused to obtain a 128-dimensional multi-scale voxel feature F Raw=(128, 50, 44). Finally, a channel-by-channel cascade approach is adopted to obtain a 384-dimensional multimodal fusion feature F. Confidence prediction is performed again to obtain the location information and classification information of the target detection box.

[0034] The present invention is applicable to autonomous driving vehicles that combine lidar and cameras. A multimodal feature mapping and fusion method is adopted. Point cloud information and image information are input into the detection network to perform feature extraction, point-by-point feature correspondence, pre-selection box generation and optimization to achieve accurate target detection. In addition, the feature extraction methods of point clouds and images can be diverse. For example, point cloud feature extraction can adopt a point-based direct processing method or a voxel space division method, and image feature extraction can adopt a combination of ResNet and feature pyramid or VGG convolutional neural network. The detection box generation module needs to take time efficiency into consideration to meet the real-time requirements in actual autonomous driving scenarios.

[0035] In summary, the present invention is applicable to autonomous driving vehicles that combine lidar and cameras. The present invention can more effectively map and fuse point cloud and image features, reduce computing costs and time efficiency, and meet the real-time and accuracy requirements of target detection based on autonomous driving.

[0036] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A 3D dynamic multi-target detection method based on cross-view feature fusion, characterized in that: The steps include: Step 1, feature extraction step: including point cloud feature extraction network and image feature extraction network, through the parallel cloud feature extraction network and image feature extraction network to obtain complete multimodal features; Step 2: Feature mapping and fusion step: Feature mapping adopts a point-by-point correspondence method to obtain point cloud features and point-by-point corresponding image features; feature The fusion process first calculates the weight of each point in the point cloud feature and image feature map, and sets the point cloud weighted feature as The LiDAR weight map is the point cloud weight value, F L Represents the point cloud feature map, and the image feature map is F C , Represents channel-by-channel cascade, × represents point-by-point product operation, and the image weighted feature is Camera weight map is the image weight value, and then the weighted point cloud features and weighted image features are cascaded to obtain the fusion feature F Ripe ; Step 3: Pre-selection box generation and detection box optimization steps include the following steps: Step 1: Fusion feature F Ripe Perform confidence testing to obtain positioning confidence and classification confidence maps; Step 2: For each point in the feature map, generate a pre-selected box at two angles, which contains seven parameters (x, y, z, h, w, l, r), which represent the coordinate axis and the length, width, height and deflection angle of the pre-selected box respectively; Step 3: Optimize the detection frame to improve the richness of features and enhance the perception of contextual structural information in point clouds and image space; The point cloud feature extraction network consists of a voxel space partition, a voxel feature encoding module, and a sub-flow sparse convolution module. Assume that the range of the point cloud space along the Z, X, and Y coordinate axes is H, D, W, and the sampling step is v. H , v D , v W , the voxel set is The number of divided voxels is represented by h, d, and w respectively; let the set c = (c x , C y , C z ) is the local centroid of each voxel, that is, the coordinate mean of each voxel sampling point, then the encoded voxel feature set V in Expressed as T represents the matrix transpose, is a set of real numbers, c x 、c y 、c z is the coordinate of the voxel's center of mass, c i is the spatial coordinate of the i-th point in the voxel (x i ,y i , z i ) and the reflectivity of the laser radar at that point, t∈(0, T], represents the number of sampling points in the voxel, V off Indicates the offset of each point relative to the local centroid C; the substream sparse convolution algorithm adopts the method of filling null values with zero and clearing the distorted area to ensure the real-time performance of convolution and avoid distortion during the convolution process. According to the convolution scale calculation formula E i Refers to the point in the input feature map, F i Refers to the point where the output feature map corresponds to the input feature map, and derives the scale information after eight-fold downsampling sparse convolution, where f represents the convolution kernel size, s represents the convolution step size, and p is zero-value padding; In the step 2, when performing feature mapping, in order to accurately find the correspondence between the point cloud and the image, the voxels of the original size are traversed, and the center point of the voxel is used as the reference coordinate. The corresponding pixel point of the point in the point cloud in the image is found through the correspondence between the internal and external parameters of the camera and the laser radar in the actual situation. If the corresponding pixel cannot be found, the point is skipped; considering that there is an eight-fold scale difference between the original size information and the feature map obtained by convolution, in order to prevent the pixel coordinates in the original image from being not an integer after being reduced by eight times, the present invention adopts a bilinear interpolation method to solve the problem; let I(m k* , n k* ) is the coordinate in the original image, F(m k* , n k* ) is the coordinate in the feature map, then the formula can be obtained in Means round down, i and j are index ranges, m k* and n k* is the horizontal and vertical coordinates corresponding to the k-th point in the image coordinate system, b(i, j, m, n) = max(1-|im|, 0) × max(1-|jn|, 0), m represents the horizontal coordinate, which represents the distance from the four coordinate points to the target point (m after normalization. k* , n k* ), Represents the feature vectors of the four coordinate points in the image feature map; The step 3 includes: Step A: Uniformly sample 6×6×6 key points along the coordinate axis of the pre-selected box, find the corresponding pixel points in the image feature map, encode them using PointNet, and obtain the image feature F corresponding to the pre-selected box. Camera ; Step B: Then extract the sparse convolution features of each stage, including the features of one, two, four, and eight times downsampling. The sparse convolution fusion features of each stage are expressed as F Raw =∑ m∈M M(P(max(S m ))), where S m Represents the set of voxel feature vectors of the m-th layer, P(·) represents the PointNet network, and M(·) represents the multi-layer perceptron network; Step C: cascade the F Ripe 、F Camera and F Raw Fusion, expressed as F = [F Ripe , F Raw , F Camera ]; for each point in the feature map, six angles of 0°, 30°, 60°, 90°, 120°, and 150° are selected to generate a detection frame, which is then input into the frame optimization network to perform residual calculations on the center, size, and direction of the detection frame; the frame optimization network consists of an MLP network with two branches, which respectively perform classification confidence and location confidence prediction. The classification confidence G of the kth ROI region is k =min(1,max(0,2IoU k -0.5)), where IoU k Represents the interaction ratio between ROI and ground truth.

2. The 3D dynamic multi-target detection method according to claim 1, characterized in that: The image feature extraction network consists of ResNetl8 and a feature pyramid. ResNetl8 consists of an input module and four convolution modules. The input module preprocesses the image using a combination of ordinary convolution kernels and maximum pooling. The convolution module incorporates a residual network to address the feature degradation problem in the neural network. The feature pyramid uses a bottom-up upsampling method to enlarge the lowest feature map obtained by ResNetl8 to the top feature map, and retains the image features of each layer by superposition.

3. The 3D dynamic multi-target detection method according to claim 1, characterized in that: In step 1, the fusion feature F Ripe Generate positioning confidence and classification confidence maps through 2D convolutional network.

4. The 3D dynamic multi-target detection method according to claim 1, characterized in that: In step B, since the scales of the feature maps of each layer are different, an MLP network is used for normalization.

5. The 3D dynamic multi-target detection method according to claim 1, characterized in that: In step 3, to address the inconsistency between the location confidence and the classification confidence, the two confidences are constrained by a mandatory consistency loss function, which is expressed as Where R represents the pre-selected box, T represents the ground truth value, and L(·) represents the position confidence. The position and category information of the final detection box are predicted based on the confidence map.

6. A 3D dynamic multi-target detection system based on cross-view feature fusion, characterized in that: include: A memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the 3D dynamic multi-target detection method according to any one of claims 1 to 5 when called by the processor.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the 3D dynamic multi-target detection method according to any one of claims 1 to 5 when called by a processor.

Images