Target detection method and device based on radar and video fusion, equipment and medium

Abstract

The invention is suitable for the technical field of electric digital data processing, and provides a target detection method, device and equipment based on radar and video fusion and a medium, and the method comprises the steps: obtaining radar point cloud data and video data of a to-be-detected target; obtaining a target detection result based on a processing result of a radar and video fusion module in the improved YOLO model on the radar point cloud data and the video data; wherein the radar and video fusion module is used for preprocessing radar point cloud data and video data to obtain a radar image and a video image, and performing feature extraction, feature splicing, feature fusion, residual processing, secondary splicing and convolution processing on the radar image and the video image. According to the invention, radar and video data can be effectively fused, and the accuracy and robustness of target detection are improved.

Description

Technical Field

[0001] The present application belongs to the field of electronic digital data processing technology, and in particular, relates to a target detection method, device, equipment and medium based on radar and video fusion. Background Art

[0002] At present, the environmental perception devices used in smart transportation mainly include cameras and millimeter-wave radars. Millimeter-wave radars have the ability to detect target distance, speed, azimuth angle, etc. At the same time, they can quantitatively test the target reflection ability at a specific incident angle and give a target size estimate based on the millimeter-wave radar echo scattering characteristics.

[0003] Cameras often use multiple sensors for information fusion, and use multi-sensor information fusion technology to achieve the complementary advantages of each sensor, accurately collect the target's color, size, size, projection outline and other dimensional information to the greatest extent, and improve the comprehensiveness and accuracy of the environmental perception system.

[0004] In target detection technology, it is crucial to accurately detect targets, but traditional methods that rely on single video data do not perform well in bad weather or low lighting conditions. Millimeter-wave radar, as a supplementary sensor, can provide stable target information, but lacks intuitive images. Although there are already methods that combine radar and video for target detection, the target detection accuracy is not high, the accuracy is poor, and the robustness is also low. Summary of the invention

[0005] The embodiments of the present application provide a target detection method, apparatus, device and medium based on radar and video fusion to effectively fuse radar and video data and improve the accuracy and robustness of target detection.

[0006] This application is implemented through the following technical solutions: In a first aspect, an embodiment of the present application provides a target detection method based on radar and video fusion, comprising: Obtain radar point cloud data and video data of the target to be detected.

[0007] Based on the processing results of radar point cloud data and video data by the radar and video fusion module in the improved YOLO model, the target detection result is obtained; wherein, the radar and video fusion module is used to preprocess the radar point cloud data and video data to obtain radar images and video images, and perform feature extraction, feature splicing, feature fusion, residual processing, secondary splicing and convolution processing on the radar images and video images.

[0008] In combination with the first aspect, in some possible implementations, feature extraction, feature splicing, feature fusion, residual processing, secondary splicing and convolution processing are performed on the radar image and the video image, including: A 3x3 convolution operation is performed on the radar image to extract a first radar image feature; and a 3x3 convolution operation is performed on the video image to extract a first video image feature.

[0009] The first radar image feature and the first video image feature are concatenated through a concat operation to obtain a first concatenated feature.

[0010] The first concatenated features are extracted and fused through the transformer Encoder layer to obtain the first pre-output features.

[0011] The radar image and the video image are subjected to residual processing and 3x3 convolution operations to obtain the second pre-output features.

[0012] The first pre-output feature and the second pre-output feature are concat-channeled to obtain the third pre-output feature.

[0013] A 1x1 convolution operation is performed on the third pre-output feature to obtain the output result of the radar and video fusion module.

[0014] In combination with the first aspect, in some possible implementations, the improved YOLO model further includes a feature fusion module; Based on the processing results of radar point cloud data and video data by the radar and video fusion module in the improved YOLO model, the target detection results are obtained, including: The processing results of the radar point cloud data and the video data by the radar and video fusion module in the improved YOLO model are processed by at least one feature fusion module to obtain a target detection result; wherein the feature fusion module is used to perform residual processing, convolution, depth-separable convolution processing, concat fusion and add fusion operation processing on the input image.

[0015] In combination with the first aspect, in some possible implementations, residual processing, convolution, depth-separable convolution processing, concat fusion and add fusion operation processing are performed on the input image, including: Perform residual processing on the input image to obtain the first feature.

[0016] Perform a 1xk convolution operation on the input image to obtain the second feature.

[0017] Perform a kx1 convolution operation on the input image to obtain the third feature.

[0018] Ghost convolution and 3x3 depth-separable convolution are performed on the input image to obtain the fourth feature.

[0019] The second feature, the third feature and the fourth feature are concat-fused to obtain the first fused feature.

[0020] The first feature and the first fusion feature are added to obtain the fourth pre-output feature.

[0021] Ghost convolution and 1x1 convolution are performed on the fourth pre-output feature in sequence to obtain the output result of the feature fusion module.

[0022] In combination with the first aspect, in some possible implementations, residual processing, convolution, depth-separable convolution processing, concat fusion and add fusion operation processing are performed on the input image, including: Perform residual processing on the input image to obtain the first feature.

[0023] Perform a 1xk convolution operation on the input image to obtain the second feature Perform a kx1 convolution operation on the input image to obtain the third feature;.

[0024] Ghost convolution and 3x3 depth-separable convolution are performed on the input image to obtain the fourth feature.

[0025] The second feature, the third feature and the fourth feature are concat-fused to obtain the first fused feature.

[0026] Repeat the above steps multiple times to obtain multiple first fusion features.

[0027] An add operation is performed on the first feature and the multiple first fusion features to obtain a fourth pre-output feature.

[0028] Ghost convolution and 1x1 convolution are performed on the fourth pre-output feature in sequence to obtain the output result of the feature fusion module.

[0029] In combination with the first aspect, in some possible implementations, the improved YOLO model includes: a Backbone layer, a Neck layer and a Head layer; the feature fusion module includes: a first feature fusion module, a second feature fusion module, a third feature fusion module, a fourth feature fusion module, a fifth feature fusion module, a sixth feature fusion module, a seventh feature fusion module and an eighth feature fusion module.

[0030] The Backbone layer includes: a radar and video fusion module, a first CBS module, a first feature fusion module, a second CBS module, a second feature fusion module, a third feature fusion module, a fourth feature fusion module and an SPPF module connected in sequence.

[0031] The Neck layer includes: a PSA sampling attention module, an Upsample module, a first Concat module, a fifth feature fusion module, a second Concat module, a sixth feature fusion module, a third CBS module, a third Concat module, a seventh feature fusion module, a fourth CBS module, a fourth Concat module and an eighth feature fusion module connected in sequence.

[0032] The Head layer includes: a first Head module, a second Head module and a third Head module.

[0033] The second CBS module is also connected to the second Concat module; the third feature fusion module is also connected to the first Concat module; the SPPF module is also connected to the PSA sampling attention module; the PSA sampling attention module is also connected to the fourth Concat module; the fifth feature fusion module is also connected to the third Concat module; the sixth feature fusion module is also connected to the first Head module; the seventh feature fusion module is also connected to the second Head module; the eighth feature fusion module is also connected to the third Head module.

[0034] In combination with the first aspect, in some possible implementations, the radar and video fusion module is used to preprocess the radar point cloud data and the video data to obtain the radar image and the video image, including: The radar and video fusion module converts radar point cloud data into images to obtain radar images.

[0035] The video data is captured frame by frame using the OpenCV method to obtain the video image.

[0036] In a second aspect, an embodiment of the present application provides a target detection device based on radar and video fusion, comprising: The data acquisition module is used to obtain radar point cloud data and video data of the target to be detected.

[0037] The result output module is used to obtain the target detection result based on the processing result of the radar point cloud data and the video data by the radar and video fusion module in the improved YOLO model; wherein the radar and video fusion module is used to preprocess the radar point cloud data and the video data to obtain the radar image and the video image, and perform feature extraction, feature splicing, feature fusion, residual processing, secondary splicing and convolution processing on the radar image and the video image.

[0038] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a processor and a memory, the memory being used to store a computer program, and when the processor executes the computer program, the target detection method based on radar and video fusion as described in any one of the first aspects is implemented.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the target detection method based on radar and video fusion as described in any one of the first aspects is implemented.

[0040] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0041] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: This application processes radar point cloud data and video data through a radar and video fusion module, realizes the deep fusion of the two data, and realizes efficient and accurate target detection. Compared with the prior art, the present invention has significant performance advantages under conditions such as bad weather, low lighting or target occlusion because it integrates the rich texture information of video data and the precise ranging capability of radar data, reduces dependence on environmental conditions, and improves the accuracy of target detection; it also reduces the consumption of computing resources and maintains fast response capabilities through efficient data processing and advanced feature extraction technology, thereby improving real-time processing capabilities.

[0042] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 It is a flowchart of a target detection method based on radar and video fusion provided in one embodiment of the present application; Figure 2 is a schematic diagram of the structure of an improved YOLO model provided in an embodiment of the present application; Figure 3 is a schematic diagram of the structure of a radar and video fusion module provided in an embodiment of the present application; Figure 4 is a structural diagram of a Transformer Encoder module provided in an embodiment of the present application; Figure 5 is a schematic diagram of the structure of an MLP module provided in an embodiment of the present application; Figure 6It is a structural diagram of a feature fusion module provided in an embodiment of the present application; Figure 7 is a structural schematic diagram of a target detection device based on radar and video fusion provided in one embodiment of the present application; Figure 8 It is a structural diagram of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0045] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0046] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0047] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0048] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0049] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0050] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0051] The present application embodiment proposes a target detection method based on radar and video fusion. Figure 1 is a schematic diagram of the process of a target detection method based on radar and video fusion provided in an embodiment of the present application, with reference to Figure 1 , the method is described in detail as follows: Step 101, obtaining radar point cloud data and video data of a target to be detected.

[0052] Step 102, based on the processing result of the radar point cloud data and the video data by the radar and video fusion module in the improved YOLO model, the target detection result is obtained; wherein the radar and video fusion module is used to pre-process the radar point cloud data and the video data to obtain a radar image and a video image, and perform feature extraction, feature splicing, feature fusion, residual processing, secondary splicing and convolution processing on the radar image and the video image.

[0053] For example, Figure 2 As shown, the improved YOLO model includes: a Backbone layer, a Neck layer and a Head layer; the feature fusion modules include: a first feature fusion module GI1, a second feature fusion module GI2, a third feature fusion module GI3, a fourth feature fusion module GI4, a fifth feature fusion module GI5, a sixth feature fusion module GI6, a seventh feature fusion module GI7 and an eighth feature fusion module GI8.

[0054] The Backbone layer includes: a radar and video fusion module TF, a first CBS module CBS1, a first feature fusion module GI1, a second CBS module CBS2, a second feature fusion module GI2, a third feature fusion module GI3, a fourth feature fusion module GI4 and an SPPF module connected in sequence.

[0055] The Neck layer includes: a PSA sampling attention module, an Upsample module, a first Concat module Concat1, a fifth feature fusion module GI5, a second Concat module Concat2, a sixth feature fusion module GI6, a third CBS module CBS3, a third Concat module Concat3, a seventh feature fusion module GI7, a fourth CBS module CBS4, a fourth Concat module Concat4 and an eighth feature fusion module GI8 connected in sequence.

[0056] The Head layer includes: a first Head module Head1, a second Head module Head2 and a third Head module Head3.

[0057] The second CBS module CBS2 is also connected to the second Concat module Concat2; the third feature fusion module GI3 is also connected to the first Concat module Concat1; the SPPF module is also connected to the PSA sampling attention module; the PSA sampling attention module is also connected to the fourth Concat module Concat4; the fifth feature fusion module GI5 is also connected to the third Concat module Concat3; the sixth feature fusion module GI6 is also connected to the first Head module Head1; the seventh feature fusion module GI7 is also connected to the second Head module Head2; the eighth feature fusion module GI8 is also connected to the third Head module Head3.

[0058] Specifically, in the Backbone layer: Radar and video fusion module TF: This is the initial processing stage of the framework, responsible for fusing the input millimeter-wave radar data and video data to provide a unified data format for subsequent feature extraction. CBS modules (CBS1, CBS2, CBS3 and CBS4): This module consists of multiple convolutional layers, batch normalization layers and activation function layers, which are used to extract and enhance image features. Feature fusion modules (GI1, GI2, GI3, GI4, GI5, GI6, GI7 and GI8): Responsible for further processing and fusing features to enhance the adaptability and robustness of the network to different types of image features. SPPF module: Spatial pyramid pooling fusion module, used to integrate features of different scales and enhance the model's ability to recognize targets of different sizes.

[0059] In the Neck layer: Feature fusion module (GI1, GI2, GI3, GI4, GI5, GI6, GI7 and GI8): It appears again in the Neck part, indicating its importance in the feature extraction process, and is used to further refine and fuse features from the Backbone. Concat module: Responsible for merging feature maps from different sources or different processing stages, increasing the dimension and information content of the features, and providing support for more accurate target detection. Upsample module: Used to increase the resolution of feature maps, which is particularly important in target detection because it can help the model identify small targets or details more clearly. PSA sampling attention module PSA: Sampling attention module, which may be used to increase the model's attention to the target area, thereby improving the accuracy of detection.

[0060] In the Head layer: The Head module (Head1, Head2, and Head3) is responsible for the final detection tasks, such as bounding box regression and category classification. It may contain several fully connected layers or convolutional layers, as well as activation functions and loss functions for outputting the final detection results.

[0061] The entire improved YOLO framework realizes the complete process from data fusion to feature extraction and finally target detection through the collaborative work of these modules. Each module is optimized for a specific task, enabling the entire system to effectively detect targets in complex environments.

[0062] Exemplarily, feature extraction, feature splicing, feature fusion, residual processing, secondary splicing and convolution processing are performed on radar images and video images, including: A 3x3 convolution operation is performed on the radar image to extract a first radar image feature; and a 3x3 convolution operation is performed on the video image to extract a first video image feature.

[0063] The first radar image feature and the first video image feature are concatenated through a concat operation to obtain a first concatenated feature.

[0064] The first concatenated features are extracted and fused through the transformer Encoder layer to obtain the first pre-output features.

[0065] The radar image and the video image are subjected to residual processing and 3x3 convolution operations to obtain the second pre-output features.

[0066] The first pre-output feature and the second pre-output feature are concat-channeled to obtain the third pre-output feature.

[0067] A 1x1 convolution operation is performed on the third pre-output feature to obtain the output result of the radar and video fusion module.

[0068] Exemplarily, the radar and video fusion module is used to pre-process radar point cloud data and video data to obtain radar images and video images, including: The radar and video fusion module converts radar point cloud data into images to obtain radar images.

[0069] The video data is captured frame by frame using the OpenCV method to obtain the video image.

[0070] Specifically, Figure 3 As shown in the figure, the TF module first converts the point cloud data of the millimeter wave radar into a two-dimensional image, and uses OpenCV technology to capture video images frame by frame. Then, both types of images are processed through a 3x3 convolution kernel to extract image features. Subsequently, these features are merged in the channel dimension to generate a fused feature map.

[0071] The fused feature map is further processed by the Transformer Encoder layer, such as Figure 4 As shown in the figure, this layer consists of several key components: the Layer Norm layer is responsible for normalizing the features to maintain the stability of network training; the Multi-Head Attention layer enables the model to focus on multiple areas of the image at the same time; the Dropout layer randomly discards some features to avoid overfitting; Figure 5 As shown in the figure, the MLP Block layer contains a Linear layer and a GELU activation function, as well as a Dropout layer, which together enhance the model's ability to handle nonlinear problems. In addition, the Transformer Encoder layer adopts a residual structure, which ensures that information is effectively transmitted in the deep network by adding the input to the output of the MLP Block.

[0072] After the encoder layer, the original radar and video images will be processed again by 3x3 convolution and residual processing, which further extracts and fuses the image information. Finally, the output of the encoder layer is concatenated with the features after residual processing, and the feature extraction and channel number adjustment are performed through a 1x1 convolution kernel to ensure that the output features can be smoothly connected with the subsequent layers of the network.

[0073] The following is a further elaboration of the detailed workflow of the relevant part of the TF module: 1. Image preprocessing: The raw point cloud data of the millimeter-wave radar is first converted into an image format so that the radar data can be processed and analyzed in the form of images.

[0074] The video data is captured frame by frame through OpenCV, converting the continuous video stream into a series of individual image frames.

[0075] 2. Preliminary convolution operation: Apply 3x3 convolution kernels to the radar image and video image for convolution operation respectively. This step can be expressed as:

[0076]

[0077] here Represents a 3x3 convolution operation, which is used to extract image features.

[0078] 3. Feature stitching: The feature map of radar image and video image after convolution operation and Splice in the channel dimension to get the fused feature map :

[0079] 4. Transformer Encoder layer: Fusion feature map It is sent to the Transformer Encoder layer for further feature extraction and fusion. This layer consists of multiple components: Layer Norm: Normalize the feature map. The formula is:

[0080] in, are input features, are learnable parameters.

[0081] Multi-Head Attention: Through the multi-head attention mechanism, the model can focus on different parts of the image. The formula is

[0082]

[0083] in, denote the query, key, and value matrices respectively.

[0084] Dropout: Randomly discard some features to prevent overfitting, usually with a certain probability p during training.

[0085] MLP Block: Contains two linear transformation layers, with a GELU activation function and a Dropout layer in the middle. The formula is:

[0086] Residual connection: Add the output of the Encoder layer to the input. The formula is:

[0087] 5. Residual processing and further convolution operations: The original radar image and video image are residually processed and the 3x3 convolution operation is applied again to further extract and fuse information:

[0088]

[0089] 6. Final feature fusion: The features processed by the Transformer Encoder layer With residual processed radar and video image features and To splice:

[0090] Finally, a 1x1 convolution operation is used to extract features and adjust the number of channels of the concatenated features to meet the requirements of subsequent network layers:

[0091] here Represents a 1x1 convolution kernel, which is used to extract the final features and adjust the number of channels.

[0092] Through this design, the TF module not only realizes the effective fusion of radar and video images, but also significantly improves the accuracy and efficiency of target detection through in-depth feature extraction and fusion of the Transformer Encoder layer.

[0093] Exemplarily, the improved YOLO model also includes a feature fusion module; Based on the processing results of radar point cloud data and video data by the radar and video fusion module in the improved YOLO model, the target detection results are obtained, including: The processing results of the radar point cloud data and the video data by the radar and video fusion module in the improved YOLO model are processed by at least one feature fusion module to obtain a target detection result; wherein the feature fusion module is used to perform residual processing, convolution, depth-separable convolution processing, concat fusion and add fusion operation processing on the input image.

[0094] Exemplarily, the input image is subjected to residual processing, convolution, depthwise separable convolution, concat fusion and add fusion operations, including: Perform residual processing on the input image to obtain the first feature.

[0095] Perform a 1xk convolution operation on the input image to obtain the second feature.

[0096] Perform a kx1 convolution operation on the input image to obtain the third feature.

[0097] Ghost convolution and 3x3 depth-separable convolution are performed on the input image to obtain the fourth feature.

[0098] The second feature, the third feature and the fourth feature are concat-fused to obtain the first fused feature.

[0099] The first feature and the first fusion feature are added to obtain the fourth pre-output feature.

[0100] Ghost convolution and 1x1 convolution are performed on the fourth pre-output feature in sequence to obtain the output result of the feature fusion module.

[0101] Exemplarily, the input image is subjected to residual processing, convolution, depthwise separable convolution, concat fusion and add fusion operations, including: Perform residual processing on the input image to obtain the first feature.

[0102] Perform a 1xk convolution operation on the input image to obtain the second feature Perform a kx1 convolution operation on the input image to obtain the third feature;.

[0103] Ghost convolution and 3x3 depth-separable convolution are performed on the input image to obtain the fourth feature.

[0104] The second feature, the third feature and the fourth feature are concat-fused to obtain the first fused feature.

[0105] Repeat the above steps multiple times to obtain multiple first fusion features.

[0106] An add operation is performed on the first feature and the multiple first fusion features to obtain a fourth pre-output feature.

[0107] Ghost convolution and 1x1 convolution are performed on the fourth pre-output feature in sequence to obtain the output result of the feature fusion module.

[0108] The GI module is designed to significantly improve the accuracy and efficiency of target detection through highly customized convolution operations and feature fusion strategies. Specifically, Figure 6As shown in the figure, the GI module first dispatches the input image to four independent processing branches. In the first branch, the image undergoes residual processing, which enhances the flow of information in the deep network by adding residual connections between the input and the output of the convolution layer, helps alleviate the gradient vanishing problem, and enables the network to learn more complex features. The second and third branches apply 1xk and kx1 convolution operations, respectively. These operations focus on extracting features of the image in a specific direction, while effectively controlling the computational cost by using a smaller convolution kernel size. The fourth branch uses Ghost convolution, an innovative convolution method that can increase the parameters and capacity of the network without significantly increasing the computational burden, and then further refines the feature extraction through 3x3 depthwise separable convolution (DW convolution).

[0109] In the feature fusion stage, the outputs of branches two and three are first concatenated in the channel dimension to integrate feature information from different directions. Then, this fused feature is concatenated again with the feature processed by Ghost convolution and DW convolution of branch four to achieve deep fusion of features. In order to further enhance the expressiveness of the features, the entire feature extraction and fusion process can be repeated n times as needed, where n is a configurable parameter that enables the network to adjust its complexity and performance according to specific tasks. After multiple iterations, the residual features of branch one are added to the fused features to achieve deeper feature fusion. Finally, further feature extraction and parameter reduction operations are performed through Ghost convolution, which uses 1x1 convolution to adjust the number of channels to ensure the compatibility of the output features with subsequent network layers.

[0110] The detailed working mechanism of the GI module is as follows: 1. Input image processing: Input Image It is sent to the GI module and prepared for feature extraction.

[0111] 2. Branch 1 (residual processing): image It is processed through a residual block RR, which contains a convolution operation and a possible activation function:

[0112] Residual Connection It helps the flow of information in the network and reduces the gradient vanishing problem.

[0113] 3. Branches 2 and 3 (directional convolution): Branch two pairs of images Apply 1xk convolution:

[0114] Branch three pairs of images Apply kx1 convolution:

[0115] in, and Represents 1xk and kx1 convolution kernels, which extract image features in the horizontal and vertical directions respectively.

[0116] 4. Branch 4 (Ghost convolution and DW convolution): Ghost convolution G first processes the image :

[0117] Next, a 3x3 depth-wise separable convolution DW is applied:

[0118] Ghost convolution enhances the feature extraction capability by increasing the parameters and capacity of the network without significantly increasing the computational burden.

[0119] 5. Feature Fusion The outputs of branches two and three are first concatenated in the channel dimension:

[0120] This fused feature is concat-operated with the feature of branch 4 again:

[0121] 6. Repeat the operation: The entire process of feature extraction and fusion can be repeated n times, where n is a configurable parameter, to further enhance the expressiveness of the features.

[0122] 7. Residual feature fusion: Add the residual features of branch one to the fused features:

[0123] 8. Further feature extraction and parameter reduction: Further processing using Ghost convolution :

[0124] Finally, the number of channels is adjusted through 1x1 convolution:

[0125] in, Represents a 1x1 convolution kernel, which is used to extract the final features and adjust the number of channels to match the requirements of subsequent network layers.

[0126] The design of the GI module improves the performance of the network in target detection tasks through highly customized convolution operations and feature fusion strategies. Through flexible configuration and efficient feature extraction mechanism, the GI module enhances the adaptability and robustness of the network to different types of image features, providing strong technical support for target detection in complex environments. The design of this module enables the network to effectively detect targets under various conditions, such as different lighting and weather conditions, thereby improving the accuracy and reliability of detection.

[0127] The above-mentioned target detection method based on radar and video fusion processes radar point cloud data and video data through radar and video fusion modules, realizes deep fusion of the two data, and realizes efficient and accurate target detection through the improved TF-YOLO architecture. Compared with the prior art, the present invention shows significant performance advantages under conditions such as bad weather, low lighting or target occlusion because it integrates the rich texture information of video data and the precise ranging capability of radar data, reduces the dependence on environmental conditions, and improves the accuracy of target detection; it also reduces the consumption of computing resources through efficient data processing and advanced feature extraction technology, and maintains fast response capability, thereby improving real-time processing capabilities.

[0128] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0129] Corresponding to the target detection method based on radar and video fusion described in the above embodiment, Figure 5 A structural block diagram of a target detection device based on radar and video fusion provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0130] See also Figure 7 , the target detection device based on radar and video fusion in the embodiment of the present application may include: The data acquisition module 201 is used to acquire radar point cloud data and video data of the target to be detected.

[0131] The result output module 202 is used to obtain the target detection result based on the processing result of the radar point cloud data and the video data by the radar and video fusion module in the improved YOLO model; wherein the radar and video fusion module is used to pre-process the radar point cloud data and the video data to obtain the radar image and the video image, and perform feature extraction, feature splicing, feature fusion, residual processing, secondary splicing and convolution processing on the radar image and the video image.

[0132] Exemplarily, feature extraction, feature splicing, feature fusion, residual processing, secondary splicing and convolution processing are performed on radar images and video images, including: A 3x3 convolution operation is performed on the radar image to extract a first radar image feature; and a 3x3 convolution operation is performed on the video image to extract a first video image feature.

[0133] The first radar image feature and the first video image feature are concatenated through a concat operation to obtain a first concatenated feature.

[0134] The first concatenated features are extracted and fused through the transformer Encoder layer to obtain the first pre-output features.

[0135] The radar image and the video image are subjected to residual processing and 3x3 convolution operations to obtain the second pre-output features.

[0136] The first pre-output feature and the second pre-output feature are concat-channeled to obtain the third pre-output feature.

[0137] A 1x1 convolution operation is performed on the third pre-output feature to obtain the output result of the radar and video fusion module.

[0138] Exemplarily, the improved YOLO model also includes a feature fusion module; The result output module 202 can also be used to: The processing results of the radar point cloud data and the video data by the radar and video fusion module in the improved YOLO model are processed by at least one feature fusion module to obtain a target detection result; wherein the feature fusion module is used to perform residual processing, convolution, depth-separable convolution processing, concat fusion and add fusion operation processing on the input image.

[0139] Exemplarily, the input image is subjected to residual processing, convolution, depthwise separable convolution, concat fusion and add fusion operations, including: Perform residual processing on the input image to obtain the first feature.

[0140] Perform a 1xk convolution operation on the input image to obtain the second feature.

[0141] Perform a kx1 convolution operation on the input image to obtain the third feature.

[0142] Ghost convolution and 3x3 depth-separable convolution are performed on the input image to obtain the fourth feature.

[0143] The second feature, the third feature and the fourth feature are concat-fused to obtain the first fused feature.

[0144] The first feature and the first fusion feature are added to obtain the fourth pre-output feature.

[0145] Ghost convolution and 1x1 convolution are performed on the fourth pre-output feature in sequence to obtain the output result of the feature fusion module.

[0146] Exemplarily, the input image is subjected to residual processing, convolution, depthwise separable convolution, concat fusion and add fusion operations, including: Perform residual processing on the input image to obtain the first feature.

[0147] Perform a 1xk convolution operation on the input image to obtain the second feature Perform a kx1 convolution operation on the input image to obtain the third feature;.

[0148] Ghost convolution and 3x3 depth-separable convolution are performed on the input image to obtain the fourth feature.

[0149] The second feature, the third feature and the fourth feature are concat-fused to obtain the first fused feature.

[0150] Repeat the above steps multiple times to obtain multiple first fusion features.

[0151] An add operation is performed on the first feature and the multiple first fusion features to obtain a fourth pre-output feature.

[0152] Ghost convolution and 1x1 convolution are performed on the fourth pre-output feature in sequence to obtain the output result of the feature fusion module.

[0153] Exemplarily, the improved YOLO model includes: a Backbone layer, a Neck layer and a Head layer; the feature fusion module includes: a first feature fusion module, a second feature fusion module, a third feature fusion module, a fourth feature fusion module, a fifth feature fusion module, a sixth feature fusion module, a seventh feature fusion module and an eighth feature fusion module.

[0154] The Backbone layer includes: a radar and video fusion module, a first CBS module, a first feature fusion module, a second CBS module, a second feature fusion module, a third feature fusion module, a fourth feature fusion module and an SPPF module connected in sequence.

[0155] The Neck layer includes: a PSA sampling attention module, an Upsample module, a first Concat module, a fifth feature fusion module, a second Concat module, a sixth feature fusion module, a third CBS module, a third Concat module, a seventh feature fusion module, a fourth CBS module, a fourth Concat module and an eighth feature fusion module connected in sequence.

[0156] The Head layer includes: a first Head module, a second Head module and a third Head module.

[0157] The second CBS module is also connected to the second Concat module; the third feature fusion module is also connected to the first Concat module; the SPPF module is also connected to the PSA sampling attention module; the PSA sampling attention module is also connected to the fourth Concat module; the fifth feature fusion module is also connected to the third Concat module; the sixth feature fusion module is also connected to the first Head module; the seventh feature fusion module is also connected to the second Head module; the eighth feature fusion module is also connected to the third Head module.

[0158] Exemplarily, the radar and video fusion module is used to pre-process radar point cloud data and video data to obtain radar images and video images, including: The radar and video fusion module converts radar point cloud data into images to obtain radar images.

[0159] The video data is captured frame by frame using the OpenCV method to obtain the video image.

[0160] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0161] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0162] The present application also provides a terminal device, see Figure 8 The terminal device 300 may include: at least one processor 310 and a memory 320, wherein the memory 320 is used to store a computer program 321, and the processor 310 is used to call and run the computer program 321 stored in the memory 320 to implement the steps in any of the above-mentioned method embodiments, for example Figure 1 Step 101 to step 102 in the embodiment shown. Alternatively, when the processor 310 executes the computer program, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 7 The functions of each module are shown.

[0163] Exemplarily, the computer program 321 may be divided into one or more modules / units, one or more modules / units are stored in the memory 320, and are executed by the processor 310 to complete the present application. The one or more modules / units may be a series of computer program segments that can complete specific functions, and the program segments are used to describe the execution process of the computer program in the terminal device 300.

[0164] Those skilled in the art will understand that Figure 8 It is only an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, such as input and output devices, network access devices, buses, etc.

[0165] The processor 310 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0166] The memory 320 may be an internal storage unit of the terminal device, or an external storage device of the terminal device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. The memory 320 is used to store the computer program and other programs and data required by the terminal device. The memory 320 may also be used to temporarily store data that has been output or is to be output.

[0167] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0168] The target detection method based on radar and video fusion provided in the embodiment of the present application can be applied to terminal devices such as computers, wearable devices, vehicle-mounted devices, tablet computers, laptops, netbooks, etc. The embodiment of the present application does not impose any restrictions on the specific type of terminal devices.

[0169] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in each embodiment of the target detection method based on radar and video fusion can be implemented.

[0170] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in each embodiment of the above-mentioned target detection method based on radar and video fusion when executing the computer program product.

[0171] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0172] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0173] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0174] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0175] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A target detection method based on radar and video fusion, characterized in that: include: Obtain radar point cloud data and video data of the target to be detected; Based on the processing results of the radar point cloud data and the video data by the radar and video fusion module in the improved YOLO model, a target detection result is obtained; wherein the radar and video fusion module is used to preprocess the radar point cloud data and the video data to obtain a radar image and a video image, and perform feature extraction, feature splicing, feature fusion, residual processing, secondary splicing and convolution processing on the radar image and the video image.

2. The target detection method based on radar and video fusion as claimed in claim 1, characterized in that: The performing feature extraction, feature splicing, feature fusion, residual processing, secondary splicing and convolution processing on the radar image and the video image includes: Performing a 3x3 convolution operation on the radar image to extract a first radar image feature; performing a 3x3 convolution operation on the video image to extract a first video image feature; Perform feature splicing on the first radar image feature and the first video image feature through a concat operation to obtain a first splicing feature; Extracting and fusing the first concatenated features through a transformer encoder layer to obtain first pre-output features; Performing residual processing and a 3x3 convolution operation on the radar image and the video image to obtain a second pre-output feature; Perform concat channel concatenation on the first pre-output feature and the second pre-output feature to obtain a third pre-output feature; A 1x1 convolution operation is performed on the third pre-output feature to obtain an output result of the radar and video fusion module.

3. The target detection method based on radar and video fusion as claimed in claim 2, characterized in that: The improved YOLO model also includes a feature fusion module; The target detection result is obtained by processing the radar point cloud data and the video data by the radar and video fusion module in the improved YOLO model, including: The processing results of the radar point cloud data and the video data by the radar and video fusion module in the improved YOLO model are processed by at least one of the feature fusion modules to obtain a target detection result; wherein the feature fusion module is used to perform residual processing, convolution, depth-separable convolution processing, concat fusion and add fusion operation processing on the input image.

4. The target detection method based on radar and video fusion as claimed in claim 3, characterized in that: The performing residual processing, convolution, depth-separable convolution processing, concat fusion and add fusion operation processing on the input image includes: Perform residual processing on the input image to obtain the first feature; Perform 1xk convolution operation on the input image to obtain the second feature; Perform a kx1 convolution operation on the input image to obtain the third feature; Perform Ghost convolution and 3x3 depth-separable convolution on the input image to obtain the fourth feature; Concat the second feature, the third feature and the fourth feature to obtain a first fused feature; Performing an add operation on the first feature and the first fusion feature to obtain a fourth pre-output feature; Ghost convolution and 1x1 convolution are performed on the fourth pre-output feature in sequence to obtain the output result of the feature fusion module.

5. The target detection method based on radar and video fusion as claimed in claim 3, characterized in that: The performing residual processing, convolution, depth-separable convolution processing, concat fusion and add fusion operation processing on the input image includes: Perform residual processing on the input image to obtain the first feature; Perform 1xk convolution operation on the input image to obtain the second feature; Perform a kx1 convolution operation on the input image to obtain the third feature; Perform Ghost convolution and 3x3 depth-separable convolution on the input image to obtain the fourth feature; Concat the second feature, the third feature and the fourth feature to obtain a first fused feature; Repeat the above steps multiple times to obtain multiple first fusion features; Performing an add operation on the first feature and the multiple first fusion features to obtain a fourth pre-output feature; Ghost convolution and 1x1 convolution are performed on the fourth pre-output feature in sequence to obtain the output result of the feature fusion module.

6. The target detection method based on radar and video fusion as claimed in claim 3, characterized in that: The improved YOLO model includes: a Backbone layer, a Neck layer and a Head layer; the feature fusion module includes: a first feature fusion module, a second feature fusion module, a third feature fusion module, a fourth feature fusion module, a fifth feature fusion module, a sixth feature fusion module, a seventh feature fusion module and an eighth feature fusion module; The Backbone layer includes: a radar and video fusion module, a first CBS module, the first feature fusion module, a second CBS module, the second feature fusion module, the third feature fusion module, the fourth feature fusion module and an SPPF module connected in sequence; The Neck layer includes: a PSA sampling attention module, an Upsample module, a first Concat module, the fifth feature fusion module, a second Concat module, the sixth feature fusion module, a third CBS module, a third Concat module, the seventh feature fusion module, a fourth CBS module, a fourth Concat module and the eighth feature fusion module connected in sequence; The Head layer includes: a first Head module, a second Head module and a third Head module; The second CBS module is also connected to the second Concat module; the third feature fusion module is also connected to the first Concat module; the SPPF module is also connected to the PSA sampling attention module; the PSA sampling attention module is also connected to the fourth Concat module; the fifth feature fusion module is also connected to the third Concat module; the sixth feature fusion module is also connected to the first Head module; the seventh feature fusion module is also connected to the second Head module; the eighth feature fusion module is also connected to the third Head module.

7. The target detection method based on radar and video fusion as claimed in claim 2, characterized in that: The radar and video fusion module is used to pre-process the radar point cloud data and the video data to obtain a radar image and a video image, including: The radar and video fusion module converts the radar point cloud data into an image to obtain the radar image; The video data is intercepted frame by frame by using the OpenCV method to obtain the video image.

8. A target detection device based on radar and video fusion, characterized in that: include: A data acquisition module is used to acquire radar point cloud data and video data of the target to be detected; The result output module is used to obtain the target detection result based on the processing result of the radar point cloud data and the video data by the radar and video fusion module in the improved YOLO model; wherein the radar and video fusion module is used to pre-process the radar point cloud data and the video data to obtain a radar image and a video image, and perform feature extraction, feature splicing, feature fusion, residual processing, secondary splicing and convolution processing on the radar image and the video image.

9. A terminal device, comprising: A processor and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the target detection method based on radar and video fusion as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the target detection method based on radar and video fusion as described in any one of claims 1 to 7 is implemented.

Images