Multi-mode all-weather target detection method and device, electronic equipment and medium

Abstract

The invention relates to the technical field of computer vision, in particular to a multi-mode all-weather target detection method and device, electronic equipment and a medium, and the method comprises the steps: obtaining an original input image, and carrying out the multi-scale preprocessing of the original input image, and obtaining an image after illumination compensation; performing 2D target detection on the image after illumination compensation based on a preset YOLOv7 network to obtain a 2D detection result; projecting the 2D detection result to a point cloud space based on a preset calibration matrix, generating a target view cone region, and performing feature extraction on the point cloud data in the target view cone region to obtain geometric features of the target view cone region; and decoding the geometric features to obtain the category of the detection target and the 3D bounding box information of the detection target. Therefore, the problems of high calculation complexity, insufficient generalization ability, difficulty in balancing detection precision and efficiency and the like in the background technology are solved, and the accuracy and real-time performance of target detection are improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a multimodal all-weather target detection method, device, electronic device, and medium. Background Art

[0002] Multimodal detection technology plays an important role in the environmental perception of combined assisted driving systems. It improves the system's environmental perception capabilities by fusing heterogeneous data such as cameras, laser radar (LiDAR), millimeter-wave radar and ultrasonic sensors.

[0003] Among related technologies, current mainstream solutions include the unified representation-based multimodal fusion solution (BEVFusion) and the multi-task learning-based fusion framework (M3Net). The unified representation-based multimodal fusion solution (BEVFusion) maps camera images and LiDAR point clouds into a bird's-eye view (BEV) space, achieving cross-modal fusion through feature concatenation. For example, BEVFusion utilizes BEV feature alignment to address spatial consistency in multimodal data. However, detection accuracy for distant targets decreases due to BEV resolution limitations, and information is lost during projection or quantization. M3Net, a multi-task learning-based fusion framework, simultaneously handles 3D detection, segmentation, and occupancy prediction through Modality Adaptive Feature Integration (MAFI) and task-specific query initialization. This solution achieves an average decoder density (NDS) of 71.0% on the nuScenes dataset, but its high decoder complexity limits its real-time performance.

[0004] In practical applications, despite the rapid development of combined assisted driving perception, it still faces bottlenecks in computational complexity, poor generalization and adaptability to complex scenarios, and a difficult balance between detection accuracy and efficiency. On the one hand, the large number of parameters required for cross-modal feature splicing and fusion leads to high computational complexity and an inability to meet real-time requirements. On the other hand, existing single-sensor-based methods significantly degrade in performance in extreme weather (such as rain and fog) or occlusion scenarios. For example, LiDAR point clouds experience increased noise in rain and fog, and cameras fail in low light. In addition, while lidar-based image detection offers high accuracy, it is inefficient. While monocular 3D-based detection methods offer high detection speeds, they struggle to achieve high levels of accuracy because depth information is calculated from consecutive frames of RGB images, a problem that urgently needs to be addressed. Summary of the Invention

[0005] The present application provides a multimodal all-weather target detection method, device, electronic device and medium to solve the problems of high computational complexity, insufficient generalization capability and difficulty in balancing detection accuracy and efficiency in the background technology, thereby improving the accuracy and real-time performance of target detection.

[0006] The first embodiment of the present application provides a multimodal all-weather target detection method, comprising the following steps: Acquire an original input image, and perform multi-scale preprocessing on the original input image to obtain an image after illumination compensation; Performing 2D target detection on the illumination-compensated image based on a preset YOLOv7 network to obtain a 2D detection result; Projecting the 2D detection results into a point cloud space based on a preset calibration matrix to generate a target view cone area, and performing feature extraction on the point cloud data within the target view cone area to obtain geometric features of the target view cone area; The geometric features are decoded to obtain the category of the detected target and the 3D bounding box information of the detected target.

[0007] According to one embodiment of the present application, performing multi-scale preprocessing on the original input image to obtain an illumination-compensated image includes: Enhance the original input image based on a preset multi-scale Retinex algorithm to obtain an enhanced image; Performing color restoration on the enhanced image based on a preset color restoration algorithm to obtain a color-restored image; The color-restored image is converted from the RGB color space to the HSV color space to obtain the illumination-compensated image.

[0008] According to one embodiment of the present application, performing 2D target detection on the illumination-compensated image based on a preset YOLOv7 network to obtain a 2D detection result includes: Based on a preset YOLOv7 network, the image after illumination compensation is size-normalized, and feature extraction is performed on the normalized image to obtain initial features; Performing feature fusion on the initial features to obtain fused features, and performing prediction on the fused features to obtain multiple prediction results; The multiple prediction results are post-processed based on a preset non-maximum suppression algorithm, and the prediction result with the highest confidence is used as the 2D detection result.

[0009] According to one embodiment of the present application, projecting the 2D detection result into a point cloud space based on a preset calibration matrix to generate a target frustum area includes: generating a 2D bounding box based on the 2D detection result; The 2D bounding box is projected into the point cloud space based on a preset calibration matrix to construct the target frustum area.

[0010] According to one embodiment of the present application, extracting features from point cloud data within the target view frustum region to obtain geometric features of the target view frustum region includes: Filtering point cloud data from the target viewing cone area to obtain a local point cloud set; Performing feature extraction on each point in the local point cloud set to obtain a plurality of local geometric features; The local geometric features are aggregated to obtain geometric features of the target viewing cone area.

[0011] According to the multimodal, all-weather target detection method provided in an embodiment of the present application, the original input image is subjected to multi-scale preprocessing to obtain an illumination-compensated image. 2D target detection is performed on the illumination-compensated image based on a preset YOLOv7 network to obtain a 2D detection result. The 2D detection result is projected into a point cloud space based on a preset calibration matrix, and feature extraction is performed on the point cloud data within the target view cone to obtain the geometric features of the target view cone. The geometric features are decoded to obtain the category of the detected target and the 3D bounding box information of the detected target. This method solves the problems of high computational complexity, insufficient generalization capability, and difficulty in balancing detection accuracy and efficiency in the background art, thereby improving the accuracy and real-time performance of target detection.

[0012] A second embodiment of the present application provides a multimodal all-weather target detection device, comprising: An acquisition module is used to acquire an original input image and perform multi-scale preprocessing on the original input image to obtain an image after illumination compensation; A detection module is used to perform 2D target detection on the illumination-compensated image based on a preset YOLOv7 network to obtain a 2D detection result; a feature extraction module, configured to project the 2D detection results into a point cloud space based on a preset calibration matrix to generate a target view cone region, and perform feature extraction on the point cloud data within the target view cone region to obtain geometric features of the target view cone region; The decoding module is used to decode the geometric features to obtain the category of the detection target and the 3D bounding box information of the detection target.

[0013] According to one embodiment of the present application, the acquisition module is configured to: Enhance the original input image based on a preset multi-scale Retinex algorithm to obtain an enhanced image; Performing color restoration on the enhanced image based on a preset color restoration algorithm to obtain a color-restored image; The color-restored image is converted from the RGB color space to the HSV color space to obtain the illumination-compensated image.

[0014] According to one embodiment of the present application, the detection module is configured to: Based on a preset YOLOv7 network, the image after illumination compensation is size-normalized, and feature extraction is performed on the normalized image to obtain initial features; Performing feature fusion on the initial features to obtain fused features, and performing prediction on the fused features to obtain multiple prediction results; The multiple prediction results are post-processed based on a preset non-maximum suppression algorithm, and the prediction result with the highest confidence is used as the 2D detection result.

[0015] According to one embodiment of the present application, the feature extraction module is used to: generating a 2D bounding box based on the 2D detection result; The 2D bounding box is projected into the point cloud space based on a preset calibration matrix to construct the target frustum area.

[0016] According to one embodiment of the present application, the feature extraction module is used to: Filtering point cloud data from the target viewing cone area to obtain a local point cloud set; Performing feature extraction on each point in the local point cloud set to obtain a plurality of local geometric features; The local geometric features are aggregated to obtain geometric features of the target viewing cone area.

[0017] According to the multimodal, all-weather target detection device provided in an embodiment of the present application, the original input image is subjected to multi-scale preprocessing to obtain an illumination-compensated image. 2D target detection is performed on the illumination-compensated image based on a preset YOLOv7 network to obtain a 2D detection result. The 2D detection result is projected into a point cloud space based on a preset calibration matrix, and feature extraction is performed on the point cloud data within the target view cone area to obtain the geometric features of the target view cone area. The geometric features are decoded to obtain the category of the detected target and the 3D bounding box information of the detected target. This solves the problems of high computational complexity, insufficient generalization capability, and difficulty in balancing detection accuracy and efficiency in the background art, thereby improving the accuracy and real-time performance of target detection.

[0018] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the multimodal all-weather target detection method as described in the above embodiment.

[0019] A fourth aspect of the present application provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the multimodal all-weather target detection method as described in the above embodiments.

[0020] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1 A flowchart of a multimodal all-weather target detection method provided according to an embodiment of the present application; Figure 2 is a flowchart of all-weather image processing according to one embodiment of the present application; Figure 3 is a flow chart of a 2D detection process according to one embodiment of the present application; Figure 4 is a flow chart of a 3D detection process according to one embodiment of the present application; Figure 5 Flowchart of a multimodal all-weather target detection method according to one embodiment of the present application; Figure 6 is a block diagram of a multimodal all-weather target detection device according to an embodiment of the present application; Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0023] The following describes the multimodal all-weather target detection method, device, electronic device and medium according to the embodiments of the present application with reference to the accompanying drawings.

[0024] The multimodal all-weather target detection method of the embodiment of the present application adopts a technical solution that uses a multi-scale Retinex image enhancement algorithm as image preprocessing, a lightweight YOLOv7 as a 2D prior, and PointNet to extract and detect features of the point cloud in the viewing cone obtained based on the 2D detection frame.

[0025] Specifically, Figure 1 A flowchart of a multimodal all-weather target detection method provided in an embodiment of the present application.

[0026] like Figure 1 As shown, the multimodal all-weather target detection method includes the following steps: In step S101 , an original input image is obtained, and multi-scale preprocessing is performed on the original input image to obtain an image after illumination compensation.

[0027] Furthermore, in some embodiments, the original input image is subjected to multi-scale preprocessing to obtain an illumination-compensated image, including: enhancing the original input image based on a preset multi-scale Retinex algorithm to obtain an enhanced image; performing color restoration on the enhanced image based on a preset color restoration algorithm to obtain a color-restored image; and converting the color-restored image from an RGB color space to an HSV color space to obtain an illumination-compensated image.

[0028] Specifically, the embodiment of the present application can obtain the original input image through multiple cameras, and use the multi-scale Retinex algorithm to pre-process the RGB image, combined with adaptive contrast enhancement (CLAHE) and color space conversion (RGB→HSV), to optimize the lighting distribution for rain, snow and fog scenes, that is, separate the brightness channel (V channel) in the HSV space, suppress haze noise through histogram equalization, and use the multi-scale Retinex algorithm to restore image details, enhance target edge features, improve the image quality of low-visibility scenes, provide robust input for subsequent detection, and reduce the false detection rate caused by weather interference.

[0029] Furthermore, the core idea of ​​the Retinex algorithm of the embodiment of the present application is to decompose the image into a reflection component and an illumination component. The reflection component represents the inherent characteristics of the object, while the illumination component represents the lighting conditions. The goal of the Retinex algorithm of the embodiment of the present application is to remove the influence of the illumination component and retain and enhance the reflection component. This process involves logarithmic transformation of the image and convolution operation using Gaussian function as the center surround function. Multi-scale Retinex is an algorithm further improved on the basis of single-scale Retinex. It performs single-scale Retinex enhancement on images at different scales and accumulates and averages the results to obtain the final enhanced image. As Figure 2The specific steps are as follows: First, define a scale list sigma_list containing different standard deviation values.

[0030] Secondly, loop through each standard deviation sigma in the scale list, call the single-scale Retinex algorithm to enhance the image, and gradually accumulate the results.

[0031] Furthermore, the accumulated image is divided by the length of the scale list to obtain the final enhanced image.

[0032] Finally, color restoration is achieved by performing a logarithmic operation on the pixel values ​​of each channel and multiplying them by coefficients alpha and beta.

[0033] Furthermore, the color restoration process of the embodiment of the present application specifically includes the following steps: first, the R, G, and B channels of the input image are separated, and the pixel values ​​of the three channels are respectively subjected to logarithmic operations to obtain a logarithmic image, and the logarithmic image is multiplied by coefficients alpha and beta to obtain a color-restored image.

[0034] Furthermore, the multi-scale Retinex and color restoration operations are combined to obtain the final enhanced image. The specific steps are as follows: (1) Convert the input image to floating-point type and add 1.0 to avoid division by zero errors.

[0035] (2) Call the multi-scale Retinex algorithm to enhance the image and obtain the enhanced image.

[0036] (3) Call the color restoration algorithm to restore the color of the image and obtain the color restored image.

[0037] (4) Adjust the brightness and color of the enhanced image to obtain the final enhanced image.

[0038] (5) Limit the pixel range of the final enhanced image to ensure that the pixel value is between 0-255.

[0039] (6) Convert the final enhanced image to an unsigned integer type and return it.

[0040] In step S102, 2D target detection is performed on the illumination-compensated image based on a preset YOLOv7 network to obtain a 2D detection result.

[0041] Furthermore, in some embodiments, 2D target detection is performed on the illumination-compensated image based on a preset YOLOv7 network to obtain a 2D detection result, including: based on the preset YOLOv7 network, size normalizing the illumination-compensated image, and performing feature extraction on the normalized image to obtain initial features; performing feature fusion on the initial features to obtain fused features, and predicting the fused features to obtain multiple prediction results; post-processing the multiple prediction results based on a preset non-maximum suppression algorithm, and taking the prediction result with the highest confidence as the 2D detection result.

[0042] Specifically, the embodiment of the present application detects the processed image based on the lightweight 2D detector of yolov7, improves the downsampling module Transition Block in yolov7, combines the convolution and maximum pooling layer operations, reduces feature loss, and provides accurate target position for 3D detection.

[0043] For example, Figure 3 As shown, the 2D detection process includes the following steps: First, the input image is preprocessed and uniformly scaled to a resolution of 640×640, and the model is adapted through operations such as adaptive padding and normalization.

[0044] Secondly, feature extraction is performed. The number of channels is adjusted through convolution and downsampling (such as MaxPool). The downsampling operation can remove redundant information in the feature map, reduce parameters and quickly increase the receptive field (understanding the receptive field). Both the maximum pooling layer and the average pooling layer can complete downsampling. The average pooling layer focuses on retaining more image background information. The maximum pooling layer retains sufficient texture features. The texture information of the object is more important for the detection of sonar images. However, downsampling is a process of feature loss, and the pooling operation is not learnable. Using convolution operations instead of maximum pooling can obtain better results. However, this method will increase the amount of calculation. In order to balance feature learning and detection efficiency, the embodiment of the present application applies the Transition block module in the designed detection network to combine the convolution and maximum pooling layer operations.

[0045] Furthermore, feature fusion is performed. Deep features are amplified and superimposed with shallow detail features to improve small target detection capabilities, while shallow features are compressed and combined with deep semantic features to optimize large target detection.

[0046] We further perform prediction and output: each feature grid predicts 3 anchor boxes (Anchor), and the output results include bounding boxes, category probabilities, and confidence levels.

[0047] Finally, post-processing is performed using non-maximum suppression (NMS) to filter overlapping boxes and retain the detection results with the highest confidence.

[0048] In step S103, the 2D detection results are projected into the point cloud space based on the preset calibration matrix to generate a target view frustum area, and feature extraction is performed on the point cloud data within the target view frustum area to obtain the geometric features of the target view frustum area.

[0049] Furthermore, in some embodiments, the 2D detection results are projected into the point cloud space based on a preset calibration matrix to generate a target frustum area, including: generating a 2D bounding box according to the 2D detection results; and projecting the 2D bounding box into the point cloud space based on the preset calibration matrix to construct a target frustum area.

[0050] Furthermore, in some embodiments, feature extraction is performed on point cloud data within the target view frustum area to obtain geometric features of the target view frustum area, including: filtering point cloud data from the target view frustum area to obtain a local point cloud set; performing feature extraction on each point in the local point cloud set to obtain multiple local geometric features; and aggregating the local geometric features to obtain geometric features of the target view frustum area.

[0051] Specifically, if Figure 4 As shown in the figure, based on the above 2D detection results, local features are extracted by sliding the view cone and multi-scale information is integrated with the fully convolutional network to achieve efficient and accurate 3D object detection. The specific steps are as follows: First, a frustum sequence is generated. Based on the 2D region, multiple frustums (similar to multiple 3D boxes at different depths) are generated along the camera's line of sight, covering the area where the object may be located. Each frustum segments the point cloud using a sliding step size, forming local point groups.

[0052] Specifically, a bounding box B(x_min, y_min, x_max, y_max) is generated based on the 2D detection results. The 2D box is projected into the point cloud space through the calibration matrix, and a frustum Frustum={p | p∈PointCloud, pProjection∈B} is constructed. PointNet is then applied within the frustum to extract local geometric features, replacing global point cloud processing. Compared to the global feature extraction network, the embodiment of the present application generates a series of frustums based on the 2D detection results and performs feature extraction within the frustum area, improving the efficiency of feature extraction while speeding up point cloud processing, achieving fast and accurate target detection.

[0053] Next, local features are extracted. For each point within the viewing frustum, features are extracted using the PointNet network. The point coordinates are converted to positions relative to the center of the frustum. Then, through a multi-layer network and maximum pooling, the scattered point information is compressed into a feature vector for each frustum.

[0054] Furthermore, feature fusion and learning are performed. The feature vectors of all frustums are concatenated into a single 2D image. Convolutional layers are used to downsample (compress) and upsample (restore) the feature map, fusing information from different frustums and capturing multi-scale features (such as near details and distant holistic features).

[0055] Finally, predict the 3D bounding box. Predict the position, size, and angle of the 3D box.

[0056] In step S104 , the geometric features are decoded to obtain the category of the detected target and the 3D bounding box information of the detected target.

[0057] Specifically, the multimodal features containing rich information are input into the detection head for decoding, and the category of the detected target and the position of the external 3D detection frame where the detected target is located are output.

[0058] Therefore, the embodiment of the present application proposes an image processing algorithm under rainy, snowy and foggy weather conditions, which provides good image quality for perception under different conditions of all weather conditions, improves the generalization of the detection method, and utilizes a lightweight 2D detection model to maintain good detection accuracy while having a smaller number of parameters, providing a priori for 3D detection, and forming a viewing cone based on the projection of the 2D detection frame into the point cloud coordinate system and performing feature extraction within the area to achieve 3D perception, avoid global extraction of point cloud features, and improve detection efficiency.

[0059] In order to facilitate those skilled in the art to more clearly and intuitively understand the multimodal all-weather target detection method of the embodiment of the present application, the following is combined with Figure 5 Provide detailed explanation.

[0060] The hardware and software platform for building, training and testing the algorithm in this embodiment is Ubuntu20.04+NvidiaRTX6000+Pytorch1.12+cuda11.4+cudnn9.0+python3.8.

[0061] like Figure 5 As shown, the multimodal all-weather target detection method includes the following steps: First, input image data, and perform all-weather algorithm processing on the input image, and use the YOLOv7 algorithm to identify the target object and output the 2D coordinates; Secondly, the 2D coordinates are projected into the point cloud coordinate system, and a series of point cloud frustums are formed based on the 2D coordinates; Furthermore, PointNet is used to extract point cloud features, and multi-scale features are fused through a fully convolutional network; Finally, the detection head outputs a 3D bounding box that marks the location of the target object.

[0062] According to the multimodal all-weather target detection method proposed in the embodiment of the present application, the original input image is preprocessed at multiple scales to obtain an image after illumination compensation; 2D target detection is performed on the image after illumination compensation based on the preset YOLOv7 network to obtain a 2D detection result; the 2D detection result is projected into the point cloud space based on a preset calibration matrix, and feature extraction is performed on the point cloud data within the target view cone area to obtain the geometric features of the target view cone area; the geometric features are decoded to obtain the category of the detected target and the 3D bounding box information of the detected target. Thus, the problems of high computational complexity, insufficient generalization ability, and difficulty in balancing detection accuracy and efficiency in the background technology are solved. By using image processing algorithms to process the collected RGB images, the detection accuracy in rainy, snowy and foggy weather conditions can be improved, the generalization ability can be improved, and the detection model can be converted to reduce the model complexity. The 2D detection is used to provide a priori, which is projected into the point cloud to form a series of view cones, reducing the time consumption of global feature extraction and improving detection efficiency.

[0063] Next, a multimodal all-weather target detection device proposed according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0064] Figure 6 4 is a block diagram of a multimodal all-weather target detection device according to an embodiment of the present application.

[0065] like Figure 6 As shown, the multimodal all-weather target detection device 10 includes: an acquisition module 100, a detection module 200, a feature extraction module 300 and a decoding module 400.

[0066] Among them, the acquisition module 100 is used to acquire the original input image and perform multi-scale preprocessing on the original input image to obtain the image after illumination compensation; the detection module 200 is used to perform 2D target detection on the image after illumination compensation based on the preset YOLOv7 network to obtain the 2D detection result; the feature extraction module 300 is used to project the 2D detection result to the point cloud space based on the preset calibration matrix, generate the target cone area, and perform feature extraction on the point cloud data within the target cone area to obtain the geometric features of the target cone area; the decoding module 400 is used to decode the geometric features to obtain the category of the detected target and the 3D bounding box information of the detected target.

[0067] Furthermore, in some embodiments, the acquisition module 100 is used to: enhance the original input image based on a preset multi-scale Retinex algorithm to obtain an enhanced image; restore the color of the enhanced image based on a preset color restoration algorithm to obtain a color-restored image; and convert the color-restored image from the RGB color space to the HSV color space to obtain a lighting-compensated image.

[0068] Furthermore, in some embodiments, the detection module 200 is used to: perform size normalization on the illumination-compensated image based on a preset YOLOv7 network, and perform feature extraction on the normalized image to obtain initial features; perform feature fusion on the initial features to obtain fused features, and predict the fused features to obtain multiple prediction results; post-process the multiple prediction results based on a preset non-maximum suppression algorithm, and use the prediction result with the highest confidence as the 2D detection result.

[0069] Furthermore, in some embodiments, the feature extraction module 300 is used to: generate a 2D bounding box according to the 2D detection result; and project the 2D bounding box into the point cloud space based on a preset calibration matrix to construct a target frustum area.

[0070] Furthermore, in some embodiments, the feature extraction module 300 is used to: filter point cloud data from the target viewing cone area to obtain a local point cloud set; perform feature extraction on each point in the local point cloud set to obtain multiple local geometric features; and aggregate the local geometric features to obtain geometric features of the target viewing cone area.

[0071] It should be noted that the aforementioned explanation of the embodiment of the multimodal all-weather target detection method is also applicable to the multimodal all-weather target detection device of this embodiment, and will not be repeated here.

[0072] According to the multimodal, all-weather target detection device proposed in the embodiment of the present application, the original input image is subjected to multi-scale preprocessing to obtain an illumination-compensated image. 2D target detection is performed on the illumination-compensated image based on a preset YOLOv7 network to obtain a 2D detection result. The 2D detection result is projected into a point cloud space based on a preset calibration matrix, and feature extraction is performed on the point cloud data within the target view cone area to obtain the geometric features of the target view cone area. The geometric features are decoded to obtain the category of the detected target and the 3D bounding box information of the detected target. This solves the problems of high computational complexity, insufficient generalization capability, and difficulty in balancing detection accuracy and efficiency in the background art, thereby improving the accuracy and real-time performance of target detection.

[0073] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include: A memory 701 , a processor 702 , and a computer program stored in the memory 701 and executable on the processor 702 .

[0074] When the processor 702 executes the program, the multimodal all-weather target detection method provided in the above embodiment is implemented.

[0075] Furthermore, the electronic device further includes: The communication interface 703 is used for communication between the memory 701 and the processor 702 .

[0076] The memory 701 is used to store computer programs that can be run on the processor 702 .

[0077] The memory 701 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0078] If the memory 701, processor 702, and communication interface 703 are implemented independently, the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0079] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.

[0080] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0081] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multimodal all-weather target detection method.

[0082] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0083] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0084] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0085] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" is any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (not exhaustive) of computer-readable media include: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0086] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logical functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0087] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0088] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0089] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A multimodal all-weather target detection method, characterized in that: The following steps are involved: Acquire an original input image, and perform multi-scale preprocessing on the original input image to obtain an image after illumination compensation; Performing 2D target detection on the illumination-compensated image based on a preset YOLOv7 network to obtain a 2D detection result; Projecting the 2D detection results into a point cloud space based on a preset calibration matrix to generate a target view cone area, and performing feature extraction on the point cloud data within the target view cone area to obtain geometric features of the target view cone area; The geometric features are decoded to obtain the category of the detected target and the 3D bounding box information of the detected target.

2. The method according to claim 1, characterized in that The performing multi-scale preprocessing on the original input image to obtain an illumination-compensated image includes: Enhance the original input image based on a preset multi-scale Retinex algorithm to obtain an enhanced image; Performing color restoration on the enhanced image based on a preset color restoration algorithm to obtain a color-restored image; The color-restored image is converted from the RGB color space to the HSV color space to obtain the illumination-compensated image.

3. The method according to claim 1, characterized in that The 2D target detection is performed on the illumination-compensated image based on the preset YOLOv7 network to obtain a 2D detection result, including: Based on a preset YOLOv7 network, the image after illumination compensation is size-normalized, and feature extraction is performed on the normalized image to obtain initial features; Performing feature fusion on the initial features to obtain fused features, and performing prediction on the fused features to obtain multiple prediction results; The multiple prediction results are post-processed based on a preset non-maximum suppression algorithm, and the prediction result with the highest confidence is used as the 2D detection result.

4. The method according to claim 1, wherein The projecting of the 2D detection result into the point cloud space based on the preset calibration matrix to generate the target frustum area includes: generating a 2D bounding box based on the 2D detection result; The 2D bounding box is projected into the point cloud space based on a preset calibration matrix to construct the target frustum area.

5. The method according to claim 1, wherein The extracting features of the point cloud data within the target viewing cone area to obtain geometric features of the target viewing cone area includes: Filtering point cloud data from the target viewing cone area to obtain a local point cloud set; Performing feature extraction on each point in the local point cloud set to obtain a plurality of local geometric features; The local geometric features are aggregated to obtain geometric features of the target viewing cone area.

6. A multi-modal all-weather target detection device, characterized in that: include: An acquisition module is used to acquire an original input image and perform multi-scale preprocessing on the original input image to obtain an image after illumination compensation; A detection module is used to perform 2D target detection on the illumination-compensated image based on a preset YOLOv7 network to obtain a 2D detection result; a feature extraction module, configured to project the 2D detection results into a point cloud space based on a preset calibration matrix to generate a target view cone region, and perform feature extraction on the point cloud data within the target view cone region to obtain geometric features of the target view cone region; The decoding module is used to decode the geometric features to obtain the category of the detection target and the 3D bounding box information of the detection target.

7. The device according to claim 6, characterized in that The acquisition module is used to: Enhance the original input image based on a preset multi-scale Retinex algorithm to obtain an enhanced image; Performing color restoration on the enhanced image based on a preset color restoration algorithm to obtain a color-restored image; The color-restored image is converted from the RGB color space to the HSV color space to obtain the illumination-compensated image.

8. The device according to claim 6, characterized in that The detection module is used to: Based on a preset YOLOv7 network, the image after illumination compensation is size-normalized, and feature extraction is performed on the normalized image to obtain initial features; Performing feature fusion on the initial features to obtain fused features, and performing prediction on the fused features to obtain multiple prediction results; The multiple prediction results are post-processed based on a preset non-maximum suppression algorithm, and the prediction result with the highest confidence is used as the 2D detection result.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal all-weather target detection method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the multimodal all-weather target detection method according to any one of claims 1 to 5.

Images