Multi-modal target detection method and device and multi-modal identification system

Abstract

The invention provides a multi-modal target detection method and device and a multi-modal recognition system. The method comprises the steps that a scene image is acquired, and the scene image is an image of a current environment acquired by adopting image acquisition equipment; acquiring a point cloud image, the point cloud image being an image of point cloud data of the current environment acquired by using a laser radar; according to the scene image and the point cloud image, fusion is carried out to obtain a target image, and the fusion mode at least comprises image fusion and / or feature fusion; and performing target identification on the target image to obtain a target object in the target image. According to the scheme, the problem of low accuracy of target detection based on single data in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multimodal target detection method, device, computer program product and multimodal recognition system. Background Art

[0002] Traditional object detection methods mainly rely on 3D point cloud data provided by LiDAR or 2D image data provided by cameras. However, single-modality data sources often have limitations, affecting the accuracy and reliability of 3D object detection. Summary of the invention

[0003] The main purpose of the present application is to provide a multimodal target detection method, device, computer program product and multimodal recognition system to at least solve the problem of low accuracy of target detection based on single data in the prior art.

[0004] To achieve the above-mentioned purpose, according to one aspect of the present application, a multimodal target detection method is provided, comprising: acquiring a scene image, wherein the scene image is an image of a current environment acquired by an image acquisition device; acquiring a point cloud image, wherein the point cloud image is an image of point cloud data of the current environment acquired by a laser radar; fusing the scene image and the point cloud image to obtain a target image, wherein the fusion method includes at least image merging and / or feature fusion; performing target recognition on the target image to obtain a target object in the target image.

[0005] According to another aspect of the present application, a multimodal target detection device is provided, including: a first acquisition unit, used to acquire a scene image, wherein the scene image is an image of a current environment acquired by an image acquisition device; a second acquisition unit, used to acquire a point cloud image, wherein the point cloud image is an image of point cloud data of the current environment acquired by a laser radar; a fusion unit, used to fuse the scene image and the point cloud image to obtain a target image, wherein the fusion method includes at least image merging and / or feature fusion; and an identification unit, used to perform target identification on the target image to obtain a target object in the target image.

[0006] According to another aspect of the present application, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of any one of the multi-modal target detection methods.

[0007] According to another aspect of the present application, a multimodal recognition system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any of the multimodal target detection methods.

[0008] By applying the technical solution of the present application and taking the scene image collected by the image acquisition device and the point cloud image collected by the lidar as input for multimodal fusion, the features of the point cloud and the image can be fully utilized, the limitations of a single modality can be effectively reduced, and the present solution can achieve better target detection effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0010] Figure 1 A hardware structure block diagram of a mobile terminal for performing a multi-modal target detection method provided in an embodiment of the present application is shown;

[0011] Figure 2 A schematic diagram of a process flow of a multimodal target detection method provided according to an embodiment of the present application is shown;

[0012] Figure 3 A schematic diagram showing the network framework of the present solution is shown;

[0013] Figure 4 A schematic diagram of the process of another multimodal target detection method is shown;

[0014] Figure 5 A structural block diagram of a multi-modal target detection device provided according to an embodiment of the present application is shown.

[0015] The above drawings include the following reference numerals:

[0016] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0017] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0018] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0019] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] As introduced in the background technology, the accuracy of target detection based on single data in the prior art is low. To solve the above problem, the embodiments of the present application provide a multimodal target detection method, device, computer program product and multimodal recognition system.

[0021] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0022] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a multi-modal target detection method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0023] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the display method of device information in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0024] In this embodiment, a multimodal target detection method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0025] Figure 2 FIG. 1 is a flow chart of a multi-modal target detection method according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0026] Step S201, obtaining a scene image, wherein the scene image is an image of the current environment obtained by an image acquisition device;

[0027] Specifically, the visual image of the current environment is obtained through an image acquisition device, such as an RGB camera. This image contains rich color, texture and spatial detail information, which is used to depict objects, backgrounds and details in the environment. The acquisition of scene images is the basis for subsequent target recognition and feature extraction, and is one of the key inputs of the visual perception system.

[0028] Specifically, camera image data (i.e., scene images) are acquired by one or more cameras installed on the same vehicle or robot. The images captured by the camera provide rich color, texture, and shape information, which is essential for identifying the type of object and its surface features. In this solution, the image captured by the camera also corresponds to the same scene scanned by the lidar, that is, the two collect data in the same environment at the same time, but provide complementary perspectives from the perspectives of depth information and visual information respectively.

[0029] Step S202, obtaining a point cloud image, wherein the point cloud image is an image of point cloud data of the current environment obtained by using a laser radar;

[0030] Specifically, a point cloud image is actually a visual representation of the point cloud data obtained after the LiDAR scans the current environment. Point cloud data is generated by the LiDAR emitting lasers and receiving reflected signals. Each point cloud point carries a three-dimensional coordinate (x, y, z), representing the precise location of the surface point of an object in the environment. Although often referred to as a "point cloud image", it is actually a projection or visualization of point cloud data on a two-dimensional screen, which usually lacks color and texture information, but provides depth and geometric structure, which is essential for target detection and positioning in three-dimensional space.

[0031] Specifically, point cloud data is usually obtained by a laser radar (LiDAR) sensor. LiDAR emits laser pulses and receives light reflected by objects. It calculates the distance between the object and the sensor based on the round-trip time of light, thereby constructing a point cloud map of objects in a three-dimensional space. In the field of autonomous driving, LiDAR is installed on vehicles to capture three-dimensional information about the vehicle's surroundings in real time, such as the position and shape of other vehicles, pedestrians, obstacles, etc.

[0032] Step S203, fusing the scene image and the point cloud image to obtain a target image, wherein the fusion method at least includes image merging and / or feature fusion;

[0033] Specifically, image merging is usually performed at the visualization level, that is, combining the depth information of the point cloud data with the color and texture information of the scene image to generate a composite image, in which the point cloud data can be rendered into a specific color or shape and superimposed on the scene image.

[0034] Feature fusion, a deeper level of fusion occurs at the feature level, that is, after the image and point cloud data are extracted separately, their feature information is combined to obtain a more complete and descriptive feature representation. Feature fusion can be achieved by addition, multiplication, splicing, etc., thereby utilizing the depth information of the point cloud data and the semantic information of the scene image to enhance the robustness and accuracy of target recognition.

[0035] Specifically, data-level fusion and feature-level fusion are based on the point cloud data and image data acquired at the same time or scene. Through the registration and calibration of the coordinate system, it is ensured that each point in the point cloud data can be accurately mapped to the corresponding position on the image, thereby achieving data-level fusion and attaching the semantic information on the image (such as category labels) to the point cloud data to form a point cloud with semantic labels. Subsequently, this semantically rich point cloud data is input into the deep learning network together with the image features for feature-level fusion to improve the accuracy and robustness of 3D object detection.

[0036] Step S204: performing target recognition on the target image to obtain the target object in the target image.

[0037] Specifically, the fused target image is processed for target recognition. This step involves using a deep learning model, such as a convolutional neural network (CNN) or other target detection network, to identify and classify objects in the image. The model will identify and locate the target objects, such as vehicles and pedestrians, based on the feature information in the target image, including color, texture, shape, and depth. The result of target recognition usually includes the category label of each target and its position in the image, which can be given in the form of a two-dimensional bounding box or a three-dimensional bounding box.

[0038] Through this embodiment, the scene image collected by the image acquisition device and the point cloud image collected by the lidar are used as input for multimodal fusion, which can fully utilize the features of the point cloud and the image, and can effectively reduce the limitations of a single modality, so that this solution can achieve better target detection effects.

[0039] Specifically, 3D point cloud data can provide accurate depth information and geometric structure, but lacks semantic information such as color and texture that image data has, making it difficult for the model to accurately detect small objects and distant objects with a small number of point clouds, and difficult to distinguish between similar objects. The image acquired by the camera can provide rich semantic feature information such as shape, texture and color, but lacks depth information. Therefore, the solution of this application considers fusing the two modalities, while taking advantage of the advantages of point cloud data and image data to improve target detection performance. Depending on the fusion stage, the 3D target detection method of point cloud and image fusion can be divided into data-level fusion, feature-level fusion and decision-level fusion. EPnet fusion belongs to feature-level fusion, which combines different sensors such as lidar point cloud and camera images to effectively enhance point features. Effectively improves the detection effect of 3D objects.

[0040] Specifically, existing technologies that rely solely on single-mode data such as point clouds or images to achieve detection tasks are defective. Cameras are low-cost and can obtain image information with wide scenes and rich texture information, but lack depth information and are heavily dependent on lighting conditions, which can easily cause overexposure or shadow occlusion. It is difficult to make up for the lack of information by relying on algorithms. LiDAR can directly obtain the three-dimensional coordinate information of an object and is not affected by lighting conditions, but the point cloud generated by LiDAR is relatively sparse, and the semantic information is not rich enough to identify and classify objects. The feature fusion of the cloud and the image is not sufficient. Simply adding the features of the point cloud and image data may be worse than using data from a single mode, or the effect may not be significantly improved.

[0041] Therefore, this application designs a multimodal 3D target detection solution based on point cloud and image fusion, aiming to solve the limitations of relying solely on a single data source. Traditional image methods usually rely on images for target recognition, but in the absence of depth information, images often cannot accurately reflect the spatial position of the target; and although point cloud data can provide depth and spatial structure information, its sparsity and lack of semantic features limit its application in complex scenes. In response to these problems, the solution of this application effectively combines the advantages of point clouds and images, giving full play to their complementary characteristics, thereby achieving more accurate target classification and positioning.

[0042] In this solution, images provide rich colors, textures, and visual features, which help the model accurately identify and distinguish different types of targets; while point cloud data provides the depth information and spatial structure of the target, which can effectively supplement the image through multimodal fusion. The model can form a more comprehensive target recognition enhancement and understanding capability based on the two data sources, thereby achieving greater accuracy and reliability of target detection. More importantly, the use of multimodal fusion can enable the detection system to show stronger robustness and cognitive ability when facing noise, environmental interference or dynamic changes. The system can effectively reduce the errors or instability that may be caused by a single data source by combining the different characteristics of point clouds and images, thereby maintaining high-precision and accurate target detection capabilities in a changing and efficient environment.

[0043] Specifically, the network architecture involved in the solution of this application is as follows: Figure 3As shown in the figure, it includes a data-level fusion module, a lidar point cloud feature extraction module, an image feature extraction module, a feature fusion module and a target detection module. The multimodal fusion method with point cloud and image as input effectively reduces the limitations of a single modality and makes full use of the features of point cloud and image to achieve better detection effects. The detection framework of data set fusion and feature-level fusion is used at the same time. Data set fusion is to first attach the semantic segmentation information of the image to the point cloud through coordinate mapping to achieve the fusion of the two at the data level. Feature-level fusion is to effectively fuse the features of point cloud and image in the process of extracting the features of point cloud and image respectively to achieve better detection effects. The feature fusion module fuses the features from point cloud and image at different scales, and fully combines the features of the two.

[0044] In the solution of this application, fusion refers to combining different types of sensor data or their feature representations to enhance the performance of target detection. This fusion can occur at the data level, feature level, or decision level.

[0045] Data-level fusion: Merge point cloud and image data at the initial stage, for example, project the point cloud information onto the image and attach color or texture information from the image to each point cloud point.

[0046] Feature-level fusion: After data preprocessing, features are extracted from point cloud data and image data respectively, and then these features are combined at one or more levels. For example, image features and point cloud features can be complemented by feature matching or weighted summation.

[0047] Decision-level fusion: After the model has made preliminary detection results on the point cloud and image data separately, these results are combined to make the final decision.

[0048] In the EPNet algorithm, feature-level fusion is mainly involved. Through the point-by-point correspondence generator and the lidar-guided fusion module, the point cloud features and image features are mapped to the same channel, and then the weights of the image features are adaptively evaluated to combine the point cloud features with the image features to generate a more comprehensive feature representation, thereby improving the accuracy and robustness of 3D object detection. This fusion method can more effectively utilize the information of the two modalities, especially when dealing with small objects or distant objects, it can significantly improve the detection performance.

[0049] For example, fusion is mainly feature-level fusion, which means that after the point cloud data and image data are preliminarily processed, their feature representations are combined to make full use of the depth and geometry information of the point cloud and the semantic and texture information of the image. This fusion can make up for the shortcomings of a single data source, for example, the point cloud may lack texture and color information, while the image may lack depth information.

[0050] In addition, "image merging" generally refers to the process of combining two or more images into one image, which can be achieved based on a variety of different purposes and methods. Image merging can be divided into several main types, including image fusion, image stitching, image overlay, etc., each type varies according to its specific application and purpose.

[0051] Image fusion refers to the comprehensive processing of image data from different sensors or the same sensor at different time points to extract more useful information or enhance image quality. For example, in multimodal image fusion, infrared images and visible light images can be fused to obtain clear images under various lighting conditions; or RGB images can be fused with depth images to add depth information to the image. In the context of 3D object detection, image fusion refers to combining the geometric information in point cloud data with the rich visual information (such as color, texture) in image data to enhance the detection model's understanding and recognition of the target.

[0052] Image stitching is the process of combining multiple overlapping image fragments into a larger image. It is often used to create panoramic images or expand the field of view. During the stitching process, the image usually needs to be calibrated to ensure the correct alignment and fusion between different image fragments, thereby forming a coherent visual effect.

[0053] Image overlay refers to placing an image or a specific part of an image on top of another image. This is often used to add an image (such as an icon, text) to another image to provide additional information or visual effects. For example, in video surveillance, timestamps or identifiers can be superimposed on video footage; in augmented reality (AR) applications, virtual objects are superimposed on real-world images.

[0054] In the specific implementation process, when the fusion method includes image merging, the scene image and the point cloud image are fused to obtain the target image, which can be achieved by the following steps: using the DeeplabV3+ algorithm to perform semantic segmentation on the scene image to obtain a segmented scene image, wherein the segmented scene image includes the category of each pixel; projecting the points of the point cloud image into the scene image to obtain the correspondence between the points of the point cloud image and the pixels of the scene image; according to the correspondence, appending the categories of all the pixels in the segmented scene image to the point cloud image to obtain the target image.

[0055] In this solution, by attaching semantic category information to point cloud data, targets can be more accurately identified and classified, especially when dealing with small or distant targets. The color and texture information provided by the image can help the model distinguish and confirm the object category, thereby improving the accuracy of detection. Point cloud data itself may lack sufficient semantic features. Through data-level fusion, point cloud data obtains additional semantic information, which makes point cloud features richer and helps the model learn and understand target objects in complex scenes.

[0056] Specifically, first, the DeeplabV3+ algorithm is used to perform deep learning-driven semantic segmentation on the scene images obtained from the RGB camera. Semantic segmentation aims to identify which object category each pixel in the image belongs to, such as cars, pedestrians, roads, etc. DeeplabV3+ is an advanced semantic segmentation network that uses deep convolutional neural networks and spatial pyramid pooling (ASPP) modules to effectively extract multi-scale features, thereby providing accurate segmentation results while maintaining high resolution. The segmented scene image not only contains the visual information of the original RGB image, but also attaches the label of the category to which each pixel belongs, which provides rich semantic information for subsequent feature fusion.

[0057] Next, the point cloud data obtained from the LiDAR is projected into the segmented scene image through coordinate transformation to establish the relationship between each point in the point cloud image and the corresponding pixel in the scene image. This process utilizes the calibration information between the LiDAR and the camera, including the intrinsic parameter matrix, extrinsic parameter matrix, and projection matrix. By mapping the 3D point cloud data into the 2D image coordinate system, the corresponding position of each LiDAR point in the scene image can be found, thereby associating the rich semantic information of the scene image with the 3D structural information of the point cloud data.

[0058] After the correspondence between the point cloud points and the scene image pixels is obtained, the category label of each pixel in the segmented scene image is attached to its corresponding point cloud point to form a point cloud dataset with semantic category information. This point cloud data with semantic labels is called a target image or a fused point cloud image. It integrates the visual features in the RGB image and the depth information of the LiDAR point cloud, providing richer and more comprehensive input data for subsequent 3D target detection. Each point in the target image contains not only (x, y, z) coordinate information, but also category information (such as cars, pedestrians, background, etc.), which greatly enhances the descriptive ability of the point cloud data and enables the model to better understand the properties of objects in the environment.

[0059] Specifically, point cloud and image can be used as input, and semantic segmentation can be performed on the image first. The result is attached to the point cloud through coordinate mapping with the point cloud to obtain point cloud data with semantic information. At the same time, the point cloud and image are input into the EPNet network to obtain the final detection result.

[0060] Specifically, the above embodiment is data-level fusion. At this stage, the scheme attaches the semantic segmentation information of the image, that is, the classification probability of each pixel, to the corresponding point cloud data through coordinate mapping. This means that for each point in the point cloud, the system attaches a semantic label obtained from the image, such as belonging to categories such as "car", "pedestrian" or "background". This direct location-based information combination enhances the descriptive ability of the point cloud data, so that each point contains not only its three-dimensional spatial coordinates, but also possible category information.

[0061] Specifically, first, the raw point cloud data generated by the lidar and the raw image data taken by the camera are obtained. The point cloud data contains three-dimensional coordinate information (x, y, z), while the image data usually contains RGB color information. The image data is input into a pre-trained semantic segmentation network (such as DeeplabV3+), which assigns a category label to each pixel in the image, such as car, pedestrian, cyclist, or background. This step generates the category probability of each pixel, providing semantic information for subsequent fusion. Using the internal and external parameters of the camera and lidar, the points in the point cloud data are projected onto the image plane to determine the corresponding position of each point cloud point in the image. This calculates the two-dimensional coordinates of the point cloud point in the image coordinate system through the projection matrix and the calibration matrix. The category probability of each pixel obtained by image segmentation is attached to its corresponding point cloud point to form a point cloud dataset carrying semantic information. At this stage, each point cloud point not only contains three-dimensional coordinate information, but also carries additional category label information obtained from the image.

[0062] Specifically, the original RGB image acquired by the camera is input into a pre-trained deep learning network (such as DeeplabV3+) for semantic segmentation. The goal of the semantic segmentation network is to assign a category label to each pixel in the image, for example, labeling the pixel as "car", "pedestrian", "cyclist" or "background", etc. The segmentation result generated by this step contains the category information of the object to which each pixel belongs.

[0063] Next, the coordinate transformation relationship between the LiDAR point cloud and the camera image is used to project the point cloud data onto the image plane. This involves using the camera's intrinsic parameter matrix (i.e., the camera's focal length, principal point coordinates, and other parameters) and extrinsic parameter matrix (i.e., the rotation and translation of the LiDAR relative to the camera) to calculate the corresponding position of each point in the point cloud in the two-dimensional image through the projection formula. In this way, each point in the point cloud has its mapping point on the image, which can correspond to the pixel information of the image.

[0064] After determining the correspondence between the point cloud points and the image pixels, the category probability of each pixel output by the semantic segmentation network can be attached to the corresponding point cloud point. This means that each point cloud point not only contains its 3D coordinate information, but also carries semantic information obtained from the image. For example, if a pixel in the image is labeled as "car" and the pixel is mapped to a point in the point cloud, then the point cloud point is considered to be a point related to "car".

[0065] Through the above steps, a point cloud dataset with semantic information and the corresponding image data are obtained. This fused dataset contains the 3D coordinates of the point cloud and the object category probabilities from the image, which can be regarded as a form of semantic labels attached to the point cloud data. This fused dataset can provide more comprehensive input information for subsequent feature extraction and target detection, which helps to improve detection accuracy and robustness.

[0066] In some embodiments, the DeeplabV3+ algorithm is used to perform semantic segmentation on the scene image to obtain the segmented scene image, which can be specifically achieved through the following steps: constructing an image segmentation model, wherein the image segmentation model is trained by the DeeplabV3+ algorithm using multiple sets of training data, and each set of training data in the multiple sets of training data includes historical scene images acquired within a historical time period and historical scene images corresponding to the historical scene images after segmentation; inputting the scene image into the image segmentation model to obtain the segmented scene image corresponding to the scene image.

[0067] In this solution, the image segmentation model built and trained using the DeeplabV3+ algorithm can effectively learn and capture the edges and details of objects in the image, improve the accuracy and robustness of semantic segmentation, and enable the segmented scene image to more accurately represent the objects and their boundaries in the original image. The segmented scene image provides key semantic information for point cloud and image fusion, namely the category label of the pixel point where each point cloud point is located. This enables the point cloud data to obtain additional contextual information from the image, such as color, texture, and object type, further enhancing the descriptive ability of the point cloud features.

[0068] Specifically, the image segmentation model is built based on the DeeplabV3+ algorithm, which is an advanced semantic segmentation technology in the field of deep learning and is particularly good at handling dense pixel classification tasks. In order to train this model, multiple sets of historical training data are needed, each of which contains two parts: one is historical scene images, that is, environmental images acquired by image acquisition devices (such as RGB cameras) in different historical time periods; the other is historical scene images obtained by manual annotation or semi-automatic segmentation of these historical scene images, in which each pixel is assigned a specific category label.

[0069] By inputting historical scene images into the DeeplabV3+ model and comparing them with the corresponding historical segmented scene images, the model will learn how to identify and segment objects of different categories from the original images. The training process involves adjusting the parameters of the model to minimize the difference between the predicted segmentation and the true annotation, usually using the cross entropy loss function as the training target. The trained model can accurately semantically segment the newly input scene images and provide category information for subsequent processing.

[0070] Specifically, the data-level fusion module includes the input of the original point cloud and image, the semantic segmentation network, and the coordinate projection transformation. The point cloud and image data are both from the kitti dataset. The semantic segmentation network uses the DeepLabv3+ network. Structurally, the DeeplabV3+ network is mainly composed of an encoder and a decoder. The encoder extracts multi-scale features from the input image to obtain a rich feature representation. The decoder restores the encoder's feature map to the same size as the original image through upsampling and connection operations, and outputs the category probability of each pixel.

[0071] First, the image is input into the DeeplabV3+ network to obtain the category probability of each pixel. For the kitti dataset, the network output is the score of four categories, including cars, pedestrians, cyclists, and background. Then the corresponding 3D point cloud data is projected onto the 2D image plane, and the projection and calibration matrix are used to calculate each 3D point cloud P j (xj ,y j ,z j ) is the projection coordinate P' on the two-dimensional image plane j (x' j ,y' j );

[0072] P' j =R _rect_in ×R _rect ×T v_to_c ×P j ;

[0073] Among them, R _rect_in Represents the intrinsic parameter matrix of the camera, R _rect represents the calibration moment of the camera, T v_to_c Represents the transformation matrix from lidar to camera.

[0074] After projecting the lidar points into the image, the segmentation scores of the associated pixels (u, v) are appended to the lidar points to form the processed lidar point cloud.

[0075] In the specific implementation process, when the fusion method includes feature fusion, the scene image and the point cloud image are fused to obtain the target image, which can be achieved through the following steps: extracting the features of the scene image to obtain first feature data; extracting the features of the point cloud image to obtain second feature data; performing feature fusion based on the first feature data and the second feature data to obtain the target image.

[0076] In this solution, images provide rich colors, textures, and object details, while point clouds provide precise depth and geometry information. Feature fusion can combine the advantages of both to make up for the shortcomings of single modality data. For example, images may perform poorly in low-light conditions, while point clouds can provide stable location information under such conditions. In the case of occlusion, lighting changes, or other complex scenarios, feature fusion can extract more information from the two data sources, helping the model to better understand the environment and reduce false detections and missed detections. Especially for the detection of small or distant targets, the fused feature representation is more comprehensive and accurate. Feature fusion enables the model to learn complementary features from multiple sensor inputs, which helps the model to generalize better in different environments and conditions, reduce dependence on a single data source, and thus improve the adaptability and reliability of the overall system.

[0077] Specifically, first, the system extracts feature information from the scene image obtained by the RGB camera. This feature information can include visual features such as color, texture, and edges, as well as deeper semantic features such as object categories and spatial layout. The process of extracting features is usually completed through a deep convolutional neural network (CNN), such as using a network architecture such as ResNet, VGG, or Inception. The first feature data output is a high-dimensional, abstract feature representation that can capture the key information in the image.

[0078] Next, the system extracts features from the point cloud image acquired by the lidar. The point cloud image consists of a collection of three-dimensional points, each of which carries (x, y, z) coordinate information. Some systems can also include reflection intensity or other additional information. The extraction of point cloud features is usually done using point cloud processing networks such as PointNet, PointNet++, VoxelNet, or SPN. These networks are designed to process unordered point set data and can extract geometric features of point cloud data, such as shape, size, and layout. The second feature data is also a high-dimensional feature representation, but focuses on the three-dimensional structure and shape information of the point cloud data.

[0079] After acquiring the two feature representations, the system performs feature fusion, that is, combining the first feature data (features from the scene image) and the second feature data (features from the point cloud image) to produce a more comprehensive and comprehensive feature representation, namely the target image. There are many strategies for feature fusion, including feature concatenation, feature addition, feature multiplication, etc. The fused feature representation contains the visual information of the RGB image and the depth information of the point cloud, and can provide a more complete scene understanding. The target image is not an image in the traditional sense, but a fused feature representation, which can be a multi-dimensional feature representation of point cloud points with image features attached, or a comprehensive feature map after mapping image features and point cloud features to the same space.

[0080] Specifically, for the lidar point cloud feature extraction module, the lidar point cloud feature extraction module includes four set abstraction layers (SA) and four feature propagation layers (FP). The point cloud data is input into the SA layer for step-by-step downsampling to obtain more representative point cloud features at different scales, and then interpolation operation is performed through the FP layer to achieve point cloud feature upsampling and restore the point cloud features to the original scale.

[0081] Specifically, for the image feature extraction module, the image feature extraction module consists of five convolution blocks and four transposed convolution layers. The first four convolution blocks are used to extract image features at different resolutions and match them with point cloud features of the corresponding resolutions. Next, these image features of different resolutions are input into four transposed convolution layers for upsampling to restore them to the original resolution. Then, the restored image features are merged in series and passed to the fifth convolution block, and the final image features are obtained after processing. In addition, each of the first four convolution blocks contains a BN layer, a ReLU activation function, and two 3×3 convolution layers, where the stride of the second convolution layer is 2; the fifth convolution block includes a BN layer, a ReLU activation function, and a 1×1 convolution layer.

[0082] Specifically, the above embodiment is feature-level fusion. On the basis of data-level fusion, the solution further uses deep learning models to fuse point cloud features with image features at different levels of the network. This process interacts and combines the features of the two modalities through multiple fusion modules, such as the LI-Fusion module, evaluates the importance of image features to point cloud features, and realizes adaptive fusion of features. The fused feature vector contains the depth and position information of the point cloud, as well as the semantic features such as shape, texture and color provided by the image, forming a more comprehensive description of the target.

[0083] Specifically, features are extracted from the processed point cloud data and the original image data respectively. Point cloud features are extracted by SetAbstraction (SA) and Feature Propagation (FP) modules, while image features are extracted by a series of convolutional layers. The correspondence between point cloud features and image features at different scales is determined by coordinate mapping between point cloud and image. This correspondence enables point cloud features to be fused with image features at the same position. The importance of image features to point cloud features is evaluated using a LiDAR-guided fusion module. The two features are mapped to the same channel by a fully connected layer, and a weight map is generated, which is normalized to the interval [0,1] by a sigmoid activation function to represent the contribution of image features to point cloud features. Point cloud features and image features are fused according to the weight map. Image features with high weights are taken into account more and fused in series or additively with point cloud features to form a fused multimodal feature representation. The fused features are further passed through a feature propagation layer (FP) to achieve feature upsampling in the point cloud to restore the resolution of the original point cloud while retaining the fused information. Feature-level fusion can be repeated at multiple scales to ensure that the features of point cloud and image are fully integrated at different levels. Feature fusion can be performed after each SA and FP module, as well as after each image feature extraction module, to improve the accuracy of detection.

[0084] Specifically, after data-level fusion, both point clouds and images are processed through their own feature extraction networks. Specifically, point cloud data is downsampled through a series of Set Abstraction (SA) modules to form point features of different scales, and upsampled through Feature Propagation (FP) modules to restore features to higher resolutions. Image data is extracted through convolutional neural networks (such as multiple 2D convolution blocks) to extract image features of different resolutions.

[0085] Feature-level fusion first needs to establish the correspondence between point cloud features and image features. This step is achieved through a point-by-point correspondence generator, which uses the mapping matrix from point cloud to image to determine the position of each point cloud point on the image, thereby finding the image features of the corresponding pixels.

[0086] After establishing the correspondence, the feature fusion module evaluates the importance of the image features to the point cloud features. This is usually done by designing an adaptive weight mechanism, such as the lidar-guided fusion module, which uses lidar features as a guide to map the point cloud features and image features to the same channel through a fully connected layer, and then calculates a weight map indicating how much each point cloud feature should be fused with features from the image.

[0087] After obtaining the weight map, the point cloud features and image features are fused with the weights as guidance. The most common fusion operations are weighted addition or concatenation. Additive fusion means directly adding the weighted image features to the point cloud features to form a new feature representation that contains information from both. Concatenated fusion concatenates the point cloud features and image features according to the number of channels to form a wider feature vector, which preserves the original information of the two features while performing more complex feature interactions in the next layer of the model.

[0088] Feature-level fusion is performed at multiple scales to ensure that the fused features can cover objects of different sizes and distances. In this scheme, point cloud features are fused with corresponding image features at the output scales of different SA modules, which can capture both local details and overall structure of the object.

[0089] The fused features are upsampled again through the FP layer to restore to a higher resolution. In this way, the distribution of the fused features in the point cloud is closer to the distribution of the original point cloud data, which helps the model to perform more refined 3D object positioning and classification at high levels.

[0090] In some embodiments, feature fusion is performed on the first feature data and the second feature data to obtain the target image, which can be specifically achieved through the following steps: obtaining a weight map, wherein the weight map is an image of the importance of the first feature data in the target image; according to the weight map, feature fusion is performed on the first feature data and the second feature data to obtain the target image, wherein the fusion method of the first feature data and the second feature data includes addition and / or concatenation.

[0091] In this scheme, feature fusion guided by weight maps can dynamically adjust the contribution of visual features and point cloud features in the target detection task. In some scenarios, such as low-light environments, point cloud features may be more reliable than visual features, and weight maps can make the model more inclined to use point cloud features; in scenarios with sufficient lighting and rich color and texture information, visual features can dominate. The use of weight maps helps to optimize the allocation of computing resources, because the model can selectively perform deeper processing on certain areas or features according to the instructions of the weight map, avoiding wasting computing resources on irrelevant information. The addition and / or concatenation fusion method provides a simple and effective mechanism for integrating information from different modalities while maintaining the independence and complementarity of each feature, promoting the efficient integration and utilization of multimodal information by the model.

[0092] Specifically, a weight map is first generated through a certain mechanism. This weight map is actually a visual representation of the importance of the first feature data (i.e., the feature extracted from the scene image) in the fused target image. Each pixel or point in the weight map corresponds to the importance weight of the corresponding position in the scene image. These weight values ​​can reflect the relative importance of visual features relative to point cloud features in the target detection task. The generation of the weight map can rely on attention mechanisms, adaptive learning, or pre-set rules. Its purpose is to enable the model to dynamically adjust the contribution of each feature according to the specific situation to optimize the final fusion result.

[0093] Using the generated weight map, the first feature data (visual features) and the second feature data (point cloud features) are fused to generate a feature representation of the target image. The fusion process can be done by addition or series, or a combination of the two. Additive fusion refers to adding the first feature data and the second feature data according to the weight. This fusion method can adjust the strength of each feature based on the weight map to generate a weighted feature representation, so that the visual features dominate in some areas and the point cloud features are more prominent in other areas. Series fusion is to splice the first feature data and the second feature data together to form a longer feature vector or feature map. This method retains the independence of the two features while allowing the model to learn how to combine visual and point cloud information from the fused features.

[0094] Specifically, for the feature fusion module, the feature fusion module can be divided into two parts, the point-by-point correspondence generator and the lidar-guided fusion module. The point-by-point correspondence generator takes the lidar point cloud and the mapping matrix M as input, and outputs the correspondence between the lidar points and the camera images at different resolutions. Specifically, for a point P (x, y, z) in the point cloud, its corresponding position P′ (x′, y′) in the camera image can be obtained through projection transformation, and the formula is as follows:

[0095] P′=M×P;

[0096] After the corresponding relationship is established, the image sampler is used to obtain the semantic feature representation corresponding to each point. Specifically, the image sampler uses the sampling position P′ and the image feature map F as input to generate a point-by-point image feature representation V for each sampling position. Considering that the sampling position can fall between adjacent pixels, bilinear interpolation is used to obtain the image features under continuous coordinates, and the formula is as follows:

[0097] V(p) = K(F(N(p′)));

[0098] Where V(p) is the image feature corresponding to point P, K represents the bilinear interpolation function, and F(N(p′)) represents the image feature of the adjacent pixels at the sampling position p′.

[0099] The LiDAR-guided fusion module uses LiDAR features to adaptively evaluate the importance of image features in a point-by-point manner. First, the point cloud feature F P and image features F I The inputs are fed into a fully connected layer and mapped to the same channel. They are then added together to form a compact feature representation, and fed into another fully connected layer to compress them into a weight map w. The weight map w is normalized to the range of [0, 1] using the sigmoid activation function.

[0100] w=σ(Wtanh(UF P +VF I ));

[0101] Where W, U, V represent the learnable weight matrix, and σ represents the sigmoid activation function.

[0102] After obtaining the weight map w, the point cloud feature F P and image features F I Combined in series, it can be formulated as:

[0103] F LI =F p ||wF I ;

[0104] where F LI Represents the features after the fusion of point cloud and image.

[0105] In a specific implementation process, target recognition is performed on the target image to obtain the target object in the target image, which can be achieved through the following steps: obtaining multiple initial bounding boxes, wherein the initial bounding boxes are rectangular boxes that surround at least the same object in the target image; performing NMS processing on all the initial bounding boxes to obtain scores of the multiple initial bounding boxes; determining the initial bounding box with the highest score among the multiple initial bounding boxes as the target bounding box; and determining that the object surrounded by the target bounding box is the target object.

[0106] In this solution, NMS removes overlapping bounding boxes to ensure that each detected object corresponds to only one bounding box, significantly improving the accuracy of the detection results and avoiding multiple false positives of the same object. By selecting the bounding box with the highest score, NMS ensures that the model accurately locates the target object, especially in complex scenes where the target object is partially occluded or there are multiple similar objects, and can more accurately identify and locate the target.

[0107] Specifically, during 3D object detection, the model predicts multiple candidate initial bounding boxes, which can surround the same object in the target image or cover different objects. Each initial bounding box represents the model's prediction of a potential target, including the location, size, and confidence score of the bounding box.

[0108] NMS is a post-processing technique used to filter out overlapping bounding boxes and retain only the bounding boxes that are most likely to represent the real object. In this process, all initial bounding boxes are input into the NMS algorithm, and the algorithm first calculates the score of each bounding box. The score of the bounding box is usually the confidence given by the model when predicting, reflecting the probability that the bounding box encloses the real object.

[0109] NMS compares the scores of all initial bounding boxes and selects the bounding box with the highest score as the target bounding box. If two or more bounding boxes overlap and enclose the same object, NMS will filter them based on their scores and only keep the bounding box with the highest score, thus avoiding duplicate detection results. This step ensures the uniqueness and accuracy of the final detection results.

[0110] Once the target bounding box is determined, the object enclosed by the bounding box is considered to be a real target in the target image. The coordinates, size, and category label of the target bounding box will be used in the final detection output, indicating the location, size, and type of the target object in the image.

[0111] In the field of computer vision and image processing, a bounding box is a rectangular box that surrounds the target object in an image. It is used to locate and identify objects in images or videos. A bounding box is usually defined by four parameters: the coordinates of the center point (x, y), width (width) and height (height), or the coordinates of the upper left corner and lower right corner. In 3D object detection, the bounding box is extended to a bounding volume in three-dimensional space, usually a cuboid, which is defined by the three-dimensional coordinates of the center point (x, y, z), length, width, height, and rotation angle.

[0112] Specifically, for the object detection module, the object detection module uses a bin-based 3D bounding box generation method to generate corresponding 3D bounding boxes and their scores for the foreground points. The generated bounding boxes are processed by non-maximum suppression (NMS) to remove low-scoring bounding boxes, while retaining the highest-scoring bounding boxes as the final detection results.

[0113] Specifically, the bin-based 3D bounding box generation operation steps are as follows: First, use one-dimensional convolution to process the multimodal fusion features to generate the corresponding point cloud classification score. Points with a classification score greater than 0.2 are considered foreground points, and those below this value are considered background points. Next, based on the foreground points, the bin method is used to generate 3D bounding boxes and their scores, and the non-maximum suppression (NMS) processing is used to remove low-scoring bounding boxes, and finally the bounding box with the highest score is retained as the detection result.

[0114] Specifically, in detail, point cloud data is combined with image data through data-level fusion to obtain point cloud data carrying image semantic information. Then, in the feature-level fusion stage, the lidar point cloud feature extraction module and the image feature extraction module are used to extract the features of the point cloud and image respectively, and then the two are fused at multiple scales through the feature fusion module to generate multimodal fusion features containing point cloud geometric information and image semantic information.

[0115] Specifically, in detail, the fused features will be upsampled and refined through the feature propagation layer (FP) to restore the features to the resolution of the original point cloud. This process helps maintain the density of the point cloud while ensuring that the fused features can cover all point cloud points, providing more detailed information for subsequent detection.

[0116] Specifically, in detail, the fused features are passed to the object detection module, which uses a bin-based 3D bounding box generation method to classify and regress each point in the point cloud. For point clouds classified as foreground points (i.e. points predicted to be part of an object), the module predicts the corresponding 3D bounding box parameters, such as center position, size, rotation angle, etc., as well as the confidence score of the bounding box (indicating the probability that the predicted box contains an object).

[0117] Based on the fused features, the model assigns a category label to each point cloud point, such as car, pedestrian, or bicycle, and gives the confidence of the classification. The model predicts the 3D bounding box parameters of each object by predicting the offset of the center point relative to the predefined grid as well as the object size and rotation angle.

[0118] Specifically, in detail, after the model generates multiple 3D bounding box predictions, the non-maximum suppression (NMS) algorithm is used to remove overlapping, low-confidence bounding boxes. The NMS algorithm sorts the bounding boxes according to their confidence scores, then selects the highest confidence box and calculates its intersection over union (IoU) with other boxes. If the IoU exceeds a predetermined threshold, the other boxes are suppressed, thereby retaining the most accurate predicted box.

[0119] Finally, the 3D bounding box processed by NMS is used as the output of target detection, which contains the target category, position, size and orientation information in 3D space, and the corresponding confidence score.

[0120] The specific process of non-maximum suppression NMS is as follows: First, all generated bounding boxes are sorted according to the score, and the bounding box with the highest score is selected. Then, a predefined threshold is used to suppress other bounding boxes that have significant overlap with the bounding box with the highest score. This process is applied to the remaining bounding boxes in turn, and the intersection-over-union ratio (IoU) of each candidate bounding box with the highest scoring box is calculated. IoU represents the ratio of the intersection and union between the predicted box and the true bounding box.

[0121] The calculation formula of the NMS algorithm is as follows:

[0122]

[0123] Among them, S i is the bounding box score after comparison and calculation, s i is the initial score of the bounding box, M is the bounding box with the highest score, and b i is the current bounding box, IoU(M,b i ) is M and b i The intersection-over-union ratio, N t is the threshold value.

[0124] Specifically, object detection refers to the process of identifying and locating specific targets (such as cars, pedestrians, cyclists, etc.) from three-dimensional point cloud data and two-dimensional image data. Specifically, it is achieved by fusing the features of point cloud and image and using deep learning network to identify and locate 3D targets.

[0125] The point cloud data and image data are preprocessed separately, including the coordinate transformation of the point cloud and the semantic segmentation of the image, in preparation for the subsequent feature extraction. The semantic segmentation information of the image is attached to the point cloud through coordinate mapping to achieve semantic enrichment of the point cloud data. This step maps the classification score of each pixel on the image to the corresponding point cloud point, so that the point cloud point carries additional category information. A series of Set Abstraction (SA) modules and Feature Propagation (FP) modules are used to extract features of different scales from the point cloud data. Multi-scale features of the image are extracted through convolutional neural networks (such as DeeplabV3+).

[0126] After the feature extraction stage, the point cloud features are fused with the image features through the LI-Fusion module. This includes using a point-by-point correspondence generator to determine the correspondence between the point cloud and image features, and using a lidar-guided fusion module to evaluate and fuse the contribution of image features to the point cloud features.

[0127] The fused features are input into the target detection module, which uses a bin-based 3D bounding box generation method to generate corresponding 3D bounding boxes and their scores for the foreground points. The bin-based 3D bounding box generation method consists of two parts: predicting the category of the bounding box and predicting the size and direction of the bounding box. The network predicts a score for each point, indicating the possibility that the point is an object, as well as the location and size of a 3D bounding box. Non-maximum suppression (NMS): The generated bounding boxes are processed by the NMS algorithm to remove overlapping bounding boxes and retain the most likely detection results to reduce false positives and improve detection accuracy.

[0128] In some embodiments, after determining that the initial bounding box with the highest score among the multiple initial bounding boxes is the target bounding box, the method further includes the following steps: obtaining a loss function; calculating a loss value of the target bounding box according to the loss function; and adjusting the target bounding box according to a back propagation algorithm until the loss value is less than a preset loss threshold, thereby obtaining an adjusted target bounding box.

[0129] In this scheme, through the guidance of the loss function and the optimization of the back-propagation algorithm, the model can continuously adjust the position and size of the predicted target bounding box until the gap between the predicted result and the actual result is minimized, thereby significantly improving the accuracy of bounding box positioning.

[0130] Specifically, the loss function is a function in machine learning and deep learning that measures the difference between the model's predicted results and the actual results. In object detection tasks, multi-objective loss functions, including classification loss and regression loss, are usually used to evaluate the gap between the model's predicted bounding box and the true bounding box. The design of the loss function should be able to fully reflect the performance of the model in positioning and classification tasks. Common loss functions include cross entropy loss (for classification tasks) and smooth L1 loss (for regression tasks, especially in bounding box coordinate prediction).

[0131] During the training phase, the model predicts the target bounding box. To evaluate the accuracy of the prediction, the predicted bounding box needs to be compared with the actual, annotated bounding box. This is usually achieved by inputting the predicted value of the target bounding box (including position, size, and category label) and the true value into the loss function defined earlier. The loss function calculates a number, the loss value, which reflects the difference between the predicted bounding box and the actual bounding box. The smaller the loss value, the closer the model's prediction is to the actual annotated bounding box.

[0132] The backpropagation algorithm is a commonly used optimization method in deep learning model training. It updates the model parameters to reduce the loss value by calculating the gradient of the loss function relative to the model parameters. In this process, the model will continuously adjust parameters such as weights and biases, as well as the predicted value of the target bounding box, to make the model's prediction closer to the actual value. The training will be iterated until the loss value is lower than the preset loss threshold. The preset loss threshold is a set loss value standard that considers the model training to have converged sufficiently. Once the model training reaches the convergence condition, that is, the loss value is less than the preset threshold, the training process will stop, and the model can predict the target bounding box more accurately.

[0133] Specifically, the total loss of the network includes the consistency enforcement loss L ce , classification loss L cls and the regression loss L reg The three loss functions are:

[0134]

[0135] L cls = -α(1-c t ) γ logc t

[0136]

[0137] In the consistency enforcement loss, D represents the predicted value of the bounding box, G represents the true value of the bounding box, and c represents the classification confidence of D. The regression loss function uses a bin-based method to first predict which bin the center point falls in, and then regress the remaining offset within the bin. α represents the adjustment factor of the positive sample, and c t represents the model's predicted probability for each category, γ represents the parameter that controls the sample weight distribution, which is usually set to 2, where E and S represent the cross entropy loss and smooth L1 loss, respectively, and b u and Represents the true value and predicted value of the bounding box, r u and represents the true value and predicted value of the residual, h, w, l, θ represent the height, length, width and angle of the bounding box respectively.

[0138] The consistency enforcement loss is to ensure that there is consistency between the classification confidence of the point cloud and the confidence of the 3D bounding box during the model's prediction process. In 3D object detection, the confidence calculation is usually based on the accuracy of the object classification and the precision of the bounding box regression. If a point is classified as a car, but the confidence of its corresponding bounding box is very low, then the prediction may not be reliable enough. The consistency enforcement loss encourages the model to output confidence scores that are logically consistent by comparing the predicted classification confidence and the bounding box confidence. That is, if the model's classification confidence for a point is high, then the confidence of its corresponding 3D bounding box should also be high.

[0139] The purpose of consistency enforcement loss is to enhance the consistency between point cloud detection and image detection, so that when the model predicts the target bounding box, its classification confidence (i.e., the degree of certainty of the model about the target category) and the confidence of the bounding box (i.e., the accuracy of the model about the position of the bounding box) are logically consistent. This loss function can help the model avoid the situation where the classification is accurate but the positioning is inaccurate, or the positioning is accurate but the classification is wrong, thereby improving the overall performance of the entire detection framework.

[0140] During training, the consistency enforcement loss guides the model to learn more consistent detection results by calculating the difference between the model's classification and localization confidence for the same object. For example, if the model's classification confidence for a point is higher than the threshold, but its corresponding 3D bounding box confidence is lower, the consistency enforcement loss will generate a larger error signal, prompting the model to adjust parameters to improve positioning accuracy.

[0141] Classification loss is used to train the model to recognize and classify different objects in 3D space. It measures the difference between the class labels predicted by the model and the actual labels. During training, the model tries to minimize the classification loss to improve the accuracy of the prediction of the target class. Classification loss is usually calculated using the cross entropy loss function, which can effectively handle multi-classification problems and encourage the model to predict the probability of each class closer to the one-hot encoding of the actual class.

[0142] The goal of classification loss is to accurately identify the category of the target in 3D space. It measures the difference between the model's predicted category label and the actual category label, ensuring that the model can distinguish different categories of targets such as cars, pedestrians, bicycles, etc. from the fused features. This loss function is usually based on cross entropy, which can encourage the model to learn the distinguishing features of each category and provide highly accurate classification predictions.

[0143] Regression loss is used to optimize the model's prediction of 3D bounding box parameters. It measures the difference between the bounding box parameters predicted by the model (such as center point position, size, rotation angle, etc.) and the actual bounding box parameters. Regression loss forces the model's predicted bounding box to be closer to the actual target boundary during training. In order to handle the wide range of possible changes in bounding box parameters, regression loss usually uses Smooth L1 loss or IoU loss, which can provide more stable gradients when the parameters change greatly, helping the network converge.

[0144] The regression loss is responsible for optimizing the model's prediction of the 3D bounding box parameters, including the position, size, and rotation angle of the bounding box. It ensures that the model can not only correctly identify the target, but also accurately locate and describe the size and direction of the target in 3D space. Regression loss usually uses Smooth L1 loss or IoU loss. These loss functions can provide stable gradients when processing the geometric parameters of the bounding box, helping the model to effectively learn and adjust parameters.

[0145] By minimizing the total loss function (a weighted combination of the above three losses), the model will continuously adjust its weights and parameters during the training process to make the predicted category labels, bounding box sizes and positions as close to the true values ​​as possible. This optimization process will promote the model to more effectively utilize the information of the two modalities when processing the fusion of point cloud and image data, overcome the limitations of a single modality, and thus achieve higher accuracy and robustness in 3D object detection tasks.

[0146] In summary, in the scheme of the present application, the image data and the point cloud data each carry complementary information. The image data contains rich colors, textures and visual features, which are conducive to object classification and detail recognition; the point cloud data provides depth and position information in three-dimensional space, which is conducive to spatial positioning and size estimation. First, the semantic information of the image (such as the category probability of each pixel obtained by semantic segmentation) is attached to the corresponding point cloud data through coordinate transformation and mapping. This step ensures that the point cloud data carries the semantic information provided by the image and enhances the feature description ability of the point cloud points. On the basis of data-level fusion, the feature fusion module is used to further combine the features of the point cloud and the image. This includes a point-by-point correspondence generator and a lidar-guided fusion module, which aims to evaluate the contribution of image features to point cloud features and generate a richer fusion feature representation. Through weight adjustment and feature concatenation, the fused features can better reflect the shape, position and category of the target. The network performs multi-level feature extraction on the point cloud data and image data respectively, including the set abstraction layer and feature propagation layer of the point cloud, and the convolution block and transposed convolution layer of the image. This enables the network to capture information of different scales and improve the detection accuracy of the target. By fusing the features of point clouds and images at multiple levels, the network can learn more comprehensive and in-depth target information from the features of the two data sources. When image information is limited (such as insufficient lighting or occlusion), the depth information of the point cloud can be supplemented; when point cloud data is sparse, the semantic information of the image can provide additional classification basis. The object detection module uses a bin-based method to generate 3D bounding boxes, accurately estimating the 3D size and position of the object by predicting the position and size of the object in different bins and regressing the residual offset within each bin. The NMS algorithm is used in the detection results to remove duplicate bounding boxes and low-quality predictions, retaining the detection results with the highest confidence, thereby reducing false positives and improving the accuracy and reliability of detection. Through consistency enforcement loss, the network is trained to ensure the consistency of point cloud and image features in evaluating target categories and positions, which helps to improve the overall performance of the model.

[0147] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the multimodal target detection method of the present application will be described in detail below in combination with specific embodiments.

[0148] This embodiment relates to a specific multimodal target detection method. The network uses images and point clouds as inputs. In order to avoid reducing the limitations of a single modality and make full use of the features of point clouds and images, an innovative detection framework of data set fusion and feature-level fusion is used simultaneously. Data set fusion is to attach the semantic segmentation information of the image to the point cloud through coordinate mapping to achieve the fusion of the two at the data level. Feature-level fusion is to effectively fuse the features of point clouds and images in the process of extracting their features to achieve better detection results. Figure 4 As shown, the following steps are included:

[0149] Get the original point cloud data collected by the laser radar and the original image data collected by the camera, where the original point cloud data includes spatial coordinate information (x, y, z) and the original image data includes RGB information. The data set used in this solution comes from the kitti data set.

[0150] The image and point cloud data are input into the multi-stage, multi-branch fusion network. Inside the network, the image data is first input into the advanced DeeplabV3+ segmentation network to obtain the classification probability (or classification score) of each pixel.

[0151] According to the intrinsic parameters of the camera and the extrinsic parameters of the camera and lidar, the point cloud data is projected onto the corresponding image, and the classification probability of the pixel points is attached to the corresponding point cloud to obtain the processed point cloud data, thereby realizing data-level fusion of image and point cloud data.

[0152] The processed point cloud data is input into the feature-level fusion stage of the network together with the image data. This stage consists of a two-stream RPN and a fusion module. The two-stream RPN includes a point cloud branch and an image branch. The point cloud branch takes the point cloud as input and gradually extracts the point cloud features through the four-pair set abstraction layer (SA) and the feature propagation layer (FP); the image branch takes the image as input and uses a series of convolution operations to extract semantic image information. The feature fusion module first establishes the correspondence between point clouds of different scales and image feature maps, and uses the lidar features to adaptively evaluate the importance of image features in a point-by-point manner. After obtaining the corresponding weight map w, the point cloud feature F is used. P and image features F I They are combined in series to achieve feature-level fusion of point cloud and image.

[0153] The object detection module uses a bin-based 3D bounding box generation method to generate corresponding 3D bounding boxes and their scores for foreground points. The generated bounding boxes are processed by non-maximum suppression (NMS) to remove low-scoring bounding boxes while retaining the highest-scoring bounding boxes as the final detection results.

[0154] The embodiment of the present application also provides a multimodal target detection device. It should be noted that the multimodal target detection device of the embodiment of the present application can be used to execute the multimodal target detection method provided by the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions have been made and will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0155] The following is an introduction to the multimodal target detection device provided in the embodiments of the present application.

[0156] Figure 5 is a structural block diagram of a multi-modal target detection device according to an embodiment of the present application. Figure 5 As shown, the device comprises:

[0157] A first acquisition unit 10 is used to acquire a scene image, wherein the scene image is an image of the current environment acquired by an image acquisition device;

[0158] A second acquisition unit 20 is used to acquire a point cloud image, wherein the point cloud image is an image of point cloud data of the current environment acquired by a laser radar;

[0159] A fusion unit 30, configured to fuse the scene image and the point cloud image to obtain a target image, wherein the fusion method includes at least image merging and / or feature fusion;

[0160] The recognition unit 40 is used to perform target recognition on the target image to obtain the target object in the target image.

[0161] Through this embodiment, the scene image collected by the image acquisition device and the point cloud image collected by the lidar are used as input for multimodal fusion, which can fully utilize the features of the point cloud and the image, and can effectively reduce the limitations of a single modality, so that this solution can achieve better target detection effects.

[0162] In the specific implementation process, when the fusion method includes image merging, the fusion unit includes a segmentation module, a projection module and an adding module. The segmentation module is used to use the DeeplabV3+ algorithm to perform semantic segmentation on the above-mentioned scene image to obtain a segmented scene image, wherein the above-mentioned segmented scene image includes the category of each pixel point; the projection module is used to project the points of the above-mentioned point cloud image into the above-mentioned scene image to obtain the correspondence between the points of the above-mentioned point cloud image and the pixels of the above-mentioned scene image; the adding module is used to add the above-mentioned categories of all the pixels in the above-mentioned segmented scene image to the above-mentioned point cloud image according to the above-mentioned correspondence to obtain the above-mentioned target image.

[0163] In this solution, by attaching semantic category information to point cloud data, targets can be more accurately identified and classified, especially when dealing with small or distant targets. The color and texture information provided by the image can help the model distinguish and confirm the object category, thereby improving the accuracy of detection. Point cloud data itself may lack sufficient semantic features. Through data-level fusion, point cloud data obtains additional semantic information, which makes point cloud features richer and helps the model learn and understand target objects in complex scenes.

[0164] In some embodiments, the segmentation module includes a construction submodule and a processing submodule. The construction submodule is used to construct an image segmentation model, wherein the image segmentation model is trained using a DeeplabV3+ algorithm using multiple sets of training data, and each set of training data in the multiple sets of training data includes historical scene images acquired within a historical time period and historical segmented scene images corresponding to the historical scene images; the processing submodule is used to input the scene images into the image segmentation model to obtain the segmented scene images corresponding to the scene images.

[0165] In this solution, the image segmentation model built and trained using the DeeplabV3+ algorithm can effectively learn and capture the edges and details of objects in the image, improve the accuracy and robustness of semantic segmentation, and enable the segmented scene image to more accurately represent the objects and their boundaries in the original image. The segmented scene image provides key semantic information for point cloud and image fusion, namely the category label of the pixel point where each point cloud point is located. This enables the point cloud data to obtain additional contextual information from the image, such as color, texture, and object type, further enhancing the descriptive ability of the point cloud features.

[0166] In the specific implementation process, when the fusion method includes feature fusion, the fusion unit includes a first extraction module, a second extraction module and a fusion module. The first extraction module is used to extract the features of the above-mentioned scene image to obtain first feature data; the second extraction module is used to extract the features of the above-mentioned point cloud image to obtain second feature data; the fusion module is used to perform feature fusion according to the above-mentioned first feature data and the above-mentioned second feature data to obtain the above-mentioned target image.

[0167] In this solution, images provide rich colors, textures, and object details, while point clouds provide precise depth and geometry information. Feature fusion can combine the advantages of both to make up for the shortcomings of single modality data. For example, images may perform poorly in low-light conditions, while point clouds can provide stable location information under such conditions. In the case of occlusion, lighting changes, or other complex scenarios, feature fusion can extract more information from the two data sources, helping the model to better understand the environment and reduce false detections and missed detections. Especially for the detection of small or distant targets, the fused feature representation is more comprehensive and accurate. Feature fusion enables the model to learn complementary features from multiple sensor inputs, which helps the model to generalize better in different environments and conditions, reduce dependence on a single data source, and thus improve the adaptability and reliability of the overall system.

[0168] In some embodiments, the fusion module includes an acquisition submodule and a fusion submodule, the acquisition submodule is used to acquire a weight map, wherein the weight map is an image of the importance of the first feature data in the target image; the fusion submodule is used to perform feature fusion on the first feature data and the second feature data according to the weight map to obtain the target image, wherein the fusion method of the first feature data and the second feature data includes addition and / or concatenation.

[0169] In this scheme, feature fusion guided by weight maps can dynamically adjust the contribution of visual features and point cloud features in the target detection task. In some scenarios, such as low-light environments, point cloud features may be more reliable than visual features, and weight maps can make the model more inclined to use point cloud features; in scenarios with sufficient lighting and rich color and texture information, visual features can dominate. The use of weight maps helps to optimize the allocation of computing resources, because the model can selectively perform deeper processing on certain areas or features according to the instructions of the weight map, avoiding wasting computing resources on irrelevant information. The addition and / or concatenation fusion method provides a simple and effective mechanism for integrating information from different modalities while maintaining the independence and complementarity of each feature, promoting the efficient integration and utilization of multimodal information by the model.

[0170] In the specific implementation process, the recognition unit includes an acquisition module, a processing module, a first determination module and a second determination module. The acquisition module is used to acquire multiple initial bounding boxes, wherein the above-mentioned initial bounding boxes are rectangular boxes that at least surround the same object in the above-mentioned target image; the processing module is used to perform NMS processing on all of the above-mentioned initial bounding boxes to obtain scores of the multiple initial bounding boxes; the first determination module is used to determine that the above-mentioned initial bounding box with the highest score among the multiple initial bounding boxes is the target bounding box; the second determination module is used to determine that the object surrounded by the above-mentioned target bounding box is the above-mentioned target object.

[0171] In this solution, NMS removes overlapping bounding boxes to ensure that each detected object corresponds to only one bounding box, significantly improving the accuracy of the detection results and avoiding multiple false positives of the same object. By selecting the bounding box with the highest score, NMS ensures that the model accurately locates the target object, especially in complex scenes where the target object is partially occluded or there are multiple similar objects, and can more accurately identify and locate the target.

[0172] In some embodiments, the device further includes a third acquisition unit, a calculation unit and an adjustment unit, wherein the third acquisition unit is used to acquire a loss function after determining that the initial bounding box with the highest score among the multiple initial bounding boxes is the target bounding box; the calculation unit is used to calculate the loss value of the target bounding box according to the loss function; and the adjustment unit is used to adjust the target bounding box according to a back propagation algorithm until the loss value is less than a preset loss threshold, thereby obtaining an adjusted target bounding box.

[0173] In this scheme, through the guidance of the loss function and the optimization of the back-propagation algorithm, the model can continuously adjust the position and size of the predicted target bounding box until the gap between the predicted result and the actual result is minimized, thereby significantly improving the accuracy of bounding box positioning.

[0174] The multimodal target detection device includes a processor and a memory. The first acquisition unit, the second acquisition unit, the fusion unit and the recognition unit are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0175] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of low accuracy of target detection based on single data in the prior art can be solved by adjusting kernel parameters.

[0176] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0177] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the multimodal target detection method.

[0178] An embodiment of the present invention provides a processor, which is used to run a program, wherein the multi-modal target detection method is executed when the program is run.

[0179] An embodiment of the present invention provides a device, the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements at least a multi-modal target detection method step when executing the program. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0180] A computer program product includes a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multimodal target detection method in each embodiment of the present application are implemented.

[0181] The present application also provides a multimodal recognition system, comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any of the multimodal target detection methods.

[0182] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0183] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0184] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0185] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0186] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0187] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0188] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0189] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0190] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0191] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multimodal target detection method, characterized in that: include: Acquire a scene image, wherein the scene image is an image of the current environment acquired by an image acquisition device; Acquire a point cloud image, wherein the point cloud image is an image of point cloud data of the current environment acquired by using a laser radar; Fusing the scene image and the point cloud image to obtain a target image, wherein the fusion method includes at least image merging and / or feature fusion; Perform target recognition on the target image to obtain the target object in the target image.

2. The method according to claim 1, characterized in that In the case where the fusion method includes image merging, fusing the scene image and the point cloud image to obtain a target image includes: Using the DeeplabV3+ algorithm, the scene image is semantically segmented to obtain a segmented scene image, wherein the segmented scene image includes the category of each pixel point; Projecting the points of the point cloud image into the scene image to obtain a correspondence between the points of the point cloud image and the pixel points of the scene image; According to the corresponding relationship, the categories of all pixels in the segmented scene image are added to the point cloud image to obtain the target image.

3. The method according to claim 2, characterized in that The DeeplabV3+ algorithm is used to perform semantic segmentation on the scene image to obtain a segmented scene image, including: Constructing an image segmentation model, wherein the image segmentation model is trained by DeeplabV3+ algorithm using multiple sets of training data, each set of training data in the multiple sets of training data includes historical scene images acquired within a historical time period and historical scene images corresponding to the historical scene images after segmentation; The scene image is input into the image segmentation model to obtain the segmented scene image corresponding to the scene image.

4. The method according to claim 1, characterized in that When the fusion mode includes feature fusion, fusing the scene image and the point cloud image to obtain a target image includes: Extracting features of the scene image to obtain first feature data; Extracting features of the point cloud image to obtain second feature data; Feature fusion is performed according to the first feature data and the second feature data to obtain the target image.

5. The method according to claim 4, characterized in that Performing feature fusion according to the first feature data and the second feature data to obtain the target image includes: Obtaining a weight map, wherein the weight map is an image of the importance of the first feature data in the target image; According to the weight map, feature fusion is performed on the first feature data and the second feature data to obtain the target image, wherein the fusion method of the first feature data and the second feature data includes addition and / or concatenation.

6. The method according to claim 1, characterized in that Performing target recognition on the target image to obtain a target object in the target image includes: Acquire multiple initial bounding boxes, wherein the initial bounding boxes are rectangular boxes that enclose at least the same object in the target image; Performing NMS processing on all the initial bounding boxes to obtain scores of the multiple initial bounding boxes; Determine the initial bounding box with the highest score among the multiple initial bounding boxes as the target bounding box; An object enclosed by the target bounding box is determined to be the target object.

7. The method according to claim 6, characterized in that After determining that the initial bounding box with the highest score among the multiple initial bounding boxes is the target bounding box, the method further includes: Get the loss function; Calculating a loss value of the target bounding box according to the loss function; According to the back propagation algorithm, the target bounding box is adjusted until the loss value is less than a preset loss threshold, thereby obtaining an adjusted target bounding box.

8. A multimodal target detection device, characterized in that: include: A first acquisition unit, configured to acquire a scene image, wherein the scene image is an image of a current environment acquired by an image acquisition device; A second acquisition unit is used to acquire a point cloud image, wherein the point cloud image is an image of point cloud data of the current environment acquired by using a laser radar; A fusion unit, configured to fuse the scene image and the point cloud image to obtain a target image, wherein the fusion method includes at least image merging and / or feature fusion; The recognition unit is used to perform target recognition on the target image to obtain the target object in the target image.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the multimodal target detection method described in any one of claims 1 to 7 are implemented.

10. A multimodal recognition system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the multimodal target detection method described in any one of claims 1 to 7.

Images