Target detection method and device based on fusion mode, computer equipment and medium

Abstract

The embodiment of the invention provides a target detection method and device based on a fusion mode, computer equipment and a medium, and relates to the technical field of automatic driving, and the method comprises the steps: obtaining self-collected visual data and first point cloud data; extracting second point cloud data corresponding to the category in the first point cloud data in the open source data set; integrating the first point cloud data and the second point cloud data to obtain third point cloud data; performing feature extraction on the visual data and the third point cloud data to obtain visual modal BEV features and point cloud modal BEV features; obtaining a modal offset matrix based on the visual modal BEV features and the point cloud modal BEV features; performing feature fusion on the visual modal BEV features and the point cloud modal BEV features based on a modal offset matrix to obtain fusion features; and performing target detection based on the fusion features. According to the invention, the problem of sample imbalance of the self-collected data set is solved, and the robustness and precision of the target detection method are improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a target detection method, device, computer equipment and medium based on fusion modality. Background Art

[0002] Autonomous vehicles refer to motor vehicles that mainly rely on artificial intelligence, visual computing, radar, GPS positioning, and structured road vehicle-road collaboration technologies to enable the car to have the ability of environmental perception, path planning and autonomous control, so that the embedded edge computing terminal can automatically operate. Since 2020, the concept of L2 and L3 autonomous driving has begun to be popularized in the industry. 2021 is a very important node in the development of autonomous driving technology. Many industry insiders regard 2021 as the first year for autonomous driving technology to enter the L3 level. Different from traditional driving vehicles, the core feature of autonomous driving vehicles is the application and dominance of AI technology. Its driving process is also achieved by the computer through the uninterrupted collection of various types of information during driving, and the information analysis and self-learning through machine learning or deep learning, and then the vehicle is controlled, so as to achieve the system engineering of computer autonomous driving vehicles. For autonomous driving vehicles, what they pursue is safe driving under any road conditions and any environment and timely response to extreme situations to ensure the life safety of drivers, vehicle safety and the safety of road participants.

[0003] At present, the field of autonomous driving is an important field for the deployment and implementation of deep learning algorithms. Autonomous driving algorithms are also in the process of rapid update and iteration. Various novel algorithms are constantly proposed in the fields of target detection, lane line recognition, target tracking, semantic segmentation, dynamic target trajectory prediction, multi-task learning, etc. The core of the autonomous driving algorithm technology framework is divided into three parts: environmental perception, decision planning, and control execution. The environmental perception part converts sensor data into machine language of the scene where the vehicle is located, which can specifically include: object detection, recognition and tracking, environmental modeling, motion estimation, etc. In recent years, as high-level assisted driving solutions are increasingly accepted by the market, the requirements for autonomous driving perception algorithms are also getting higher and higher. BEV (Bird Eye View) perception has received widespread attention. The BEV perception paradigm mainly provides a unified space to facilitate the integration of various tasks and sensors. The BEV perception system can convert the information obtained by multiple cameras or radars into a bird's-eye view, and then perform tasks such as target detection and instance segmentation, which can more intuitively display the size and direction of objects in the BEV space. The accuracy of the 3D detection model (BEV model) currently proposed by the academic community needs to be further improved. The accuracy of the model's detection is not high for small targets and few sample targets. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a target detection method based on fusion modality to solve the technical problems of data set sample imbalance and alignment between visual modality and point cloud modality in the prior art. The method includes:

[0005] Obtain self-collected visual data and the first point cloud data;

[0006] Extracting second point cloud data corresponding to the category in the first point cloud data from the open source data set;

[0007] Integrate the first point cloud data with the second point cloud data to obtain third point cloud data;

[0008] Performing feature extraction on the visual data and the third point cloud data respectively to obtain visual modality BEV features and point cloud modality BEV features;

[0009] Obtaining a modal offset matrix based on the visual modality BEV features and the point cloud modality BEV features;

[0010] Based on the modality offset matrix, the visual modality BEV feature and the point cloud modality BEV feature are subjected to feature fusion to obtain a fusion feature;

[0011] Target detection is performed based on the fused features.

[0012] The embodiment of the present invention also provides a target detection device based on fusion modality to solve the technical problems of imbalanced data set samples and alignment between visual modality and point cloud modality in the prior art. The device includes:

[0013] A data acquisition module, used to acquire self-collected visual data and first point cloud data;

[0014] A data extraction module, configured to extract second point cloud data corresponding to a category in the first point cloud data from an open source data set;

[0015] An integration module, used for integrating the first point cloud data with the second point cloud data to obtain third point cloud data;

[0016] A feature extraction module, used for performing feature extraction on the visual data and the third point cloud data respectively to obtain visual modality BEV features and point cloud modality BEV features;

[0017] A modal offset matrix acquisition module, used to obtain a modal offset matrix based on the visual modal BEV feature and the point cloud modal BEV feature;

[0018] A feature fusion module, used for fusing the visual modality BEV feature and the point cloud modality BEV feature based on the modality offset matrix to obtain a fused feature;

[0019] The target detection module is used to perform target detection based on the fusion features.

[0020] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements any of the above-mentioned target detection methods based on fusion modality to solve the technical problems in the prior art of imbalanced data set samples and alignment of visual modality and point cloud modality.

[0021] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned fusion-modality-based target detection methods to solve the technical problems of data set sample imbalance and alignment between visual modality and point cloud modality in the prior art.

[0022] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least: obtaining self-collected visual data and first point cloud data; extracting second point cloud data corresponding to the category in the first point cloud data in the open source data set; integrating the first point cloud data with the second point cloud data to obtain third point cloud data; performing feature extraction on the visual data and the third point cloud data respectively to obtain visual modal BEV features and point cloud modal BEV features; obtaining a modal offset matrix based on the visual modal BEV features and the point cloud modal BEV features; performing feature fusion on the visual modal BEV features and the point cloud modal BEV features based on the modal offset matrix to obtain fused features; and performing target detection based on the fused features. This application adds open source datasets to the self-collected datasets. On the one hand, it can solve the negative return problem caused by various differences between multiple datasets, and on the other hand, it can well solve the problem of sample imbalance in the self-collected datasets. The BEV spatial features of the visual modality and the point cloud modality are used as input to predict the modal offset matrix of the two, and then the features of the two modalities are fused according to the generated modal offset matrix, which can make the modalities more aligned and improve the accuracy of the target detection algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 is a flow chart of a target detection method based on fusion modality provided by an embodiment of the present invention;

[0025] Figure 2 is a structural block diagram of a computer device provided by an embodiment of the present invention;

[0026] Figure 3 It is a structural block diagram of a target detection device based on fusion modality provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0028] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0029] In an embodiment of the present invention, a target detection method based on fusion modality is provided, such as Figure 1 As shown, the method includes:

[0030] Step S101, obtaining self-collected visual data and first point cloud data;

[0031] Step S102: extracting second point cloud data corresponding to the category in the first point cloud data from the open source data set;

[0032] Step S103: Integrate the first point cloud data with the second point cloud data to obtain third point cloud data;

[0033] Step S104, performing feature extraction on the visual data and the third point cloud data respectively to obtain visual modality BEV features and point cloud modality BEV features;

[0034] Step S105, obtaining a modal offset matrix based on the visual modality BEV features and the point cloud modality BEV features;

[0035] Step S106: performing feature fusion on the visual modality BEV feature and the point cloud modality BEV feature based on the modality offset matrix to obtain a fused feature;

[0036] Step S107: performing target detection based on the fusion features.

[0037] In this embodiment, the purpose of expanding the self-collected data set is achieved by adding an open source data set to the self-collected data set and utilizing the data of multiple data sets. The traditional approach of utilizing multiple data sets is to put the data of multiple data sets together for training. Due to the differences in the acquisition equipment, acquisition scenarios, and equipment installation locations of each data set, directly training the data of each data set together will result in negative returns. Based on the self-collected data set, this embodiment extracts target point clouds of corresponding categories from the open source data set for the purpose of expanding the types and numbers of whitelisted targets in each category on the self-collected data set. During the training process, the target point clouds are randomly placed in the point cloud of the self-collected data set for training together. This can solve the problem of negative returns caused by various differences between multiple data sets on the one hand, and can also solve the problem of sample imbalance in the self-collected data set on the other hand.

[0038] For example, there are fewer engineering vehicles in the self-collected dataset, so some point clouds belonging to the engineering vehicle category are cut out from the point cloud of the open source dataset and saved separately. During the training of the self-collected dataset, the cut engineering vehicle target point cloud is fused with the point cloud of the self-collected dataset for training.

[0039] During specific implementation, the image dataset targets in the open source dataset can also be intercepted, and the self-collected dataset can also be expanded to solve the problem of sample imbalance in the self-collected dataset.

[0040] In addition, the main difficulty in developing multimodal fusion algorithms is the modal alignment problem. In reality, due to factors such as equipment accuracy and acquisition environment, there will be alignment problems between the visual modality and the point cloud modality. The algorithm in the prior art is to transfer the features of the two to the BEV space and then add them directly. Since there has always been a problem of misalignment in the spatiotemporal domain caused by engineering errors between multiple sensor modalities, the algorithm accuracy of the fused modality is affected. In this embodiment, the BEV spatial features of the two are used as input to predict the modal offset matrix of the two, and then the BEV spatial features of the two modalities are fused based on the generated offset matrix. This makes the modalities more aligned and the algorithm accuracy is higher.

[0041] In one embodiment, the step of integrating the first point cloud data with the second point cloud data to obtain third point cloud data includes:

[0042] Performing point cloud intensity enhancement on the second point cloud data;

[0043] The first point cloud data and the second point cloud data after point cloud strength enhancement are integrated to obtain the third point cloud data.

[0044] In specific implementation, the purpose of enhancing the intensity of point cloud data is to improve the robustness and accuracy of the algorithm. This process involves modifying the intensity information in the point cloud data to generate more diverse data, thereby helping the network to better learn the characteristics of the target under different conditions. When changing the intensity information of the point cloud data, it is first necessary to determine how to change the point cloud intensity. The enhancement strategy may include but is not limited to increasing or decreasing the point cloud intensity value, introducing random noise, or adjusting the intensity value according to a certain distribution model. Modifying the intensity value is to change the intensity information in each point cloud by a certain amplitude during the training phase. The following methods can be used:

[0045] Linear transformation: for example, multiplying the intensity value of each point by a fixed coefficient, or adding / subtracting a fixed offset;

[0046] Non-linear transformation: For example, intensity values ​​are transformed using non-linear functions such as exponential or logarithmic functions;

[0047] Random Perturbation: Adds random noise to intensity values ​​to simulate different lighting or material conditions.

[0048] In one embodiment, the step of performing point cloud strength enhancement on the second point cloud data includes:

[0049] Set the preset intensity coefficient;

[0050] The intensity of each point in the second point cloud data is enhanced based on the preset intensity coefficient.

[0051] In addition to the structural information of the point cloud, the target object information also includes different point cloud intensity information generated by different materials of the target object. The existing data enhancement does not start from the intensity of the point cloud. The applicant of this application has found that the intensity of the point cloud has a great influence on the accuracy of the network during the point cloud training process. Therefore, the embodiments of this application can generate more diverse data by changing the point cloud intensity of the target to a certain extent during training. The data enhancement scheme for point cloud intensity can improve the robustness of the algorithm and make the algorithm more accurate.

[0052] In one embodiment, the extracting features of the visual data and the third point cloud data respectively to obtain visual modality BEV features and point cloud modality BEV features includes:

[0053] Preprocessing the visual data and the third point cloud data respectively to obtain visual preprocessing data and third point cloud preprocessing data;

[0054] Mapping the visual pre-processed data and the third point cloud pre-processed data into the same BEV space;

[0055] Performing feature extraction on the visual preprocessing data in the BEV space to obtain the visual modality BEV feature;

[0056] Feature extraction is performed on the third point cloud preprocessed data in the BEV space to obtain the point cloud modal BEV feature.

[0057] In specific implementation, the point cloud modal BEV feature extraction for the third point cloud data mainly includes the following steps:

[0058] Third, point cloud data preprocessing: cleaning point cloud data and removing noise points; converting point cloud data from the original coordinate system to the vehicle coordinate system or other unified coordinate system;

[0059] Projection to BEV space: Create a two-dimensional grid to represent the bird's-eye view area around the vehicle, project the three-dimensional points in the third point cloud data onto this two-dimensional grid, and each point is assigned to the corresponding grid cell according to its coordinates;

[0060] Feature extraction: In each grid cell, multiple features can be extracted, such as: height feature: the highest point or average height of the points in each cell, density feature: the number of points in each cell, intensity feature: the reflection intensity of the points in each cell, which can be the average intensity or intensity distribution;

[0061] Generate BEV feature map: store the extracted eigenvalues ​​in a two-dimensional array to form a BEV feature map, which is the point cloud modal BEV feature.

[0062] The visual modality BEV feature extraction for visual data mainly includes the following steps:

[0063] Image data preprocessing: Calibrate the image to ensure the consistency of the image and point cloud data in time and space, and perform image distortion correction as needed;

[0064] Convert to BEV space: Use the camera internal and external parameter matrix to convert the pixel coordinates in the image to the three-dimensional world coordinate system, and use perspective projection to convert the feature points in the image to BEV space;

[0065] Feature extraction: Apply feature extraction algorithms (such as convolutional neural networks (CNNs)) to the image to obtain image features, map the image features to corresponding locations in the BEV space, and use feature resampling or interpolation as needed to align with the point cloud data.

[0066] Generate BEV feature map: Map the image features into BEV space to form a BEV feature map, which is the visual modality BEV feature. This process can use a mapping table or function to correspond the feature points in the image to the positions on the BEV grid.

[0067] In one embodiment, obtaining a modality offset matrix based on the visual modality BEV feature and the point cloud modality BEV feature includes:

[0068] The visual modality BEV features and the point cloud modality BEV features are input into a convolutional neural network model for calculation to obtain the modality offset matrix.

[0069] In one embodiment, the step of fusing the visual modality BEV feature and the point cloud modality BEV feature based on the modality offset matrix to obtain a fused feature includes:

[0070] Adjusting the spatial position of the visual modality BEV feature according to the modality offset matrix so that the visual modality BEV feature is aligned with the point cloud modality BEV feature;

[0071] The point cloud modality BEV feature and the visual modality BEV feature after spatial position adjustment are added to obtain the fusion feature.

[0072] Specifically, the modal offset matrix describes the displacement or rotation of the visual modality relative to the point cloud modality in the BEV space. This embodiment uses the BEV spatial features of the two as input to predict the modal offset matrices of the two, and then adjusts the spatial position of the visual modality BEV features according to the generated modal offset matrix, and finally adds the features of the two modalities to obtain the fused features, which makes the modalities more aligned and the algorithm more accurate.

[0073] In one embodiment, the step of integrating the first point cloud data with the second point cloud data to obtain third point cloud data includes:

[0074] Preprocessing the second point cloud data so that the second point cloud data meets the format requirement of the first point cloud data, wherein the preprocessing includes point cloud cropping and format conversion;

[0075] The preprocessed second point cloud data is inserted into the first point cloud data to obtain third point cloud data.

[0076] In specific implementation, when inserting data, it is necessary to consider information such as the position and orientation of the point cloud to ensure that the inserted target point cloud is reasonable in the scene.

[0077] The target detection method of the present application adopts a data enhancement method of intercepting the target point cloud to address the problem that the existing algorithm cannot obtain positive benefits by fusing the entire point cloud of multiple data in experiments, fully utilizing the value of open source data sets and improving the accuracy of the algorithm; in view of the problem that the existing algorithm only considers the spatial structure information of the point cloud when enhancing the data, the present application enhances the intensity information of the target point cloud to further improve the accuracy of the algorithm; in view of the problem that the existing algorithm directly adds multiple modalities and has limited multi-modal benefits, the present application adds an offset matrix module to solve the alignment problem between modalities and further improves the accuracy of the algorithm.

[0078] In this embodiment, a computer device is provided, such as Figure 2 As shown, it includes a memory 201, a processor 202, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, any of the above-mentioned target detection methods based on fusion modality is implemented.

[0079] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0080] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned fusion modality-based target detection methods.

[0081] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include transitory media such as modulated data signals and carrier waves.

[0082] Based on the same inventive concept, an object detection device based on fusion modality is also provided in an embodiment of the present invention, as described in the following embodiment. Since the principle of solving the problem by the object detection device based on fusion modality is similar to that of the object detection method based on fusion modality, the implementation of the object detection device based on fusion modality can refer to the implementation of the object detection method based on fusion modality, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0083] Figure 3 is a structural block diagram of a target detection device based on fusion modality according to an embodiment of the present invention. Figure 3 As shown, it includes: a data acquisition module 301, a data extraction module 302, an integration module 303, a feature extraction module 304, a modal offset matrix acquisition module 305, a feature fusion module 306 and a target detection module 307. The structure is described below.

[0084] A data acquisition module 301 is used to acquire self-collected visual data and first point cloud data;

[0085] A data extraction module 302 is used to extract second point cloud data corresponding to the category in the first point cloud data from the open source data set;

[0086] An integration module 303, used to integrate the first point cloud data with the second point cloud data to obtain third point cloud data;

[0087] A feature extraction module 304 is used to extract features from the visual data and the third point cloud data to obtain visual modality BEV features and point cloud modality BEV features;

[0088] A modal offset matrix acquisition module 305 is used to obtain a modal offset matrix based on the visual modal BEV feature and the point cloud modal BEV feature;

[0089] A feature fusion module 306 is used to fuse the visual modality BEV feature and the point cloud modality BEV feature based on the modality offset matrix to obtain a fused feature;

[0090] The target detection module 307 is used to perform target detection based on the fusion features.

[0091] In one embodiment, the integration module 303 is further configured to:

[0092] Performing point cloud intensity enhancement on the second point cloud data;

[0093] The first point cloud data and the second point cloud data after point cloud strength enhancement are integrated to obtain the third point cloud data.

[0094] In one embodiment, the integration module 303 is further configured to:

[0095] Set the preset intensity coefficient;

[0096] The intensity of each point in the second point cloud data is enhanced based on the preset intensity coefficient.

[0097] In one embodiment, the feature extraction module 304 is further used to:

[0098] Preprocessing the visual data and the third point cloud data respectively to obtain visual preprocessing data and third point cloud preprocessing data;

[0099] Mapping the visual pre-processed data and the third point cloud pre-processed data into the same BEV space;

[0100] Performing feature extraction on the visual preprocessing data in the BEV space to obtain the visual modality BEV feature;

[0101] Feature extraction is performed on the third point cloud preprocessed data in the BEV space to obtain the point cloud modal BEV feature.

[0102] In one embodiment, the modal shift matrix acquisition module 305 is further used to:

[0103] The visual modality BEV features and the point cloud modality BEV features are input into a convolutional neural network model for calculation to obtain the modality offset matrix.

[0104] In one embodiment, the feature fusion module 306 is further used to:

[0105] Adjusting the spatial position of the visual modality BEV feature according to the modality offset matrix so that the visual modality BEV feature is aligned with the point cloud modality BEV feature;

[0106] The point cloud modality BEV feature and the visual modality BEV feature after spatial position adjustment are added to obtain the fusion feature.

[0107] In one embodiment, the integration module 303 is further configured to:

[0108] Preprocessing the second point cloud data so that the second point cloud data meets the format requirement of the first point cloud data, wherein the preprocessing includes point cloud cropping and format conversion;

[0109] The preprocessed second point cloud data is inserted into the first point cloud data to obtain third point cloud data.

[0110] The embodiment of the present invention achieves the following technical effects: obtaining self-collected visual data and first point cloud data; extracting second point cloud data corresponding to the category in the first point cloud data from the open source data set; integrating the first point cloud data with the second point cloud data to obtain third point cloud data; extracting features from the visual data and the third point cloud data respectively to obtain visual modality BEV features and point cloud modality BEV features; obtaining a modal offset matrix based on the visual modality BEV features and the point cloud modality BEV features; performing feature fusion of the visual modality BEV features and the point cloud modality BEV features based on the modal offset matrix to obtain fused features; and performing target detection based on the fused features. This application can solve the problem of negative returns caused by various differences between multiple data sets on the one hand, and can solve the problem of sample imbalance in the self-collected data set on the other hand by adding an open source data set to the self-collected data set; using the BEV spatial features of the visual modality and the point cloud modality as input to predict the modal offset matrix of the two, and then performing feature fusion of the two modalities based on the generated modal offset matrix, so that the modalities can be more aligned and the accuracy of the target detection algorithm can be improved.

[0111] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A target detection method based on fusion modality, characterized in that: include: Obtain self-collected visual data and the first point cloud data; Extracting second point cloud data corresponding to the category in the first point cloud data from the open source data set; Integrate the first point cloud data with the second point cloud data to obtain third point cloud data; Performing feature extraction on the visual data and the third point cloud data respectively to obtain visual modality BEV features and point cloud modality BEV features; Obtaining a modal offset matrix based on the visual modality BEV features and the point cloud modality BEV features; Based on the modality offset matrix, the visual modality BEV feature and the point cloud modality BEV feature are subjected to feature fusion to obtain a fusion feature; Target detection is performed based on the fused features.

2. The target detection method based on fusion modality according to claim 1, characterized in that: The step of integrating the first point cloud data with the second point cloud data to obtain third point cloud data includes: Performing point cloud intensity enhancement on the second point cloud data; The first point cloud data and the second point cloud data after point cloud strength enhancement are integrated to obtain the third point cloud data.

3. The target detection method based on fusion modality according to claim 2, characterized in that: The step of performing point cloud strength enhancement on the second point cloud data includes: Set the preset intensity coefficient; The intensity of each point in the second point cloud data is enhanced based on the preset intensity coefficient.

4. The target detection method based on fusion modality according to claim 1, characterized in that: The step of extracting features from the visual data and the third point cloud data to obtain visual modality BEV features and point cloud modality BEV features includes: Preprocessing the visual data and the third point cloud data respectively to obtain visual preprocessing data and third point cloud preprocessing data; Mapping the visual pre-processed data and the third point cloud pre-processed data into the same BEV space; Performing feature extraction on the visual preprocessing data in the BEV space to obtain the visual modality BEV feature; Feature extraction is performed on the third point cloud preprocessed data in the BEV space to obtain the point cloud modal BEV feature.

5. The target detection method based on fusion modality according to claim 1, characterized in that: The obtaining of a modal offset matrix based on the visual modality BEV feature and the point cloud modality BEV feature comprises: The visual modality BEV features and the point cloud modality BEV features are input into a convolutional neural network model for calculation to obtain the modality offset matrix.

6. The target detection method based on fusion modality according to claim 5, characterized in that: The step of fusing the visual modality BEV feature and the point cloud modality BEV feature based on the modality offset matrix to obtain a fused feature includes: Adjusting the spatial position of the visual modality BEV feature according to the modality offset matrix so that the visual modality BEV feature is aligned with the point cloud modality BEV feature; The point cloud modality BEV feature and the visual modality BEV feature after spatial position adjustment are added to obtain the fusion feature.

7. The target detection method based on fusion modality according to claim 1, characterized in that: The step of integrating the first point cloud data with the second point cloud data to obtain third point cloud data includes: Preprocessing the second point cloud data so that the second point cloud data meets the format requirement of the first point cloud data, wherein the preprocessing includes point cloud cropping and format conversion; The preprocessed second point cloud data is inserted into the first point cloud data to obtain third point cloud data.

8. A target detection device based on fusion modality, characterized in that: include: A data acquisition module, used to acquire self-collected visual data and first point cloud data; A data extraction module, configured to extract second point cloud data corresponding to a category in the first point cloud data from an open source data set; An integration module, used for integrating the first point cloud data with the second point cloud data to obtain third point cloud data; A feature extraction module, used for performing feature extraction on the visual data and the third point cloud data respectively to obtain visual modality BEV features and point cloud modality BEV features; A modal offset matrix acquisition module, used to obtain a modal offset matrix based on the visual modal BEV feature and the point cloud modal BEV feature; A feature fusion module, used for fusing the visual modality BEV feature and the point cloud modality BEV feature based on the modality offset matrix to obtain a fused feature; The target detection module is used to perform target detection based on the fusion features.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the target detection method based on fusion modality described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the object detection method based on fusion modality according to any one of claims 1 to 7.

Images