Object detection method and device based on general features of objects, equipment and medium

Abstract

The application relates to an object detection method and device based on general features of an object, equipment and a medium, wherein the method comprises the following steps: obtaining a laser radar point cloud image, a visible light image and a thermal infrared image; inputting the laser radar point cloud image into a preset backbone network to extract point cloud features; fusing the visible light image and the thermal infrared image to obtain a dual-light fusion image; extracting target image features from the dual-light fusion image; performing cross-modal fusion on the target image features and the point cloud features based on a preset bird's eye view space; and performing target detection on the fusion result in the bird's eye view space. Thus, the problem that a target detection method in the related art cannot obtain depth information of a detection task and thus cannot meet various complex scenes of an environment perception system in various fields is solved, and the best inference effect of target detection is achieved by extracting intermediate layer features from data collected by different sensors and fusing effective features of the sensors.

Description

Technical Field

[0001] This application relates to the fields of general object feature extraction and multi-sensor feature fusion, and in particular to a target detection method, device, equipment and medium based on general object features. Background Technology

[0002] With the rapid development of multi-sensor fusion methods for environmental perception tasks, a range of approaches has gradually emerged, from multi-sensor feature representations to more complex and robust multi-sensor fusion deep learning models and technologies. Based on fusion rules, multi-sensor fusion methods can be categorized into three main types: pre-fusion, feature fusion, and post-fusion, corresponding to data-level, feature-level, and target-level fusion, respectively. Feature fusion represents the main development trend in multi-sensor fusion.

[0003] In related technologies, most adopt pre-fusion methods to achieve the purpose of using multimodal sensor fusion to detect targets, that is, to fuse the data collected by the sensors and extract the object features through data-level processing technology.

[0004] However, when detecting targets, the pre-fusion method is prone to a lot of resource waste and information redundancy after fusing information from different sensors, resulting in data and computing power loss. At the same time, after detecting targets through the pre-fusion method, environmental noise can easily affect the quality of feature extraction, thus failing to meet the requirements of environmental perception systems in various complex scenarios in different fields. Summary of the Invention

[0005] This application provides a target detection method, device, electronic device, and storage medium based on general object features to solve the problem that target detection methods in related technologies cannot obtain depth information of the detection task, thus failing to meet the needs of environmental perception systems in various complex scenarios across different fields.

[0006] The first aspect of this application provides a target detection method based on general object features, including the following steps:

[0007] Acquire point cloud images from lidar, visible light images from a visible light camera, and thermal infrared images from an infrared camera;

[0008] The lidar point cloud image is input into a preset backbone network. Point cloud features are segmented based on a multi-scale point cloud segmentation network to extract the point cloud features of the lidar point cloud image. A dual-light fused image is then obtained by fusing the visible light image and the thermal infrared image.

[0009] Target image features are extracted from the dual-light image, and cross-modal fusion of the target image features and the point cloud features is performed based on a preset bird's-eye view space. Target detection is then performed in the bird's-eye view space based on the cross-modal fusion result.

[0010] According to one embodiment of this application, before acquiring the lidar point cloud image acquired by the lidar, the visible light image acquired by the visible light camera, and the thermal infrared image acquired by the infrared camera, the method further includes:

[0011] According to the preset calibration strategy, the external parameters of the lidar point cloud image are calibrated, and the internal parameters of the visible light image and the thermal infrared image are calibrated.

[0012] Synchronize the timestamps of the visible light camera and the infrared camera.

[0013] According to one embodiment of this application, a dual-light fused image is obtained by fusing the visible light image and the thermal infrared image, including:

[0014] The converted visible light image and the converted thermal infrared image are acquired to generate a converted visible light image and a converted thermal infrared image.

[0015] Based on a preset transformation model, the converted visible light image and the converted thermal infrared image are registered respectively to obtain...

[0016] The registered visible light image and the registered thermal infrared image are obtained, wherein the preset transformation model is:

[0017]

[0018] Among them, (x i ,y i (x,y) and (t) are the coordinates in the visible light image and the infrared image, respectively. x ,t y ) is the offset matrix; based on the preset dual discriminator generative adversarial network, the high-resolution converted visible light image and the low-resolution registered thermal infrared image are fused to obtain the dual-light fused image.

[0019] According to one embodiment of this application, target image features are extracted from the two-light image, and cross-modal fusion is performed on the target image features and the point cloud features based on a preset bird's-eye view space. Target detection is then performed in the bird's-eye view space based on the cross-modal fusion result, including:

[0020] The 2D image features of the two-light image are extracted based on the ResNet101 backbone network, and the 2D image features of the two-light fused image are transformed into 3D point cloud features of the preset bird's-eye view space using the BEVFormer network to obtain the preset bird's-eye view space point cloud features.

[0021] Using a preset fusion weighting algorithm, the laser radar point cloud features and the preset bird's-eye view spatial point cloud features are fused across modes to obtain the cross-modal fusion result;

[0022] Target detection is performed in the bird's-eye view space based on the cross-modal fusion results.

[0023] According to the target detection method based on general object features according to the embodiments of this application, LiDAR point cloud images, visible light images, and thermal infrared images are acquired. The LiDAR point cloud images are input into a preset backbone network to extract point cloud features. A dual-light fusion image is obtained by fusing the visible light image and the thermal infrared image. Target image features are extracted from the dual-light fusion image. Based on a preset bird's-eye view space, cross-modal fusion is performed on the target image features and point cloud features. The fusion result is then used for target detection in the bird's-eye view space. This solves the problem that related target detection methods cannot obtain depth information for the detection task, thus failing to meet the needs of environmental perception systems in various complex scenarios across different fields. By extracting intermediate layer features from data collected by different sensors and fusing effective sensor features, the optimal inference effect for target detection is achieved.

[0024] A second aspect of this application provides a target detection device based on general object features, comprising:

[0025] The acquisition module is used to acquire point cloud images from the lidar, visible light images from the visible light camera, and thermal infrared images from the infrared camera.

[0026] The extraction module is used to input the lidar point cloud image into a preset backbone network, segment point cloud features based on a multi-scale point cloud segmentation network, extract the point cloud features of the lidar point cloud image, and obtain a dual-light fused image by fusing the visible light image and the thermal infrared image; and

[0027] The detection module is used to extract target image features from the dual-light image, and perform cross-modal fusion of the target image features and the point cloud features based on a preset bird's-eye view space, and perform target detection in the bird's-eye view space based on the cross-modal fusion result.

[0028] According to one embodiment of this application, before acquiring the lidar point cloud image collected by the lidar, the visible light image collected by the visible light camera, and the thermal infrared image collected by the infrared camera, the acquisition module is further configured to:

[0029] According to the preset calibration strategy, the external parameters of the lidar point cloud image are calibrated, and the internal parameters of the visible light image and the thermal infrared image are calibrated.

[0030] Synchronize the timestamps of the visible light camera and the infrared camera.

[0031] According to one embodiment of this application, the extraction module is specifically used for:

[0032] The converted visible light image and the converted thermal infrared image are acquired to generate a converted visible light image and a converted thermal infrared image;

[0033] Based on a preset transformation model, the converted visible light image and the converted thermal infrared image are registered respectively to obtain a registered visible light image and a registered thermal infrared image, wherein the preset transformation model is:

[0034]

[0035] Among them, (x i ,y i (x,y) and (t) are the coordinates in the visible light image and the infrared image, respectively. x ,t y ) is the offset matrix;

[0036] A dual-discriminator generative adversarial network is used to fuse a high-resolution converted visible light image and a low-resolution registered thermal infrared image to obtain a dual-light fused image.

[0037] According to one embodiment of this application, the detection module is specifically used for:

[0038] The 2D image features of the two-light image are extracted based on the ResNet101 backbone network, and the 2D image features of the two-light fused image are transformed into 3D point cloud features of the preset bird's-eye view space using the BEVFormer network to obtain the preset bird's-eye view space point cloud features.

[0039] Using a preset fusion weighting algorithm, the laser radar point cloud features and the preset bird's-eye view spatial point cloud features are fused across modes to obtain the cross-modal fusion result;

[0040] Target detection is performed in the bird's-eye view space based on the cross-modal fusion results.

[0041] According to the target detection device based on general object features according to the embodiments of this application, a LiDAR point cloud image, a visible light image, and a thermal infrared image are acquired. The LiDAR point cloud image is input into a preset backbone network to extract point cloud features. A dual-light fusion image is obtained by fusing the visible light image and the thermal infrared image. Target image features are extracted from the dual-light fusion image. Based on a preset bird's-eye view space, cross-modal fusion of the target image features and point cloud features is performed. The fusion result is then used for target detection in the bird's-eye view space. This solves the problem that related target detection methods cannot obtain depth information for the detection task, thus failing to meet the needs of environmental perception systems in various complex scenarios across different fields. By extracting intermediate layer features from data collected by different sensors and fusing effective sensor features, the optimal inference effect for target detection is achieved.

[0042] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target detection method based on general object features as described in the above embodiments.

[0043] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the target detection method based on general object features as described in the above embodiments.

[0044] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0045] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0046] Figure 1 This is a flowchart of a target detection method based on general object features provided in an embodiment of this application;

[0047] Figure 2 This is a schematic diagram of the framework of a target detection method based on general object features using lidar and dual-light fusion according to an embodiment of this application;

[0048] Figure 3 This is a schematic diagram of a deep learning network framework according to an embodiment of this application;

[0049] Figure 4 This is a schematic diagram illustrating the steps of fusing visible light and infrared images according to an embodiment of this application;

[0050] Figure 5This is a schematic diagram of a generative adversarial network model structure according to an embodiment of this application;

[0051] Figure 6 A block diagram of a target detection device based on general object features according to an embodiment of this application;

[0052] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0053] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0054] The following description, with reference to the accompanying drawings, outlines a target detection method, apparatus, electronic device, and storage medium based on general object features according to embodiments of this application. Addressing the problem mentioned in the background art that target detection methods cannot acquire depth information for the detection task, thus failing to meet the needs of environmental perception systems in various complex scenarios across different fields, this application provides a target detection method based on general object features. In this method, LiDAR point cloud images, visible light images, and thermal infrared images are acquired. The LiDAR point cloud image is input into a preset backbone network to extract point cloud features. A dual-light fusion image is obtained by fusing the visible light image and the thermal infrared image. Target image features are extracted from the dual-light fusion image. Based on a preset bird's-eye view space, cross-modal fusion of the target image features and point cloud features is performed. The fusion result is then used for target detection in the bird's-eye view space. This solves the problem that target detection methods in related technologies cannot acquire depth information for the detection task, thus failing to meet the needs of environmental perception systems in various complex scenarios across different fields. By extracting intermediate layer features from data collected by different sensors and fusing effective sensor features, optimal inference performance for target detection is achieved.

[0055] Specifically, Figure 1 This is a flowchart illustrating a target detection method based on general object features provided in an embodiment of this application.

[0056] like Figure 1 As shown, this target detection method based on general object features includes the following steps:

[0057] In step S101, the lidar point cloud image collected by the lidar, the visible light image collected by the visible light camera, and the thermal infrared image collected by the infrared camera are acquired.

[0058] Furthermore, in some embodiments, before acquiring the visible light image captured by the visible light camera and the thermal infrared image captured by the infrared camera, the method further includes: calibrating the external parameters of the lidar point cloud image according to a preset calibration strategy, and calibrating the internal parameters of the visible light image and the thermal infrared image; and synchronizing the timestamps of the visible light camera and the infrared camera.

[0059] The preset calibration strategy can be formulated by those skilled in the art based on the actual needs of target detection, and is not specifically limited here.

[0060] Specifically, before acquiring images of the target detection object, this embodiment first needs to define the general feature attributes of the target detection object. For example, it needs to define the object category, the bounding box (length, width, center point) coordinates of the 2D object image, the 3D object size (length, width, height), the center point coordinates of the 3D object, the rotation angle of the target detection object, etc., and define the feature attributes of the target detection object, such as the object's static attribute, motion attribute, occlusion and truncation ratio, etc.; secondly, after the feature attributes of the target detection object are defined, this embodiment can acquire data of the target detection object through a multi-mode sensor with spatiotemporal synchronization function.

[0061] Furthermore, in this embodiment of the application, when acquiring images of the target detection object, the multi-mode sensor requires at least one LiDAR scanner, a visible light camera, and an infrared camera. The LiDAR scanner, visible light camera, and infrared camera respectively acquire corresponding images of the target detection object. Before acquisition, the visible light camera and infrared camera need to be calibrated internally, and the system composed of the visible light camera, infrared camera, and LiDAR needs to be calibrated externally to achieve spatiotemporal synchronization. The calibration method for the visible light camera and infrared camera can adopt Zhang's calibration method. The main calibration method involves calibrating with a calibration plate, calculating the mapping relationship of the calibration plate from the world coordinate system to the imaging coordinate system, and calculating the position coordinates of the calibration plate in 3D space. The transformation relationship can be expressed as:

[0062]

[0063] Where (u,v) are the coordinates in the pixel coordinate system, (X... w ,Y w Z w () represents the coordinates in the world coordinate system, f x f is the product of the pixel's physical scale along the X-axis and the focal length. y =U0 is the product of the physical scale of the pixel in the Y-axis direction and the focal length, u0 is the offset of the image origin in the pixel plane in the X-axis direction, and v0 is the offset of the image origin in the pixel plane in the Y-axis direction.

[0064] For example, such as Figure 2 As shown, this embodiment of the application takes the joint calibration between a lidar and a camera as an example. Constraints are established between the 3D laser point of the lidar and the 3D coordinates on the calibration board measured by the camera. The lidar coordinate system is rotated and translated to the camera coordinate system to obtain its rotation and translation matrices, respectively. t lc Therefore, the transformation relationship from the lidar coordinate system to the camera coordinate system can be obtained as follows:

[0065]

[0066] Where (x, y, z) are the coordinates in the lidar coordinate system, and (x, y, z, 1) is the coordinate matrix operation performed by expanding the three-dimensional coordinate vector into a four-dimensional vector.

[0067] Furthermore, in this embodiment of the application, after jointly calibrating the general feature attributes of the target detection object and synchronizing the timestamps of the visible light camera and the infrared camera, the lidar point cloud image collected by the lidar, the visible light image collected by the visible light camera, and the thermal infrared image collected by the infrared camera can be obtained. At the same time, after acquiring the lidar point cloud image, the visible light image, and the thermal infrared image, the acquired images are calibrated using the general feature attributes of the target detection object defined above, and information such as the size, position coordinates, and rotation angle of the image are obtained respectively.

[0068] In step S102, the lidar point cloud image is input into a preset backbone network, the point cloud features are segmented based on a multi-scale point cloud segmentation network, the point cloud features of the lidar point cloud image are extracted, and a dual-light fused image is obtained by fusing the visible light image and the thermal infrared image.

[0069] Preferably, the preset backbone network used in the embodiments of this application can be the multi-scale point cloud segmentation network PointNet++, or other deep learning networks, without specific limitations.

[0070] Specifically, such as Figure 3 As shown, in this embodiment of the application, after acquiring lidar point cloud images, visible light images, and thermal infrared images, the lidar point cloud images are input into the PointNet++ network to achieve foreground and background point segmentation and extraction of candidate features from the point cloud images. Since the labeled data of the point cloud images clearly marks the foreground and background points, but the number of background points is far greater than the number of foreground points, in practice, the network model can use focal loss to reduce the loss contribution of easily classified samples and increase the loss proportion of difficult-to-classify samples to solve the class imbalance problem. Specifically, this can be manifested as follows:

[0071] L focal(p t )=-α t (1-p t ) γ log(p t ),

[0072]

[0073] Among them, L focal (p t ) is the focal loss function, α t As an adjustment parameter to suppress the imbalance of positive and negative sample numbers, (1-p t ) γ The proportion of loss for hard-to-classify samples, p t To determine whether a sample point is a foreground point, p is the probability predicted by the model to belong to the foreground point.

[0074] Furthermore, in this embodiment of the application, the extraction of candidate features of point cloud images in the foreground point includes randomly sampling several points in each frame of point cloud data, obtaining the feature vector of each point through encoding and decoding of the PointNet++ network, and using it to segment each point to finally obtain the feature points of the object. After the above processing, the point cloud features of the lidar point cloud image in the foreground point can be extracted.

[0075] Furthermore, in some embodiments, obtaining a dual-light fused image by fusing a visible light image and a thermal infrared image includes: acquiring a converted visible light image and a thermal infrared image to generate a converted visible light image and a converted thermal infrared image; registering the converted visible light image and the converted thermal infrared image respectively based on a preset transformation model to obtain a registered visible light image and a registered thermal infrared image, wherein the preset transformation model is:

[0076]

[0077] Among them, (x i ,y i (x,y) and (t) are the coordinates in the visible light image and the infrared image, respectively. x ,t y ) is the offset matrix; based on the preset dual discriminator generative adversarial network, the high-resolution converted visible light image and the low-resolution registered thermal infrared image are fused to obtain the dual-light fused image.

[0078] Specifically, such as Figure 4 As shown, the embodiments of this application require preprocessing of the acquired and labeled visible light and infrared images. The main processes are calculating image offset and performing dual-light image registration.

[0079] Specifically, in this embodiment, for the calibrated visible light camera and infrared camera, a timer is used to externally trigger the dual-light camera and synchronously acquire dual-light image data. The obtained dual-light images are scaled to a uniform size. The pixel difference between the visible light image and the infrared image in the world coordinate system is calculated using the camera parameters and scale factor obtained from camera calibration. Then, pixel difference offset and rotation are used to make the position of each pixel of the object consistent in the dual-light image, ultimately achieving the purpose of dual-light image registration, thereby obtaining a registered visible light image and a registered thermal infrared image. The preset transformation model of the dual-light image can be expressed as:

[0080]

[0081] Among them, (x i ,y i (x,y) and (t) are the coordinates in the visible light image and the infrared image, respectively. x ,t y ) is the offset matrix.

[0082] Furthermore, such as Figure 4 As shown in the figure, after the visible light image and thermal infrared image in this embodiment of the application are registered, image fusion is required. This is achieved by using a generative adversarial network with dual discriminators to fuse the low-resolution infrared image and the high-resolution visible light image. The network model structure is as follows: Figure 5 As shown, a fused image is generated by generator G to deceive discriminator D. Discriminator D aims to distinguish the image fused by generator G from the original two-light image. Finally, the two-light fused image is output through the network model, while retaining the thermal radiation characteristics of the infrared image and the texture information of the visible light image.

[0083] In step S103, target image features are extracted from the dual-light image, and cross-modal fusion of target image features and point cloud features is performed based on a preset bird's-eye view space. Target detection is then performed in the bird's-eye view space based on the cross-modal fusion result.

[0084] Furthermore, in some embodiments, target image features are extracted from the dual-light image, and cross-modal fusion of the target image features and point cloud features is performed based on a preset bird's-eye view space. Target detection is then performed in the bird's-eye view space based on the cross-modal fusion result. This includes: extracting 2D image features from the dual-light image based on the ResNet101 backbone network, and using the BEVFormer network to transform the 2D image features of the dual-light fused image into 3D point cloud features in a preset bird's-eye view space, obtaining preset bird's-eye view space point cloud features; performing cross-modal fusion of the lidar point cloud features and the preset bird's-eye view space point cloud features using a preset fusion weighting algorithm, obtaining a cross-modal fusion result; and performing target detection in the bird's-eye view space based on the cross-modal fusion result.

[0085] The preset bird's-eye view space is the BEV (Bird's Eye View) space.

[0086] Specifically, such as Figure 3 As shown, in this embodiment, after outputting the dual-light fusion image through the network model, the dual-light fusion image is input into the ResNet101 backbone network, and the 2D image features of the dual-light image are extracted based on the ResNet101 backbone network. The main method is to use the ResNet101 backbone network to extract the feature map of the 2D dual-light fusion image. Since the deep feature map has stronger semantic features and the shallow feature map has stronger positional information, the semantic information of the deep layer is transferred to the shallow layer through the FPN (Feature Pyramid Networks) structure, and the positioning information of the shallow layer is transferred to the deep layer through the PAN (Path Aggregation Network) structure. Finally, the purpose of enhancing the feature fusion of different layers is achieved, and multi-scale feature maps are output.

[0087] Furthermore, such as Figure 3 As shown, this embodiment of the application extracts 2D image features from two-light images using a ResNet101 backbone network and an attention mechanism. The BEVFormer network is then used to transform the 2D image features of the fused two-light image into 3D point cloud features in BEV space, resulting in BEV space point cloud features. Specifically, during the image feature to BEV space conversion, the BEVFormer network performs the conversion using a network-partitioned BEV query mechanism and a spatial cross-attention mechanism.

[0088] Specifically, in this embodiment, a space of size H×W×C is set, where H and W are the dimensions of the BEV space, and C is the height coordinate perpendicular to the H×W plane. With the target object as the origin, the BEV query mechanism continuously queries to find the columnar 3D reference point of each coordinate point (x,y) in the space, and finally completes the conversion of all 2D image features to the BEV space.

[0089] Furthermore, such as Figure 2 As shown, this application utilizes a convolutional network to extract the features of the transformed dual-light fusion image and projects the image features into the BEV space. At the same time, it fuses the features with the point cloud features of the LiDAR point cloud image. Since the BEV space and the LiDAR point cloud space are sparse, a depthwise continuous convolutional network can be used to extract the dual-light fusion image.

[0090] Furthermore, in this embodiment, given a set of BEV space image features and a set of LiDAR point cloud features, each pixel in the BEV space contains features extracted from the dual-light fusion image, thereby creating a dense BEV feature map. The main process is as follows: First, for each target pixel in the BEV space image features, find its nearest neighboring LiDAR 3D points on the 2D BEV plane, and back-project these LiDAR 3D points into the 3D space. Second, calculate the 3D neighbor offset between the original LiDAR points and the target detection object image, establishing the geometric relationship between the point cloud features and image features in the BEV space. Predict the image features and continuous geometric offsets using a deep continuous convolutional network, ultimately completing the cross-modal fusion of image features and LiDAR point clouds in the BEV space. This yields the feature values ​​of the detected object in the BEV space. The fused BEV space is then processed by a neural network to calculate and output the detection results. Finally, the loss between the anchor box and predefined object general features is calculated using an anchor-base approach. Finally, a trained model is output to identify and detect objects.

[0091] In summary, the advantages of the embodiments of this application include:

[0092] (1) By fusing visible light images and infrared images and extracting object features from 2D images, the features of objects under various lighting conditions and harsh environments can be effectively identified. This effectively avoids the problem that visible light cameras cannot obtain effective image information under insufficient lighting conditions (including night, rain, snow, glare, etc.), thus solving the problem that visible light cameras cannot work properly in harsh environments.

[0093] (2) The method of using LiDAR and camera fusion can not only obtain information such as the edge, texture and semantics of objects, but also depth information such as spatial position, size and distance. It is suitable for multi-scene and multi-modal environmental perception tasks to provide more accurate target detection and classification.

[0094] (3) The fusion scheme based on object feature extraction effectively avoids the impact of background noise on environmental perception tasks, and improves the accuracy and speed of target detection.

[0095] (4) The effective features of objects are fused in the BEV space, which can reduce the consumption of computing power and the loss of data, and has certain advantages compared with other fusion methods.

[0096] (5) This application adopts a decoupled architecture, in which each module is independent of the others. Each module and function can be replaced according to actual needs and can be flexibly combined to reduce the dependence between modules.

[0097] According to the target detection method based on general object features according to the embodiments of this application, LiDAR point cloud images, visible light images, and thermal infrared images are acquired. The LiDAR point cloud images are input into a preset backbone network to extract point cloud features. A dual-light fusion image is obtained by fusing the visible light image and the thermal infrared image. Target image features are extracted from the dual-light fusion image. Based on a preset bird's-eye view space, cross-modal fusion is performed on the target image features and point cloud features. The fusion result is then used for target detection in the bird's-eye view space. This solves the problem that related target detection methods cannot obtain depth information for the detection task, thus failing to meet the needs of environmental perception systems in various complex scenarios across different fields. By extracting intermediate layer features from data collected by different sensors and fusing effective sensor features, the optimal inference effect for target detection is achieved.

[0098] Next, with reference to the accompanying drawings, a target detection device based on general object features proposed according to an embodiment of this application is described.

[0099] Figure 6 This is a block diagram of a target detection device based on general object features according to an embodiment of this application.

[0100] like Figure 6 As shown, the target detection device 10 based on the general features of objects includes: an acquisition module 100, an extraction module 200, and a detection module 300.

[0101] The acquisition module 100 is used to acquire point cloud images from the lidar, visible light images from the visible light camera, and thermal infrared images from the infrared camera.

[0102] The extraction module 200 is used to input the lidar point cloud image into a preset backbone network, segment the point cloud features based on a multi-scale point cloud segmentation network, extract the point cloud features of the lidar point cloud image, and obtain a dual-light fused image by fusing the visible light image and the thermal infrared image; and

[0103] The detection module 300 is used to extract target image features from the dual-light image, and perform cross-modal fusion of target image features and point cloud features based on a preset bird's-eye view space, and perform target detection in the bird's-eye view space based on the cross-modal fusion result.

[0104] Furthermore, in some embodiments, before acquiring the lidar point cloud image acquired by the lidar, the visible light image acquired by the visible light camera, and the thermal infrared image acquired by the infrared camera, the acquisition module 100 is further configured to:

[0105] According to the preset calibration strategy, the external parameters of the lidar point cloud image are calibrated, and the internal parameters of the visible light image and the thermal infrared image are calibrated.

[0106] Synchronize the timestamps of the visible light camera and the infrared camera.

[0107] Furthermore, in some embodiments, the extraction module 200 is specifically used for:

[0108] Acquire and convert visible light and thermal infrared images, and generate converted visible light and thermal infrared images;

[0109] Based on a preset transformation model, the converted visible light image and the converted thermal infrared image are registered respectively, resulting in a registered visible light image and a registered thermal infrared image. The preset transformation model is as follows:

[0110]

[0111] Among them, (x i ,y i (x,y) and (t) are the coordinates in the visible light image and the infrared image, respectively. x ,t y ) is the offset matrix;

[0112] A dual-discriminator generative adversarial network is used to fuse a high-resolution converted visible light image and a low-resolution registered thermal infrared image to obtain a dual-light fused image.

[0113] Furthermore, in some embodiments, the detection module 300 is specifically used for:

[0114] The 2D image features of the two-light image are extracted based on the ResNet101 backbone network, and the 2D image features of the two-light fused image are transformed into 3D point cloud features of the preset bird's-eye view space using the BEVFormer network to obtain the preset bird's-eye view space point cloud features.

[0115] Using a pre-defined fusion weighting algorithm, cross-modal fusion is performed on the laser radar point cloud features and the pre-defined bird's-eye view spatial point cloud features to obtain the cross-modal fusion result;

[0116] Target detection is performed in bird's-eye view space based on cross-modal fusion results.

[0117] According to the target detection device based on general object features according to the embodiments of this application, a LiDAR point cloud image, a visible light image, and a thermal infrared image are acquired. The LiDAR point cloud image is input into a preset backbone network to extract point cloud features. A dual-light fusion image is obtained by fusing the visible light image and the thermal infrared image. Target image features are extracted from the dual-light fusion image. Based on a preset bird's-eye view space, cross-modal fusion of the target image features and point cloud features is performed. The fusion result is then used for target detection in the bird's-eye view space. This solves the problem that related target detection methods cannot obtain depth information for the detection task, thus failing to meet the needs of environmental perception systems in various complex scenarios across different fields. By extracting intermediate layer features from data collected by different sensors and fusing effective sensor features, the optimal inference effect for target detection is achieved.

[0118] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0119] The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0120] When the processor 702 executes the program, it implements the target detection method based on general object features provided in the above embodiments.

[0121] Furthermore, electronic devices also include:

[0122] Communication interface 703 is used for communication between memory 701 and processor 702.

[0123] The memory 701 is used to store computer programs that can run on the processor 702.

[0124] The memory 701 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0125] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0126] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0127] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0128] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described target detection method based on general object features.

[0129] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0130] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0131] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0132] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0133] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0134] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0135] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0136] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A target detection method based on general features of objects, characterized in that, The method comprises the following steps: acquiring a laser radar point cloud image collected by a laser radar, a visible light image collected by a visible light camera, and a thermal infrared image collected by an infrared camera; inputting the laser radar point cloud image into a preset backbone network, segmenting point cloud features based on a multi-scale point cloud segmentation network, extracting point cloud features of the laser radar point cloud image, and fusing a dual-light fusion image according to the visible light image and the thermal infrared image; and extracting target image features from the dual-light image, performing cross-modal fusion on the target image features and the point cloud features based on a preset bird's eye view space, and performing target detection in the bird's eye view space based on a cross-modal fusion result; wherein the extraction of the target image features from the dual-light image, the cross-modal fusion on the target image features and the point cloud features based on the preset bird's eye view space, and the target detection in the bird's eye view space based on the cross-modal fusion result comprise: extracting 2D image features of the dual-light image based on a ResNet101 backbone network, and converting the 2D image features of the dual-light fusion image to 3D point cloud features in the preset bird's eye view space by using a BEVFormer network to obtain the preset bird's eye view space point cloud features; performing cross-modal fusion on the laser point cloud features and the preset bird's eye view space point cloud features by using a preset fusion weighting algorithm to obtain the cross-modal fusion result; and performing target detection in the bird's eye view space based on the cross-modal fusion result.

2. The method of claim 1, wherein, Before acquiring the laser radar point cloud image collected by the laser radar, the visible light image collected by the visible light camera, and the thermal infrared image collected by the infrared camera, the method further comprises: performing external parameter calibration on the laser radar point cloud image, and performing internal parameter calibration on the visible light image and the thermal infrared image according to a preset calibration strategy; synchronizing time stamps of the visible light camera and the infrared camera.

3. The method of claim 1, wherein, Fusing the visible light image and the thermal infrared image to obtain a dual-light fusion image comprises: collecting the converted visible light image and the converted thermal infrared image to generate an already converted visible light image and an already converted thermal infrared image; respectively registering the already converted visible light image and the already converted thermal infrared image based on a preset transformation model to obtain a registered visible light image and a registered thermal infrared image, wherein the preset transformation model is: ； wherein, and are the coordinates under the visible light image and the infrared image, respectively, is the offset matrix; respectively fusing a high-resolution converted visible light image and a low-resolution registered thermal infrared image based on a preset dual-discriminator generative adversarial network to obtain a dual-light fusion image.

4. An object detection device based on general features of an object, characterized by, The method comprises: an acquisition module configured to acquire a laser radar point cloud image collected by a laser radar, a visible light image collected by a visible light camera, and a thermal infrared image collected by an infrared camera; an extraction module configured to input the laser radar point cloud image into a preset backbone network, segment point cloud features based on a multi-scale point cloud segmentation network, extract point cloud features of the laser radar point cloud image, and fuse a dual-light fusion image according to the visible light image and the thermal infrared image; and an extraction module configured to input the laser radar point cloud image into a preset backbone network, segment point cloud features based on a multi-scale point cloud segmentation network, extract point cloud features of the laser radar point cloud image, and fuse a dual-light fusion image according to the visible light image and the thermal infrared image; The detection module is configured to extract target image features from the dual-light image, perform cross-modal fusion on the target image features and the point cloud features based on a preset bird's eye view (BEV) space, and perform target detection in the BEV space based on a cross-modal fusion result. The detection module is configured to extract 2D image features of the dual-light image based on a ResNet101 backbone network, convert the 2D image features of the dual-light fusion image to 3D point cloud features in the preset BEV space by using a BEVFormer network, and obtain the preset BEV space point cloud features; perform cross-modal fusion on the laser radar point cloud features and the preset BEV space point cloud features by using a preset fusion weighting algorithm, and obtain the cross-modal fusion result; and perform target detection in the BEV space based on the cross-modal fusion result.

5. The apparatus of claim 4, wherein, Before the laser radar point cloud image collected by the laser radar, the visible light image collected by the visible light camera, and the thermal infrared image collected by the infrared camera, the acquisition module is further configured to: According to a preset calibration strategy, calibrate the external parameters of the laser radar point cloud image, and calibrate the internal parameters of the visible light image and the thermal infrared image; Synchronize the time stamps of the visible light camera and the infrared camera.

6. The apparatus of claim 4, wherein, The extraction module is specifically configured to: Collect the converted visible light image and the converted thermal infrared image to generate a converted visible light image and a converted thermal infrared image; According to a preset transformation model, respectively register the converted visible light image and the converted thermal infrared image to obtain a registered visible light image and a registered thermal infrared image, wherein the preset transformation model is: ； wherein, and are coordinates under the visible light image and the infrared image, respectively, is a shift matrix; According to a preset dual-discriminator generative adversarial network, respectively fuse the high-resolution converted visible light image and the low-resolution registered thermal infrared image to obtain a dual-light fusion image.

7. An electronic device, comprising: The computer program is executed by the processor to implement the object detection method based on the object general feature according to any one of claims 1-3. The program is executed by the processor to implement the object detection method based on the object general feature according to any one of claims 1-3.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images