Target detection method, device and electronic equipment

Abstract

The application provides a target detection method, device and electronic equipment. The method comprises: acquiring a plurality of images and a plurality of point clouds of a target scene, wherein one image corresponds to one point cloud collected synchronously; projecting pixel points of each image to a three-dimensional space to form a projection point cloud with the space points projected by the pixel points of each image, and acquiring color information of points in each point cloud to generate a laser color point cloud; acquiring an aggregation of multi-time sequence feature bird's-eye views according to a high-density point cloud formed by the projection point cloud and the laser color point cloud; wherein the feature bird's-eye view has planar image information and three-dimensional spatial distribution information; determining and outputting detection information according to the aggregation of the multi-time sequence feature bird's-eye views; wherein the detection information represents object categories and object position information in the target scene in a current time period. The method can solve the problem of how to better integrate three-dimensional target detection and two-dimensional images to improve the accuracy of 3D target detection.

Description

Technical Field

[0001] This application relates to the fields of image processing and point cloud processing technology, and in particular to a target detection method, apparatus and electronic device. Background Technology

[0002] In the process of autonomous driving, in order to achieve path planning and collision avoidance in autonomous driving tasks, it is necessary to use three-dimensional (3D) object detection methods to perceive the external environment of the driving vehicle, so as to provide information on the category and location of objects in the driving environment.

[0003] The rapid development of computer vision has led to most existing object detection methods being based on two-dimensional (2D) images. However, 2D image-based object detection techniques have significant limitations. First, 2D images are captured by cameras, and although these images contain rich semantic information, the image quality is severely affected by lighting and weather conditions, as cameras are passive sensors. Second, 2D images cannot provide depth information, which is essential for path planning and collision avoidance in autonomous driving tasks. Therefore, 3D object detection is introduced to provide more detailed information on object size and location.

[0004] How to better integrate 3D object detection and 2D images to improve the accuracy of 3D object detection remains an urgent problem to be solved. Summary of the Invention

[0005] This application provides a target detection method, apparatus, and electronic device to address the problem of how to better integrate 3D target detection and 2D images to improve the accuracy of 3D target detection.

[0006] On the one hand, this application provides a target detection method, the method comprising:

[0007] Acquire multiple frames of images and multiple frames of point clouds of the target scene, where each frame of image corresponds to a frame of point cloud acquired synchronously;

[0008] The pixels of each frame image are projected into a three-dimensional space, and the spatial points of the projected pixels of each frame image are used to form a projected point cloud. In addition, the color information of the points in the point cloud of each frame is obtained to generate a laser color point cloud.

[0009] Based on the high-density point cloud composed of the projected point cloud and the laser color point cloud, an aggregation of multi-temporal feature bird's-eye view images is obtained; wherein, the feature bird's-eye view image has planar image information and three-dimensional spatial distribution information;

[0010] Based on the aggregation of the feature bird's-eye view images from multiple time series, detection information is determined and output; wherein, the detection information represents the object category and object location information in the target scene within the current time period.

[0011] In one embodiment, the aggregation of multi-temporal feature bird's-eye view images based on the high-density point cloud composed of the projected point cloud and the laser color point cloud includes:

[0012] The information vector of each point in the high-density point cloud is obtained; wherein the information vector includes two-dimensional coordinates, three-dimensional coordinates, three-channel grayscale values, color features and features of the high-density point cloud, wherein the features of the high-density point cloud include features of the structural information of the high-density point cloud, and the two-dimensional coordinates include the two-dimensional coordinates of the point in multiple viewpoint images with different perspectives contained in a frame image.

[0013] After the high-density point cloud is segmented into cylindrical voxels, the segmented spatial point cloud is voxel-encoded based on the information vector of each point in the segmented voxel spatial point cloud to obtain the voxelized point cloud.

[0014] Asymmetric convolution is performed on the voxelized point cloud to aggregate the information vectors of different points and obtain the aggregated information point cloud.

[0015] The aggregated point cloud is compressed along the height direction to obtain a multi-temporal feature bird's-eye view;

[0016] An image feature aggregation method based on deformable attention mechanism processes multi-temporal feature bird's-eye view images to obtain the aggregation of multi-temporal feature bird's-eye view images.

[0017] In one embodiment, the step of assembling a high-density point cloud based on the projected point cloud and the laser-colored point cloud includes:

[0018] An initial high-density point cloud is formed based on the projected point cloud and the laser-colored point cloud;

[0019] The depth information of the spatial points of the matching pixel projection of each frame image in the initial high-density point cloud is obtained to obtain the color point cloud in the projected point cloud. Here, a frame image contains multiple view images from different perspectives. The matching pixel is a pixel in one view image, and the matching pixel and a pixel in another view image have the same three-dimensional projection spatial point. The depth information of the matching pixel and the pixel is the depth information of the three-dimensional projection spatial point.

[0020] A dense point cloud is formed based on the color point cloud and the laser color point cloud;

[0021] Based on the dense point cloud, the depth information of the spatial points of the non-matching pixel projection in the initial high-density point cloud is estimated to obtain the depth information of the spatial points of the non-matching pixel projection in the initial high-density point cloud.

[0022] Based on the dense point cloud, the depth information of the spatial points of the non-matching pixel projection in the initial high-density point cloud is corrected to obtain the high-density point cloud.

[0023] In one embodiment, obtaining the depth information of matching pixels in each frame of the image includes:

[0024] For each frame of an image containing multiple viewpoints from different perspectives, the pixels in the viewpoints from different perspectives are matched to obtain each pair of matching pixels between every two viewpoints. A pair of matching pixels contains two matching pixels.

[0025] Obtain the same 3D projection space point corresponding to each pair of matching pixels in each frame of the image, and obtain the coordinates of the 3D projection space point in the coordinate system of each image acquisition device and the coordinates in the world coordinate system;

[0026] Based on the coordinates of the 3D projection spatial points in the coordinate system of each image acquisition device and in the world coordinate system, the depth information of the 3D projection spatial points in each viewpoint is determined, and the depth information of the 3D projection spatial points in each viewpoint is used as the depth information of the matching pixels in the corresponding viewpoint image.

[0027] In one embodiment, after acquiring multiple frames of images and multiple frames of point clouds of the target scene, the method further includes:

[0028] The process involves filtering out key frame images and non-key frame images from multiple frames; obtaining key frame point clouds corresponding to the key frame images in time sequence; and obtaining non-key frame point clouds corresponding to the non-key frame images in time sequence.

[0029] The step of projecting the pixels of each frame of an image into a three-dimensional space, and forming a projection point cloud from the spatial points projected from the pixels of each frame of the image, includes:

[0030] The pixels of each keyframe image are projected into three-dimensional space, and the pixels of each non-keyframe image are projected into three-dimensional space, forming a projection point cloud with the spatial points projected from the pixels of each keyframe image and the spatial points projected from the pixels of each non-keyframe image.

[0031] The step of obtaining the color information of points in each frame of the point cloud to generate a laser color point cloud includes:

[0032] The color information of each keyframe point cloud and the color information of each non-keyframe point cloud are obtained to generate a laser color point cloud, wherein the laser color point cloud includes keyframe color point clouds and non-keyframe color point clouds.

[0033] In one embodiment, after filtering out keyframe images and non-keyframe images from multiple frames, the method further includes:

[0034] The denoised image and edge texture images at different scales of the keyframe image are obtained. The denoised image is input into the image feature extraction network to obtain feature maps at each scale of the denoised image. The edge texture images at different scales are superimposed on the feature maps at each scale to obtain feature maps at each scale after edge texture enhancement processing.

[0035] Based on the optical flow between the keyframe image and the non-keyframe image, the features of the non-keyframe image are deduced.

[0036] After acquiring the keyframe point cloud corresponding to the keyframe image in time sequence, and acquiring the non-keyframe point cloud corresponding to the non-keyframe image in time sequence, the method further includes:

[0037] For the keyframe point cloud, keyframe point cloud features are extracted, and deformable attention feature aggregation is performed on the keyframe point cloud features to obtain aggregated keyframe point cloud features.

[0038] For the non-keyframe point cloud, the features of the non-keyframe point cloud are deduced based on the scene flow between the keyframe point cloud and the non-keyframe point cloud.

[0039] The aggregated feature point cloud of the high-density point cloud has the following features: non-keyframe image features, non-keyframe point cloud features, keyframe image features after image contour enhancement processing, and aggregated keyframe point cloud features.

[0040] In one embodiment, acquiring the denoised image and edge texture images at different scales of the keyframe image includes:

[0041] Wavelet decomposition was performed on the keyframe image to obtain multiple different wavelet components;

[0042] Different thresholds are set for different wavelet component maps, and threshold filtering is performed on each wavelet component according to the threshold corresponding to each wavelet component to obtain multiple wavelet components after noise reduction.

[0043] The denoised image is reconstructed from the multiple wavelet components after denoising, and edge texture images of different scales are reconstructed from the multiple wavelet components after denoising.

[0044] In one embodiment, reconstructing edge texture images of different scales based on multiple denoised wavelet components includes:

[0045] Initial edge texture images of different scales are reconstructed based on the denoised wavelet components;

[0046] Erosion and dilation operations are performed on the initial edge texture image at each scale to obtain edge texture images at different scales.

[0047] In one embodiment, before inputting the denoised image into an image feature extraction network to obtain feature maps of the denoised image at various scales, the method further includes:

[0048] The denoised image is converted into a grayscale image, and the grayscale image is divided into patches of the same size. The information entropy value of each patch is obtained.

[0049] Based on the information entropy value of each patch, the grayscale image is segmented to obtain multiple image regions.

[0050] Based on the information entropy value of each image region, each image region is arranged to the corresponding position in the feature extraction network. Image regions with information entropy values ​​greater than a preset information entropy value are arranged at positions that can pass through a first number of convolutional layers, and image regions with information entropy values ​​less than or equal to a preset information entropy value are arranged at positions that can pass through a second number of convolutional layers, where the first number is greater than the second number.

[0051] In one embodiment, obtaining multiple segmented image regions after segmenting the grayscale image based on the information entropy value of each patch includes:

[0052] For each patch, based on the information entropy value of the pixels in the patch, the proportion of pixels with the same information entropy value in each group is counted to obtain the information entropy distribution statistical histogram of the patch.

[0053] After applying Gaussian smoothing filtering to the statistical histogram of information entropy distribution, the peak points of the statistical histogram of information entropy distribution are identified.

[0054] Using the number of peak points as the number of clusters and the information entropy corresponding to the peak points as the cluster centers, fuzzy C-means clustering is performed on each pixel in the information entropy distribution statistical histogram. The information entropy values ​​of pixels belonging to the same category are updated to the information entropy values ​​of the cluster centers to which the pixels belong.

[0055] After the information entropy value is updated, the grayscale image is segmented based on the updated information entropy value to obtain multiple segmented image regions.

[0056] In one embodiment, each frame contains multiple viewpoint images from different perspectives, and the step of obtaining the color information of points in each frame's point cloud to generate a laser color point cloud includes:

[0057] Project each point in the point cloud of each frame onto the view images of the corresponding frame image from various perspectives;

[0058] When a point is projected onto an integer pixel, the color information of the integer pixel onto which the point is projected is obtained as the color information of the point.

[0059] When a point is projected onto a non-integer pixel, the color information of the non-integer pixel to which the point is projected is estimated, and the estimated color information is used as the color information of the point.

[0060] In one embodiment, the color information of the non-integer pixels to which the estimated point is projected includes:

[0061] Obtain the four nearest neighbors of the non-integer pixel being projected, and determine the weight coefficients of the four pixels based on the distance and grayscale distance between each of the four pixels and the non-integer pixel being projected.

[0062] Substitute the weight coefficients and color information of the four pixels into the Gaussian function to obtain the estimated color information of the non-integer pixels projected onto the system.

[0063] In one embodiment, after acquiring the multi-frame point cloud of the target scene, the method further includes:

[0064] Filter out ground point clouds from each frame of point cloud.

[0065] On the other hand, this application provides a target detection device, comprising:

[0066] The acquisition module is used to acquire multiple frames of images and multiple frames of point clouds of the target scene, wherein one frame of image corresponds to one frame of point cloud acquired synchronously;

[0067] The point cloud generation module is used to project the pixels of each frame image into a three-dimensional space, form a projected point cloud from the spatial points projected from the pixels of each frame image, and obtain the color information of the points in each frame point cloud to generate a laser color point cloud.

[0068] The acquisition module is further configured to acquire an aggregation of multi-temporal feature bird's-eye view images based on the high-density point cloud composed of the projected point cloud and the laser color point cloud; wherein the feature bird's-eye view image has planar image information and three-dimensional spatial distribution information;

[0069] The detection module is used to determine and output detection information based on the aggregation of the feature bird's-eye view images from multiple time series; wherein, the detection information represents the object category and object location information in the target scene within the current time period.

[0070] On the other hand, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0071] The memory stores computer-executed instructions;

[0072] The processor executes computer execution instructions stored in the memory to implement the method as described in the first aspect.

[0073] On the other hand, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed, cause a computer to perform the method described in the first aspect.

[0074] On the other hand, this application provides a computer program product comprising a computer program that, when executed by a processor, implements the method described in the first aspect.

[0075] In summary, embodiments of this application provide a target detection method, comprising: acquiring multiple frames of images and multiple frames of point clouds of a target scene, wherein each frame of image corresponds to a frame of point cloud acquired synchronously; projecting the pixels of each frame of image onto a three-dimensional space to form a projected point cloud using the spatial points projected from the pixels of each frame of image; and acquiring the color information of the points in each frame of point cloud to generate a laser-colored point cloud; acquiring an aggregation of multi-temporal feature bird's-eye view images based on the high-density point cloud composed of the projected point cloud and the laser-colored point cloud; wherein the feature bird's-eye view images have planar image information and three-dimensional spatial distribution information; determining and outputting detection information based on the aggregation of the multi-temporal feature bird's-eye view images; wherein the detection information characterizes the object category and object location information in the target scene within the current time period.

[0076] In other words, by projecting each pixel in multiple frames of images of the target scene into a 3D space, a projected point cloud with color information is generated. This projected point cloud carries not only rich color information but also rich edge texture information. For the multiple frame point clouds of the target scene, the color information of points in each frame is obtained to generate a laser-colored point cloud, which also carries rich color information. The high-density point cloud obtained by combining this projected point cloud and the laser-colored point cloud not only possesses rich color and edge texture information but also, due to the addition of the projected point cloud, eliminates the drawbacks of sparsity, disorder, and uneven spatial distribution, instead exhibiting the characteristics of a larger number of points, a more ordered and uniform point cloud distribution. Therefore, by obtaining an aggregation of multi-temporal feature bird's-eye views based on this high-density point cloud, and by determining and outputting the detection information based on this aggregation of multi-temporal feature bird's-eye views, the output detection information becomes more accurate. Attached Figure Description

[0077] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0078] Figure 1 A schematic diagram illustrating an application scenario of the target detection method provided in this application;

[0079] Figure 2 A schematic flowchart of a target detection method provided in one embodiment of this application;

[0080] Figure 3 A schematic diagram of pixel projection in three-dimensional space in a target detection method provided in an embodiment of this application;

[0081] Figure 4 A schematic diagram of the projection of matching pixels in three-dimensional space in a target detection method provided in an embodiment of this application;

[0082] Figure 5 A schematic diagram of a nonlinear solution method in a target detection method provided in one embodiment of this application;

[0083] Figure 6 A schematic diagram of cylindrical pixel segmentation of a point cloud in a target detection method provided in an embodiment of this application;

[0084] Figure 7 A schematic flowchart of a target detection method provided in yet another embodiment of this application;

[0085] Figure 8 A schematic flowchart of a target detection method provided in yet another embodiment of this application;

[0086] Figure 9 A schematic diagram of feature vectors in a target detection method provided in one embodiment of this application;

[0087] Figure 10 Another schematic diagram of the feature vector in the target detection method provided in one embodiment of this application;

[0088] Figure 11 Another schematic diagram of the feature vector in the target detection method provided in one embodiment of this application;

[0089] Figure 12 Another schematic diagram of the feature vector in the target detection method provided in one embodiment of this application;

[0090] Figure 13 Another schematic diagram of the feature vector in the target detection method provided in one embodiment of this application;

[0091] Figure 14 A schematic diagram of a target detection device provided in one embodiment of this application;

[0092] Figure 15 A schematic diagram of an electronic device provided for one embodiment of this application.

[0093] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0094] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0095] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0096] First, let me explain the terms used in this application:

[0097] Point cloud: refers to a set of vectors in a three-dimensional coordinate system. Scanned data is recorded in the form of points, containing at least one point data. Each point data packet contains three-dimensional coordinates, and some point data may contain color information or reflectivity.

[0098] Self-attention mechanism: This is a configuration of artificial intelligence network based on the attention mechanism. Self-attention has no learnable parameters.

[0099] Deformable attention mechanism: This is a configuration of artificial intelligence network based on attention mechanism. The parameters of the deformable attention mechanism can be obtained by learning from the input data, and thus, the parameters of the deformable attention mechanism are adjusted as the input data changes.

[0100] Autonomous driving can alleviate driver fatigue, reduce traffic accidents, optimize travel routes, reduce traffic congestion, and promote the rational use of energy and reduction of air pollution. Therefore, it has received widespread attention from industry and academia both domestically and internationally. Accurate perception of the surrounding environment is a necessary prerequisite for the reliable operation of autonomous driving. Autonomous driving environmental perception systems typically employ various methods to extract semantic information from the data collected by sensors, thereby achieving autonomous driving. Object detection, as a fundamental component of environmental perception, has also become a popular research direction.

[0101] The rapid development of computer vision has led to most existing object detection methods being based on two-dimensional (2D) images. However, 2D image-based object detection techniques have significant shortcomings. First, 2D images are captured by cameras. Although these images contain rich semantic information, the image quality is severely affected by lighting and weather conditions, as cameras are passive sensors. Second, 2D images cannot provide depth information, which is essential for path planning and collision avoidance in autonomous driving tasks. Therefore, introducing three-dimensional (3D) object detection methods can provide more detailed information on object size and location. However, 3D point clouds acquired by radar do not contain color, texture, or other information, and suffer from sparsity, disorder, and uneven spatial distribution, posing a challenge to accurate object detection based on 3D point clouds.

[0102] Based on this, this application provides a target detection method, apparatus, and electronic device. The target detection method includes: acquiring multiple frames of images and multiple frames of point clouds of a target scene, wherein each frame of image corresponds to a synchronously acquired frame of point cloud; projecting the pixels of each frame of image onto a three-dimensional space to form a projected point cloud; and acquiring the color information of points in each frame of point cloud to generate a laser-colored point cloud; acquiring an aggregation of multi-temporal feature bird's-eye views based on the high-density point cloud composed of the projected point cloud and the laser-colored point cloud; wherein the feature bird's-eye view has planar image information and three-dimensional spatial distribution information; determining and outputting detection information based on the aggregation of the multi-temporal feature bird's-eye views; wherein the detection information represents the object category and object location information in the target scene within the current time period.

[0103] In other words, by projecting each pixel in multiple frames of images of the target scene into a 3D space, a projected point cloud with color information is generated. This projected point cloud carries not only rich color information but also rich edge texture information. For the multiple frame point clouds of the target scene, the color information of points in each frame is obtained to generate a laser-colored point cloud, which also carries rich color information. The high-density point cloud obtained by combining this projected point cloud and the laser-colored point cloud not only possesses rich color and edge texture information but also, due to the addition of the projected point cloud, eliminates the drawbacks of sparsity, disorder, and uneven spatial distribution, instead exhibiting the characteristics of a larger number of points, a more ordered and uniform point cloud distribution. Therefore, by obtaining an aggregation of multi-temporal feature bird's-eye views based on this high-density point cloud, and by determining and outputting the detection information based on this aggregation of multi-temporal feature bird's-eye views, the output detection information becomes more accurate.

[0104] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0105] It should be noted that the target detection method, apparatus and electronic equipment provided in this application can be used in the fields of image processing and point cloud processing technology, and can also be used in any field other than the fields of image processing and point cloud processing technology. The application fields of the target detection method, apparatus and electronic equipment in this application are not limited.

[0106] The target detection method provided in this application is applied to electronic devices, such as processors or cloud servers equipped with autonomous driving systems. Please see [link to application]. Figure 1This diagram illustrates the application of the target detection method provided in this application. The electronic device acquires multiple frames of images and point clouds of the target scene, where each image corresponds to a synchronously acquired point cloud. The pixels of each image are projected into a three-dimensional space to form a projected point cloud. Color information of points in each point cloud is also acquired to generate a laser-colored point cloud. Based on the high-density point cloud composed of the projected point cloud and the laser-colored point cloud, an aggregation of multi-temporal feature bird's-eye views is obtained; this feature bird's-eye view contains planar image information and three-dimensional spatial distribution information. Based on the aggregation of the multi-temporal feature bird's-eye views, detection information is determined and output; this detection information represents the object category and object location information in the target scene within the current time period.

[0107] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0108] Please see Figure 2 One embodiment of this application provides a target detection method, including:

[0109] S210, acquire multiple frames of images and multiple frames of point clouds of the target scene, wherein one frame of image corresponds to one frame of point cloud acquired synchronously.

[0110] During autonomous driving, image acquisition devices, such as cameras, can capture multiple frames of images of the target scene (which can also be understood as the driving environment) within a given time period. Simultaneously, point cloud acquisition devices, such as LiDAR, can capture multiple frames of point clouds within the target scene within the same time period. The point cloud represents the three-dimensional geometric information of the target scene.

[0111] The electronic device acquires at least one frame of the target scene to be detected within the current time period (e.g., within 1 second or 5 seconds) from the image acquisition device, and at least one frame of the point cloud of the target scene to be detected within the current time period (e.g., within 1 second or 5 seconds) from the point cloud acquisition device. Each frame of the image acquired by the electronic device corresponds one-to-one with each frame of the point cloud, and there is spatial and temporal synchronization between each frame of the image acquired by the electronic device and each frame of the point cloud.

[0112] In an optional embodiment, after acquiring multiple frames of point clouds of the target scene, ground point clouds in each frame are filtered out. When filtering ground point clouds, the GPF_Ransac method can be used to remove ground points from the point cloud. Since the ground point cloud is large in number but does not contain the target, filtering out the ground point cloud can effectively improve the computational efficiency of target detection and achieve real-time target detection.

[0113] S220 projects the pixels of each frame image into a three-dimensional space, forms a projection point cloud with the spatial points of the pixel projection of each frame image, and obtains the color information of the points in each frame point cloud to generate a laser color point cloud.

[0114] After projecting the pixels of a frame of image into three-dimensional space, each spatial point in the projected point cloud carries information including three-dimensional coordinates, three-channel grayscale values, and the three primary color information of the color image. In this embodiment, each frame of image contains multiple viewpoint images from different perspectives. Therefore, the two-dimensional coordinates carried by each point in the projected point cloud include the two-dimensional coordinates corresponding to each viewpoint image. For example, if a frame of image contains six viewpoint images from six different perspectives, then the three-dimensional spatial point corresponding to one pixel in that frame of image has six two-dimensional coordinates.

[0115] When acquiring the color information of points in a point cloud frame, the points in the point cloud frame are mapped to pixels in a synchronized image frame (which contains images from multiple viewpoints). The color information of the points in the point cloud is then determined based on the color information of the pixels. In addition, after mapping points in the point cloud to pixels, the two-dimensional coordinates of the pixels can also be obtained. The information carried by points in a laser color point cloud includes two-dimensional coordinate information, color information, three-dimensional coordinate information, and depth information (i.e., the z-coordinate depth = z in the coordinate system of each viewpoint camera). Together, these form a vector (u, v, i) representing the information of each point in the laser color point cloud. R (u, v), I G (u, v), I B (u, v), F(u, v), x, y, z). The laser-colored point cloud can be represented by its aggregated point cloud features obtained through point cloud feature extraction and aggregation methods. The information vector of the laser-colored point cloud can then be expressed as (u, v, I). R (u, v), I G (u, v), I B (u, v), F(u, v), x, y, z). Here, the two-dimensional coordinates carried by each point in the laser color point cloud contain the two-dimensional coordinates corresponding to each viewpoint in the corresponding frame image. For example, if a point in the point cloud corresponds to a pixel in a frame image, and that frame image contains six different viewpoints, then a point in the point cloud corresponds to six two-dimensional coordinates.

[0116] In an optional embodiment, each frame contains multiple viewpoint images from different perspectives. When generating the laser color point cloud, each point in each frame is projected onto the viewpoint images of the corresponding frame, and the color information of the point is determined based on the projected pixels. That is, when a point is projected onto an integer pixel, the color information of the projected integer pixel is obtained as the color information of the point. When a point is projected onto a non-integer pixel, the color information of the non-integer pixel is estimated, and the estimated color information is used as the color information of the point.

[0117] Specifically, the two-dimensional coordinates of the point cloud acquired by the lidar on images from various viewpoints can be obtained through the projection matrix. The information vectors of each point in the point cloud acquired by the lidar can then be divided into two categories. For point cloud points projected to integer pixels, their information vectors are (u, v, i). R (u, v), I G (u, v), I B (u, v), F(u, v), x, y, z). For point cloud points projected onto non-integer pixels, their color information is unknown, and their information vector is (u, v, x, y, z). Therefore, it is necessary to estimate the color information of the corresponding points.

[0118] In an optional embodiment, the step of estimating the color information of the non-integer pixels to which the point is projected includes:

[0119] Step 1: Obtain the four nearest neighbors of the non-integer pixel to which the projection is made, and determine the weight coefficients of the four pixels based on the distance and grayscale distance between each of the four pixels and the non-integer pixel to which the projection is made.

[0120] Step 2: Substitute the weight coefficients and color information of the four pixels into the Gaussian function to obtain the estimated color information of the non-integer pixels projected onto the Gaussian function.

[0121] The following section describes in detail how to estimate the color information of non-integer pixels projected onto a point.

[0122] When estimating the color information of non-integer pixels projected onto a point, Gaussian filtering is employed. Specifically, the four nearest pixels to the non-integer pixel whose color information is to be estimated (where u and v are not both integers, hereinafter referred to as the pixel to be estimated) are identified. Using a Gaussian function, the weighting coefficients for these four pixels are determined based on their positional distance and grayscale distance to the pixel to be estimated. The closer a pixel is to the pixel to be estimated, the greater its influence on the grayscale of the pixel to be estimated, and the larger its corresponding weighting coefficient. In the color estimation process for laser point clouds, information such as the two-dimensional coordinates, corresponding three-dimensional coordinates, color, color features, and corresponding spatial point features of the corresponding image pixel are considered simultaneously. This consideration of both two-dimensional and three-dimensional information makes the estimated laser point cloud color more accurate. Furthermore, the Gaussian function is used to implement distance-based weighting coefficient calculation (the coefficients exhibit a gradual change from near to far according to the Gaussian function).

[0123] Specifically, the formula for estimating the color information of the pixels to be estimated is provided below:

[0124] Formula (1):

[0125] Formula (2):

[0126] Formula (3):

[0127] Formula (4):

[0128]

[0129]

[0130]

[0131] In the above formula, I R (i,j) represents the grayscale value of the red component of an integer pixel (i,j) in the image. The grayscale information of integer pixels like (i,j) is fixed, while the grayscale information of non-integer pixels like (u,v) is obtained through training. This represents the red grayscale value of the pixel to be estimated (u,v) after t training iterations, and is calculated by comparing the red grayscale value of the nearest image pixel to the pixel to be estimated (u,v). Initialize the data. (i,j) represents the coordinates of each of the four closest pixels to the pixel to be estimated, where (i,j)∈KNEAR(u,v). MLP stands for Multilayer Perceptron.

[0132] w(i,j) represents the weighted coefficients related to the 2D coordinate distance, 2D color-to-grayscale distance, 2D image feature distance, 3D coordinate distance (obtained through camera parameter projection), and 3D point cloud feature distance. Specifically, w(i,j) is the coefficient obtained by combining wposition2d, wcgl(i,j), wfeature2d, wposition3d, and wfeature3d(i,j). Here, wposition2d represents the weighted coefficient related to the 2D coordinate distance, wcgl(i,j) represents the weighted coefficient related to the 2D color-to-grayscale distance, wfeature2d represents the weighted coefficient related to the 2D image feature distance, wposition3d represents the weighted coefficient related to the 3D coordinate distance (obtained through camera parameter projection), and wfeature3d(i,j) represents the weighted coefficient related to the 3D point cloud feature distance.

[0133] δ represents the standard deviation of the normal distribution of the Gaussian function. The larger δ is, the smaller the difference in the influence of different distances (spatial distance / color / grayscale distance) on the estimated pixel; the smaller δ is, the greater the difference in the influence of different distances on the estimated pixel. δ is the parameter to be learned.

[0134] It should be noted that wposition3d(i,j) represents a weighting coefficient related to the distance to the 3D point cloud coordinates. Through the previous processing, only the spatial coordinate information of some pixels in the image from each viewpoint can be obtained. Therefore, in calculating wposition3d(i,j), it is necessary to consider whether the spatial coordinate information corresponding to pixel (i,j) is known.

[0135] pos3dx(i,j), pos3dy(i,j), and pos3dz(i,j) represent the x, y, and z coordinates of the 3D point corresponding to pixel (i,j), respectively.

[0136] fea3d represents a 3D feature. It is also necessary to determine whether the corresponding 3D coordinates exist.

[0137] Substituting the known parameters into the formulas, and based on formula (1), the following can be determined: The value of .

[0138] It should be noted that the above describes the grayscale estimation method for the red component of non-integer pixels (u,v) (i.e., u and v are not both integers). The grayscale estimation methods for the green component and the blue component are the same as those for the red component, and will not be repeated here.

[0139] Thus, the color information of non-integer pixels is estimated, and the estimation result of the color information of the pixel to be estimated is obtained.

[0140] S230, based on the high-density point cloud composed of the projected point cloud and the laser color point cloud, obtain the aggregation of multi-temporal feature bird's-eye view images; wherein, the feature bird's-eye view image has planar image information and three-dimensional spatial distribution information.

[0141] In an optional embodiment, the step of assembling a high-density point cloud based on the projected point cloud and the laser-colored point cloud includes:

[0142] Step 1: Based on the projected point cloud and the laser-colored point cloud, form an initial high-density point cloud.

[0143] The depth information of the midpoints in the initial high-density point cloud includes the depth information of the midpoints in the projected point cloud and the depth information of the midpoints in the laser color point cloud. The depth information of the midpoints in the laser color point cloud is required, while the depth information of the midpoints in the projected point cloud is unknown and needs to be estimated.

[0144] The points in the projected point cloud include spatial points with and without matching pixel projections, as described below. Correspondingly, when estimating the depth information of the points in the projected point cloud, it is necessary to estimate not only the depth information of the spatial points with matching pixel projections but also the depth information of the spatial points without matching pixel projections.

[0145] Step 2: Obtain the depth information of the spatial points of the matching pixel projection in each frame of the initial high-density point cloud to obtain the color point cloud in the projected point cloud. Here, a frame of image contains multiple viewpoint images from different perspectives. The matching pixel is a pixel in one viewpoint image, and the matching pixel and a pixel in another viewpoint image share the same three-dimensional projection spatial point. The depth information of the matching pixel and the pixel is the depth information of the three-dimensional projection spatial point.

[0146] This step describes how to estimate the depth information of the spatial points projected by the matching pixels, that is, how to obtain the depth information of the matching pixels in each frame of the image.

[0147] In an optional embodiment, obtaining the depth information of matching pixels in each frame image includes: for multiple different viewpoint images contained in each frame image, matching the pixels in the viewpoint images from different viewpoints to obtain each pair of matching pixels between every two viewpoint images, wherein a pair of matching pixels contains two matching pixels; obtaining the same three-dimensional projection space point corresponding to each pair of matching pixels in each frame image, obtaining the coordinates of the three-dimensional projection space point in the coordinate system of each image acquisition device and the coordinates in the world coordinate system; determining the depth information of the three-dimensional projection space point in each viewpoint based on the coordinates of the three-dimensional projection space point in the coordinate system of each image acquisition device and the coordinates in the world coordinate system, and using the depth information of the three-dimensional projection space point in each viewpoint as the depth information of the matching pixels in the corresponding viewpoint image.

[0148] For an example using a single image frame, please refer to [link / reference]. Figure 3 Given the pixel coordinates (u, v) of a point p in an image, the location of the corresponding spatial point cannot be determined. This is because, according to the camera imaging principle, a point p on the image plane could be the image of any point on the ray pP. Therefore, for a single-view image, it is impossible to determine the coordinates of the spatial points corresponding to each pixel in the image. However, if a frame of an image (an image of the same scene) contains multiple viewpoint images from different perspectives, then the coordinates of the spatial points corresponding to the pixels in the viewpoint images can be determined based on the different viewpoint images.

[0149] Please see Figure 4 By matching two pixels p and p′ in images from different viewpoints, the spatial point P corresponding to these two pixels in the image plane can be determined.

[0150] One specific method for determining this is the nonlinear solution method. Please refer to [link / reference]. Figure 5 The diagram shown illustrates a nonlinear solution. Due to noise and other factors, the spatial point P may not be directly identifiable. In such cases, an optimal P is required. * Point. Given the camera projection matrix M and the known spatial point P. * Projected onto two image planes. This makes MP * It is closest to point p. And M′P * The nearest P to p′ * This is the point closest to P that is required, and it can be used as an approximation of P. The nonlinear solution formula is as follows:

[0151] The above method allows us to obtain the matching pixel pairs p and p′ from each viewpoint, along with the coordinates of the corresponding spatial point P in both the camera coordinate system and the world coordinate system. This enables the projection of pixels from the image plane into three-dimensional space and determines the image depth. Image depth is the z-axis coordinate of a spatial point in the camera coordinate system of each viewpoint. After obtaining the depth information of the matching pixels in each frame, we can obtain the color point cloud in the projected point cloud.

[0152] Step 3: Based on the color point cloud and the laser color point cloud, form a dense point cloud.

[0153] As can be seen from the above, the depth information of each point in the dense point cloud is known.

[0154] Step 4: Based on the dense point cloud, estimate the depth information of the spatial points of the non-matching pixel projections in the initial high-density point cloud to obtain the depth information of the spatial points of the non-matching pixel projections in the initial high-density point cloud.

[0155] After estimating the depth information of the spatial points projected by the dense point cloud, the depth information of the initial height point cloud is then corrected based on the dense point cloud.

[0156] As mentioned above, the projected point cloud also contains spatial points projected from these non-matching pixels, and the depth information of these spatial points also needs to be estimated. Therefore, it is necessary to estimate the depth information of the spatial points projected from the non-matching pixels in the initial high-density point cloud based on this dense point cloud.

[0157] Estimating the depth information of spatial points projected from mismatched pixels includes:

[0158] Step 1: Establish information vectors for pixels (non-integer pixels) with unknown spatial information in images from various viewpoints:

[0159] (ur, vr, I) R (ur, vr), I G (ur, vr), I B (ur, vr), F(ur, vr),

[0160] The second step is to learn the local structural features of the initial high-density point cloud based on its 3D coordinates using a combination of Principal Component Analysis (PCA) and PointNet.

[0161] Step 3: The local structural features H(xhd,yhd,zhd) of the initial high-density point cloud are...

[0162] [Fstructure1(xhd,yhd,zhd)⊙Fstructure2(xhd,yhd,zhd)] is concatenated with the information vector of the initial density point cloud to form the feature vector of the initial high-density point cloud. That is, the feature vector of each point in the initial high-density point cloud is (uhd, vhd, ihd, zhd). R (uhd, vhd), I G (uhd, vhd), I B (uhd, vhd), F(uhd, vhd),

[0163] Step 4: A feature aggregation network based on a deformable attention mechanism is used to aggregate the feature vectors of the initial high-density point cloud, obtaining the aggregated feature pcfh for each point. The aggregated feature pcfh is then input into a multilayer perceptron to predict the depth of each point in the initial high-density point cloud. That is, D... pre (uhd, vhd) = MLP(pcfh).

[0164] Thus, the depth information of each point (including spatial points of non-matching pixel projections) in the initial high-density point cloud was obtained.

[0165] Step 5: Based on this dense point cloud, correct the depth information of the spatial points of the mismatched pixel projections in the initial high-density point cloud to obtain the high-density point cloud.

[0166] Specifically, based on the formula L(D) pre )=‖(D pre -D gt )⊙I(D gt >0)‖ 2 The depth information of spatial points with mismatched pixel projections estimated in the initial high-density point cloud is corrected. Wherein, D pre It is the estimated depth information, D gt This is the true depth (i.e., the depth of the dense point cloud) used to supervise the estimated depth. Since only a subset of points in each viewpoint image (i.e., image pixels and matching pixels obtained by projecting points in the laser point cloud onto each viewpoint image through camera intrinsic and extrinsic parameters) can obtain relatively accurate depth information through previous processing, therefore D... gt Only those pixels with valid depth values, i.e., D, are considered. gt Only the depth information of the points in the dense point cloud is substituted.

[0167] I() is a supervisory function for D gt For points I = 1 where > 0, for D gt <0, I=0. ⊙ represents element-wise multiplication. L(Dpre ) is the depth prediction loss function. The loss function is used to learn and adjust the corresponding parameters, making the learned features suitable for depth estimation and object detection.

[0168] The following describes how to obtain multi-temporal feature bird's-eye view images based on this high-density point cloud.

[0169] In an optional embodiment, obtaining a multi-temporal feature bird's-eye view based on the high-density point cloud includes:

[0170] Step 1: Obtain the information vector of each point in the high-density point cloud; the information vector includes two-dimensional coordinates, three-dimensional coordinates, three-channel grayscale values, color features, and features of the high-density point cloud, wherein the features of the high-density point cloud include structural information features of the high-density point cloud, and the two-dimensional coordinates include the two-dimensional coordinates of the point in multiple different viewpoint images contained in a frame of image.

[0171] After obtaining the high-density point cloud through the above steps, the information vector of each point in the high-density point cloud can also be obtained. Specifically, the information vector of each point includes two-dimensional coordinates, three-dimensional coordinates, three-channel grayscale values, color features, and the features of the high-density point cloud. That is, the information vector is: (uhd, vhd, I R (uhd, vhd), I G (uhd, vhd), I B (uhd, vhd), F(uhd, vhd), xhd, yhd, zhd, PCF(xhd, yhd, zhd)), where uhd and vhd represent two-dimensional coordinates, I R (uhd, vhd), I G (uhd, vhd), I B (uhd, vhd) represents the three-channel grayscale values, xhd, yhd, zhd represent the three-dimensional coordinates, and F(uhd, vhd) and PCF(xhd, yhd, zhd) represent the color features and the features of this high-density point cloud.

[0172] Step 2: After segmenting the high-density point cloud into cylindrical voxels, voxel encoding is performed on the segmented point cloud based on the information vector of each point in the segmented voxel spatial point cloud to obtain the voxelized point cloud.

[0173] The distribution of this high-density point cloud is characterized by denser distribution in the near and sparser distribution in the far. The use of cylindrical voxelization is well adapted to the distribution characteristics of the point cloud.

[0174] The cylindrical voxel segmentation of this high-density point cloud includes the following steps:

[0175] The first step is to map the high-density point cloud from the Cartesian coordinate system to the cylindrical coordinate system.

[0176] Specifically, through the formula Implement coordinate mapping.

[0177] in,

[0178] The second step is to segment the point cloud space using Δρ, Δθ, and Δz as intervals. For example... Figure 6 The diagram shown is a schematic of cylindrical pixel segmentation of a point cloud.

[0179] The third step is to use PointNet to encode the segmented spatial point clouds into voxels, so that the multi-point information contained in each spatial point cloud is aggregated into a single voxel, thus obtaining the voxelized point cloud.

[0180] Thus, cylindrical voxel segmentation of the high-density point cloud was completed. Compared with cubic voxelization, cylindrical voxelization is more suitable for the near-dense and far-sparse distribution characteristics of point clouds, which can effectively improve the non-empty rate of point cloud voxels and help improve the accuracy of target detection.

[0181] Step 3: Perform asymmetric convolution on the voxelized point cloud to aggregate the information vectors of different points and obtain the aggregated information point cloud.

[0182] Correspondingly, the output is an aggregation of the voxel information described above, i.e., the output aggregated voxel information. Compared with conventional sparse convolution, asymmetric convolution enhances the response in the horizontal and vertical directions and improves the representation ability of point clouds.

[0183] Step four: Compress the aggregated point cloud along the height direction to obtain a multi-temporal feature bird's-eye view.

[0184] Step 5: Based on the deformable attention mechanism, the image feature aggregation method processes the multi-temporal feature bird's-eye view to obtain the aggregation of the multi-temporal feature bird's-eye view.

[0185] Information is aggregated from bird's-eye view images at different times based on a deformable attention mechanism. The aggregated point cloud is compressed along the height direction to obtain feature bird's-eye view images for each frame, and then processed using an image feature aggregation method based on the deformable attention mechanism. This allows for the aggregation of bird's-eye view images with features from different time periods, and the resulting aggregated features are beneficial for improving the accuracy of target detection.

[0186] For the same bird's-eye view, given an input bird's-eye view x∈R C×R×Φ Where C represents the information dimension, R represents the radius of the information bird's-eye view, and Φ represents the angle of the information bird's-eye view. Let q be the query information z. q The serial number, p q To be with z q The corresponding two-dimensional coordinates, calculated using the information aggregation formula based on the deformable attention mechanism, are as follows:

[0187]

[0188] in,

[0189]

[0190] ξmqk=MLP(pq⊙pmk⊙Δpmqk⊙‖Δpmqk‖)

[0191]

[0192] This embodiment employs a multi-head attention mechanism, where m represents the attention head number, M represents the total number of attention heads (M=8 in this embodiment), k represents the sequence number of key information points sampled around the query information point, and K represents the total number of surrounding information points followed by each point (K=4 in this embodiment). This means that only the four key information points most relevant to the query information zq are aggregated with zq. Δpmqk represents the sampling offset, which is the offset of the sampled key information point coordinates relative to the coordinate pq corresponding to the query information point zq. Δpmqk is learned based on the query information point zq through a fully connected layer (FullConnection, FC). For cases where the offset is not an integer, the features of the relevant information point are obtained by interpolation using the features of the four feature points closest to its position. The expression βmqk(ξmqk,ηmqk) represents the correlation weighting coefficients of key feature points related to the query feature point. MLP stands for Multilayer Perceptron (MLP). F DA This is an aggregated feature based on a deformable attention mechanism.

[0193] In this embodiment, the correlation weighting coefficient is a function of the coordinates and features of the query feature point, the relevant feature points, and the Euclidean distance between their coordinates and features. Compared to previous feature extraction methods where the learned correlation weighting coefficient does not change with the input sample after model training, in this embodiment, the correlation weighting coefficient changes with the input sample after model training because it is a function of the input sample's coordinates and features; therefore, the correlation weighting coefficient has greater flexibility. This results in a more adaptive correlation weighting coefficient, and the features aggregated through this method have a higher correlation with the input sample, thus helping to improve the accuracy of target detection.

[0194] For bird's-eye view images at different times, set Given a set of different time-series bird's-eye view images as input. Let l represent the bird's-eye view at time l. The calculation formula for multi-scale feature aggregation based on the deformable attention mechanism is:

[0195]

[0196]

[0197] ξmtqk=MLP(Pq⊙pmtk⊙Δpmtqk⊙‖Δpmtqk‖)

[0198]

[0199] t represents the time series, and T represents the total number of time series, typically T = 4. Other coefficients have the same meaning as above; the t in the subscript is used to distinguish different time series bird's-eye view diagrams. F MSDA This is an aggregated information bird's-eye view based on a deformable attention mechanism, outputting the aggregated feature F. MSDA This allows for the aggregation of feature bird's-eye view images across multiple time series.

[0200] S240, Based on the aggregation of the multi-time-series bird's-eye view of this feature, determine and output detection information; wherein, the detection information represents the object category and object location information in the target scene within the current time period.

[0201] By aggregating bird's-eye view images with different temporal features and inputting them into different task heads, various tasks can be accomplished, satisfying specific objectives such as 3D object detection or map segmentation. For example, aggregating bird's-eye view images with different temporal features and inputting them into a single-stage or two-stage 3D object detection head (single-stage object detection heads such as the YOLO series, two-stage object detection heads such as the R-CNN series) can achieve 3D object detection. Aggregating bird's-eye view images with different temporal features and inputting them into an image segmentation head can achieve map segmentation.

[0202] In summary, this embodiment provides a target detection method, including: acquiring multiple frames of images and multiple frames of point clouds of a target scene, wherein each frame of image corresponds to a frame of point cloud acquired synchronously; projecting the pixels of each frame of image onto a three-dimensional space to form a projected point cloud using the spatial points projected from the pixels of each frame of image; and acquiring the color information of the points in each frame of point cloud to generate a laser-colored point cloud; acquiring an aggregation of multi-temporal feature bird's-eye view images based on the high-density point cloud composed of the projected point cloud and the laser-colored point cloud; wherein the feature bird's-eye view images have planar image information and three-dimensional spatial distribution information; determining and outputting detection information based on the aggregation of the multi-temporal feature bird's-eye view images; wherein the detection information represents the object category and object position information in the target scene within the current time period.

[0203] In other words, by projecting each pixel in multiple frames of images of the target scene into a 3D space, a projected point cloud with color information is generated. This projected point cloud carries not only rich color information but also rich edge texture information. For the multiple frame point clouds of the target scene, the color information of points in each frame is obtained to generate a laser-colored point cloud, which also carries rich color information. The high-density point cloud obtained by combining this projected point cloud and the laser-colored point cloud not only possesses rich color and edge texture information but also, due to the addition of the projected point cloud, eliminates the drawbacks of sparsity, disorder, and uneven spatial distribution, instead exhibiting the characteristics of a larger number of points, a more ordered and uniform point cloud distribution. Therefore, by obtaining an aggregation of multi-temporal feature bird's-eye views based on this high-density point cloud, and by determining and outputting the detection information based on this aggregation of multi-temporal feature bird's-eye views, the output detection information becomes more accurate.

[0204] Please see Figure 7 Another embodiment of this application provides a target detection method, including:

[0205] S710 acquires multiple frames of images and multiple frames of point clouds of the target scene, where each frame of image corresponds to a frame of point cloud acquired synchronously.

[0206] The description of this step can be found in the relevant description in step S210, and will not be repeated here.

[0207] S720, filter out key frame images and non-key frame images from multiple frames; and obtain key frame point clouds corresponding to the time sequence of the key frame image, and obtain non-key frame point clouds corresponding to the time sequence of the non-key frame image.

[0208] In actual data acquisition, situations with a large amount of similar data are often encountered. High data similarity introduces a significant amount of redundant information. The computational value of redundant information is limited, and processing it is computationally intensive, resulting in low target detection efficiency. To reduce the processing of redundant data and improve network efficiency, keyframe images can be selected from multiple acquired images. It should be noted that keyframe image selection applies to images acquired by the same camera.

[0209] This embodiment provides two methods for filtering keyframe images. The first method is a keyframe image filtering method based on Euclidean distance, and the second method is a keyframe image filtering method based on image correlation coefficient.

[0210] The two methods will be described in detail below.

[0211] The first method is a keyframe image filtering method based on Euclidean distance.

[0212] First, perform dimensionality reduction on each frame of the image obtained in step S710 based on PCA, and then screen out the key frame images according to the distances between the dimensionality reduction vectors of each frame of the image.

[0213] Perform dimensionality reduction on each frame of the obtained image based on PCA. The aim is to project high-dimensional data into low-dimensional data while retaining as much of the main information of the high-dimensional data as possible. The specific steps include: The first step is to straighten the pictures. Input the picture set (i.e., multiple frames of images) I = {I1, I2…, I m}, and straighten each two-dimensional picture matrix in the picture set into a one-dimensional vector to obtain X is a matrix with m rows and n columns. Here, m represents the total number of pictures (i.e., the number of samples is m), and n represents the total number of pixels in each picture (i.e., each sample dimension is n). The purpose of PCA dimensionality reduction is to reduce the dimension of each sample (n dimensions) to a smaller dimension (k dimensions). Here, k < n. The second step is to center the samples and calculate the sample mean For each row in X, that is, each sample Subtract the sample mean To obtain the centered matrix Matrix B is a matrix with a mean of 0. The third step is to calculate the covariance matrix of the centered matrix The fourth step is to calculate the eigenvalues λ1 ≥ λ2… ≥ λ n ≥ 0 and the corresponding eigenvectors of each eigenvalue The fifth step is to select the first k eigenvalues λ1 ≥ λ2… ≥ λ n The corresponding eigenvectors To form the transformation matrix T n×k and then obtain the set of dimensionality-reduced image vectors That is, originally an image was represented by an n-dimensional vector, and after PCA dimensionality reduction, each image is represented by a k-dimensional vector.

[0214] Screen out the key frame images according to the distances between the dimensionality reduction vectors of each frame of the image. Since in the input image dataset, pictures with similar time sequences have high similarity, there is a lot of redundant information, which makes the entire process of image processing time-consuming. Therefore, it is necessary to minimize the processing of redundant information. Here, similarity is used as the evaluation index (i.e., the picture with a smaller similarity between the current frame and the adjacent previous key frame is used as the key frame) to screen out the key frames. The specific steps include: The first step is to initialize the key frames. Set the first frame as the initial key frame, I ki represents the i-th key frame image. The second step is to calculate the Euclidean distance between the subsequent frames and the key frames sequentially starting from the second frame, Set the threshold D thWhen the calculated D > D th When the distance is large and the similarity is small, the i-th frame is designated as a keyframe. The third step is to continue in this manner, selecting the r-th keyframe I. kr At that time, starting from the frame following the (r-1)th keyframe, calculate the Euclidean distance between each subsequent frame and the (r-1)th keyframe. Set threshold D th When the calculated D > D th When the distance is large and the similarity is small, the j-th frame is selected as the r-th keyframe. Fourth, repeat step three until all images have been filtered. This yields a set of dimensionality-reduced vectors of the keyframe images. and the set of image keyframes {I k1 ,I k2 …,I ks}, where s represents the total number of keyframe images.

[0215] The second method is a keyframe image selection method based on image correlation coefficients.

[0216] Each frame is a color image. For each frame, the color image is first converted to grayscale to obtain a grayscale image. Then, the grayscale distribution histogram of the resulting grayscale image is calculated. Finally, keyframes are selected based on the similarity of the grayscale distribution histograms of the corresponding images. The specific steps include: First, converting the input color image to grayscale and calculating the grayscale distribution histogram of the grayscale image. The image histogram set is obtained based on the gray-level distribution histogram of each frame. The second step is keyframe initialization, setting the first frame as the keyframe. The third step, following the same logic, involves calculating the correlation coefficients between subsequent frames and keyframes, starting from the frame following the (r-1)th keyframe. Set threshold ρ th When the calculated ρ < ρ th When the similarity is low, the j-th frame is designated as a keyframe. Fourth, repeat step three until all images have been filtered. This yields a histogram set of keyframe images. and the keyframe image set {I k1 ,I k2 …,I ks}, where s represents the total number of keyframe images.

[0217] After filtering out the keyframe images from the multi-frame image set, the non-keyframe images from the multi-frame image set can be obtained.

[0218] After filtering out the keyframe and non-keyframe images from the multi-frame image set, the keyframe point cloud corresponding to the keyframe image in time sequence is obtained, and the non-keyframe point cloud corresponding to the non-keyframe image in time sequence is obtained. That is, the point cloud synchronized with the keyframe image is defined as the keyframe point cloud, and the point cloud synchronized with the non-keyframe image is defined as the non-keyframe point cloud.

[0219] Please see Figure 8 In an optional embodiment, after filtering out keyframe images, non-keyframe images, keyframe point clouds, and non-keyframe point clouds from multiple frames, the object detection method further includes:

[0220] S810, acquire the denoised image and edge texture images at different scales of the keyframe image.

[0221] Noise reduction is performed on all keyframe images to obtain the noise-reduced image and edge texture images at different scales for each keyframe image.

[0222] The following uses a keyframe image as an example to describe how to obtain the denoised image and edge texture maps at different scales of the keyframe image.

[0223] Optionally, wavelet decomposition can be performed on the keyframe image, and a threshold can be set to denoise the keyframe image. That is, the texture and edge features of each layer of the image can be reconstructed using the high-frequency wavelet coefficients in the horizontal and vertical directions after threshold filtering.

[0224] Specifically, including:

[0225] Step 1: Perform wavelet decomposition on the keyframe image to obtain multiple different wavelet components.

[0226] First, wavelet decomposition is performed on the keyframe image. In this embodiment, Haar is used to perform three-level wavelet decomposition on the keyframe image to obtain multiple different wavelet components.

[0227] Step 2: Set different thresholds for different wavelet component images, and perform threshold filtering on each wavelet component according to the threshold corresponding to each wavelet component to obtain multiple wavelet components after noise reduction.

[0228] Based on the wavelet components obtained from the decomposition, the threshold Wavecoeffth is adaptively calculated to achieve filtering and noise reduction for each wavelet component. Where M represents the length of the keyframe image, N represents the width of the keyframe image, and σ represents the noise standard deviation. σ = mida(Wavcoeff) / 0.6745, where mida(Wavcoeff) represents the median value of each wavelet component.

[0229] Based on the calculated thresholds for each wavelet component, soft thresholding is applied to each wavelet component. The formula for soft thresholding is as follows:

[0230]

[0231] When Wavecoefffiltered is 0, the corresponding wavelet component is directly filtered out. When Wavecoefffiltered is not 0, it means that the wavelet component accounts for a large proportion in the keyframe image, and in this case, the wavelet component is obtained to reconstruct the image.

[0232] Step 3: Reconstruct the denoised image based on the multiple wavelet components after denoising, and reconstruct edge texture images of different scales based on the multiple wavelet components after denoising.

[0233] When reconstructing a denoised image from multiple wavelet components after denoising, the wavelet components in the diagonal direction contain more noise information and less image information. Therefore, the wavelet components in the diagonal direction are removed during the image reconstruction process, and the remaining filtered wavelet components are used to reconstruct the image to obtain the denoised image.

[0234] When reconstructing edge texture images of different scales from multiple denoised wavelet components, edge texture maps of different wavelet components at different levels are reconstructed using horizontal and vertical wavelet components at different levels. In an optional embodiment, initial edge texture images of different scales are first reconstructed from the denoised wavelet components. Erosion and dilation operations are then performed on each initial edge texture image to obtain edge texture images of different scales. The edges in the initial edge texture image may be discontinuous and discontinuous. The erosion operation is for further noise reduction, and the dilation operation is to connect the discontinuous points into a line, finally obtaining a closed edge.

[0235] After the above steps, the denoised image and edge texture images at different scales of the keyframe image are obtained.

[0236] S820: Input the denoised image into the image feature extraction network to obtain feature maps of each scale of the denoised image, and superimpose edge texture images of different scales onto the feature maps of each scale to obtain feature maps of each scale after edge texture enhancement processing.

[0237] The image feature extraction network is, for example, Res2Net. Res2Net can achieve fine-grained extraction of image features without increasing computational cost. The Res2Net module divides the input image into multiple data blocks, and performs convolutions of different numbers of times on different data blocks.

[0238] However, in the process of feature extraction from image data, Res2Net simply divides the data into equal parts without distinguishing between them, directly performing convolutional processing with different numbers of layers. This approach is too arbitrary. Therefore, this embodiment further designs an image feature extraction network based on image information entropy to guide Res2Net, in order to achieve appropriate feature extraction for image data of varying complexity.

[0239] For example, the Res2Net module divides image data into four blocks: x1, x2, x3, and x4. Different numbers of 3×3 convolutions are performed on each block: 0, 3, 2, and 1 convolutions for blocks x1 to x4, respectively. The more convolutions performed, the richer the information contained in the resulting block's features. For instance, for block x2, y2 contains the primary semantic information of x2, y3 contains the intermediate semantic information, and y4 contains the high-level semantic information. The module's output is a combination of y1 to y4, meaning the output includes the multi-scale features and semantic information of x2. Therefore, the more textured and complex data in the image can be placed at position x2, and the blocks can be placed in order of complexity (from high to low) to their corresponding positions in the image (x2, x3, x4, x1). This allows for guided feature extraction based on the complexity of image blocks, improving the quality of the feature maps. The Res2Net-based image feature extraction method, guided by image information entropy, sorts the complexity of image data blocks based on image information entropy.

[0240] The formula for calculating image information entropy is as follows:

[0241]

[0242] Where i represents the image gray level, K represents the total number of image gray levels, and pi represents the proportion of pixels with gray level i in the total number of pixels. Image information entropy I E The larger the value, the more complex the image. Conversely, the smaller the image information entropy I. E The smaller the value, the simpler the image.

[0243] Because Res2Net involves performing convolutional operations with different numbers of layers on different parts of the data, using image information entropy to guide Res2Net to perform convolutional operations with different numbers of layers on data of varying complexity can better accomplish the feature extraction task.

[0244] That is, in an optional embodiment, the method before inputting the denoised image into the image feature extraction network includes:

[0245] Step 1: Convert the denoised image to a grayscale image, divide the grayscale image into patches of the same size, and obtain the information entropy value of each patch.

[0246] That is, the keyframe image is converted to grayscale. The grayscale conversion uses a common method, which is to calculate a weighted average of the three grayscale values ​​based on the human eye's sensitivity to the three colors R, G, and B, according to a certain weight.

[0247] The specific formula is as follows:

[0248] I G ray(x,y)=0.3×I R (x,y)+0.59×I G (x,y)+0.11×I B (x,y)

[0249] Among them, I G ray represents the grayscale result of the image, I R For the red component of a color image, I G For the green component of a color image, I B This refers to the blue component of a color image.

[0250] The grayscale image I G The ray is used to divide the image into blocks, resulting in 3×3 blocks.

[0251] Calculate the information entropy value for each image patch, and record the pixel value in each patch as its information entropy value. The specific formula is as follows:

[0252] I E (x,y)=I E _ 3×3 (x,y)

[0253] Among them, I E _ 3×3 The information entropy value represents the image containing a 3×3 patch of pixels (x, y).

[0254] Step 2: Based on the information entropy value of each image patch, the grayscale image is segmented to obtain multiple segmented image regions.

[0255] Specifically, firstly, for each image patch, based on the information entropy values ​​of the pixels within the patch, the proportion of pixels with the same information entropy value in each group is calculated to obtain an information entropy distribution histogram for the patch. Next, after Gaussian smoothing filtering, the peak points of the information entropy distribution histogram are identified. Using the number of peak points as the number of clusters and the information entropy corresponding to the peak points as the cluster centers, fuzzy C-means clustering is performed on each pixel in the information entropy distribution histogram. The information entropy values ​​of pixels belonging to the same cluster are updated to the information entropy values ​​of the cluster centers to which the pixels of that cluster belong. After the information entropy values ​​are updated, the grayscale image is segmented based on the updated information entropy values ​​to obtain multiple segmented image regions.

[0256] First, the information entropy map of the grayscale image can be obtained based on the information entropy value of each patch. Optionally, a Gaussian smoothing filter is applied to the information entropy map to obtain a smoothed information entropy map with continuous information entropy values.

[0257] The formula for calculating the information entropy distribution histogram Him_entr of the information entropy graph is as follows:

[0258]

[0259] Where, N e In the information entropy map, N represents the total number of pixels with an information entropy value of e, where N is the total number of pixels in the keyframe image. The number of peaks in the information entropy distribution histogram is recorded using N. peak Represented by the histogram H of the information entropy distribution. im_entr In (e), the information entropy value corresponding to each peak is denoted as E. hist_peak use

[0260] Based on the number of peaks N in the information entropy distribution peak Determine the number N of clusters in fuzzy C-means clustering. cluster The specific formula is as follows:

[0261]

[0262] Where, N th This is the maximum number of clusters, used to prevent the number of clusters from becoming too large.

[0263] With E hist_peaks Initialize cluster centers and perform fuzzy C-means clustering on the pixels in the smoothed information entropy map. Based on the clustering results, change the information entropy of pixels belonging to the same category to the information entropy value of the cluster center.

[0264] The above steps complete the gradient generation of the information entropy value and the segmentation of the information entropy graph.

[0265] The segmentation results of the information entropy map are mapped onto a grayscale image to achieve grayscale image segmentation and obtain multiple segmented image regions.

[0266] Step 3: Based on the information entropy value of each image region, arrange each image region to the corresponding position in the feature extraction network. Image regions with information entropy values ​​greater than a preset information entropy value are arranged at positions that can pass through a first number of convolutional layers, and image regions with information entropy values ​​less than or equal to a preset information entropy value are arranged at positions that can pass through a second number of convolutional layers, where the first number is greater than the second number.

[0267] Based on the information entropy value of each image region in the segmentation result, the regions of the grayscale image are arranged at different positions in the Res2Net feature extraction network to achieve image feature extraction guided by information entropy. Image regions with high information entropy values ​​are arranged at positions that can pass through a larger number of convolutional layers, while image regions with low information entropy values ​​are arranged at positions that can pass through a smaller number of convolutional layers.

[0268] In the above process of feature extraction from denoised images, image information entropy is introduced to characterize the gray-level distribution complexity of local patches, guiding optimized image segmentation. Simultaneously, the data segmentation and arrangement methods in the multi-scale backbone network Res2Net are improved. Image regions with high information entropy values ​​are placed in positions that can undergo more convolutional layers, while image regions with low information entropy values ​​are placed in positions that require fewer convolutional layers. Compared to the traditional method of equally dividing and sequentially arranging data in the multi-scale backbone network Res2Net, the method proposed in this embodiment achieves appropriate feature extraction with varying levels of detail, thereby improving the efficiency and accuracy of target detection.

[0269] After processing the denoised image before it is input into the image feature extraction network, the denoised image is then input into the network to obtain feature maps at various scales. Edge texture images at different scales are then superimposed onto these feature maps to obtain feature maps at each scale that have undergone edge texture enhancement. In other words, feature maps from different layers are added to the corresponding layer's (consistent in size) edge texture maps to obtain feature maps at different layers that have undergone edge texture enhancement.

[0270] S830, based on the optical flow between the keyframe image and the non-keyframe image, infers the features of the non-keyframe image.

[0271] Optionally, non-keyframe image features can be derived using sparse feature propagation methods, dense feature aggregation methods, and high-performance video object detection methods. The following sections describe these three methods in detail.

[0272] The first method: sparse feature propagation method.

[0273] The sparse feature propagation method, in video object detection, first introduced the concept of keyframes. The sparse feature propagation method was designed because similar appearances between adjacent frames often lead to similar features. Therefore, it is unnecessary to compute features for all frames.

[0274] During image feature inference, only keyframe images have their features extracted using a complex feature extraction network. Features of non-keyframe image i are obtained by propagating the features of each pixel in the previous keyframe k based on optical flow. The motion of each pixel between keyframes and non-keyframes is recorded in a two-dimensional motion field M. i→k In this context, the process of features propagating from keyframe k to non-keyframe i can be represented as:

[0275]

[0276] Wherein, w is the characteristic warping function, F k Features representing keyframes, M i→k This represents the motion field between keyframes and non-keyframes. The non-keyframe feature F is derived through the above process. k→i This eliminates the need to compute the true features F of non-keyframe images through complex feature extraction and aggregation networks. i Sports Field M i→k This can be obtained using a lightweight optical flow estimation network. I k ,,I i The input consists of keyframe and non-keyframe images.

[0277] The second method: dense feature aggregation method.

[0278] The dense feature aggregation method is the first video object detection method to introduce the concept of temporal feature aggregation. The motivation behind this method is that degraded appearance (motion blur, collision) can impair depth features. This type of problem can be improved by aggregating nearby frames.

[0279] During image feature inference, features are extracted from all frames through a feature extraction network. For any frame i, feature maps from all frames within the time window [ir, i+r] (r = 2~12) are propagated to the i-th frame via a sparse feature propagation method. This forms a feature map set {F}. k→i |k∈[ir,i+r]}, the aggregated feature map of the i-th frame is obtained by weighted averaging of all features in the feature map set.

[0280]

[0281] Wherein, weight W k→i Based on the propagation characteristic diagram Fk→i and the true feature map F i The similarity between them is calculated. In the process of calculating the similarity weight, W... k→i (p) is not directly obtained from feature F. Instead, feature F is first projected into embedded feature F before calculating similarity. e . Where ε(·) represents a miniature fully convolutional network. The weight coefficients are obtained using the following formula:

[0282]

[0283] Features are aggregated point-by-point. For any pixel p, the weights are normalized within neighboring frames:

[0284]

[0285] The third approach: a high-performance video object detection method.

[0286] The Towards High Performance Video Object Detection (THP) method proposed by Zhu et al. can perform sparse recursive feature aggregation on keyframes. That is, given two consecutive keyframes k and k′, the aggregated features of keyframe k′ are calculated by the following formula:

[0287]

[0288] in, ⊙ represents dot product, W k→k′ (p)+W k′→k′ (p) = 1, where p represents a certain pixel.

[0289] While directly inferring features from non-critical frames via sparse feature propagation is efficient, it can lead to significant variations in appearance between adjacent frames. Furthermore, directly inferring non-critical frame features from critical frame features using optical flow is prone to errors. Therefore, a matrix Q is introduced in the optical flow estimation process to determine the temporal consistency of features. k→i Pixel-by-pixel quantization to evaluate the propagation characteristics F of non-keyframes k→i Is it a non-keyframe true feature F? i If a good approximation is found, then non-keyframe features are deduced using optical flow. Otherwise, feature extraction and feature aggregation networks are used to calculate the features of the pixel. Specifically, an optical flow estimation network is used. Predicting keyframe images I k With non-keyframe image I i The optical flow matrix M between i→k The characteristic time consistent matrix Q k→i,Right now If Q k→i (p) is less than a certain threshold Q p_th At that time, for pixel p, the propagation feature F k→i (p) and the true features of non-keyframes in the image F i (p) is inconsistent, that is, F k→i (p) is a poor approximation of the true features of non-keyframes, requiring feature extraction and feature aggregation networks to refine the true features F of pixel p. i (p) Perform calculations to obtain F again. k→i (p).

[0290] This embodiment prioritizes using the third method (a high-performance video target detection method) to derive non-keyframe image features because the third method improves the efficiency of feature extraction while ensuring the quality of the extracted features.

[0291] S840: For the keyframe point cloud, extract the keyframe point cloud features and perform deformable attention feature aggregation on the keyframe point cloud features to obtain the aggregated keyframe point cloud features.

[0292] This embodiment uses a method combining principal component analysis (PCA) and PointNet to extract features from keyframe point clouds.

[0293] S850, for the non-keyframe point cloud, deduces the features of the non-keyframe point cloud based on the scene flow between the keyframe point cloud and the non-keyframe point cloud.

[0294] Point clouds are obtained by scanning spatial objects using a lidar system to acquire the three-dimensional coordinates of the reflection points of those objects. Each object's reflection point is distributed in three-dimensional space as a matrix. Therefore, the arrangement and structure of points in a point cloud are its main characteristics, and different objects have different arrangement structures. PCA (Programmable Point Analysis) analyzes points within local spaces of the point cloud, and the distribution of feature values ​​reveals the local point cloud arrangement structure, making it an effective method for distinguishing different targets. PointNet, on the other hand, uses deep learning to obtain the features of each point in the point cloud, as well as global features. By concatenating the point cloud features obtained through these two methods, a feature set containing information about each point, local structure, and global information can be obtained. This embodiment employs two schemes to extract the structural features of the keyframe point cloud, and finally concatenates the point cloud features extracted by the two schemes to obtain the final features of each point in the keyframe point cloud, including the local structure information of the keyframe point cloud.

[0295] Specifically, the steps for extracting keyframe point cloud features based on PCA and PointNet include:

[0296] The first step involves learning the structural features of the keyframe point cloud using principal component analysis. Specifically, this includes:

[0297] Input the keyframe point cloud to be processed Calculate the keyframe point cloud P pc 3D structure tensor

[0298]

[0299]

[0300] in, It is the nearest neighbor p of a point p0 in the point cloud. i i = 1, ..., N near The geometric center of N. near Set according to the actual situation, here N near =30.

[0301] It may differ slightly from p0. Due to the three-dimensional structure tensor It is a symmetric positive definite matrix, so it has three non-negative eigenvalues ​​μ1, μ2, and μ3, and the corresponding eigenvectors are mutually orthogonal. Where μ1, μ2, ..., ... Furthermore, μ1≥μ2≥μ3≥0 represents the range of variation of the 3D point cloud along its principal feature axis. Therefore, eigenvalues ​​can be used to characterize the local 3D shape. The point cloud distribution structure corresponding to the relative magnitudes of the eigenvalues ​​is shown in Table 1.

[0302] Table 1:

[0303]

[0304] Eigenvalues ​​can be used to characterize local 3D shapes, deriving the linear L... μ Planarity P μ and scattering degree S μ The dimension represented. Specific. Represents 1D, 2D, and 3D features.

[0305] As shown in Table 1, when μ1≈μ2≈μ3≈0, the coordinates of points in the neighborhood change little in the three directions, and the points are distributed around a single point. At this time, L μ ≈P μ ≈0,S μ ≈1.

[0306] When μ1 >> μ2 ≈ μ3 ≥ 0, the coordinates of points in the neighborhood are relatively dispersed in one direction and relatively concentrated in the other two directions, with the points distributed near a line segment. At this time, L μ ≈1,P μ≈S μ ≈0.

[0307] When μ1≈μ2>>μ3≥0, the points in the corresponding neighborhood are relatively dispersed in two directions and relatively concentrated in another direction, with the points distributed around a planar region. At this time, L μ ≈0,P μ ≈1,S μ ≈0.

[0308] When μ1≈μ2≈μ3>>0, the points in the corresponding neighborhood are relatively dispersed in the three directions with similar dispersion, and the points are distributed within a sphere. At this time, L μ ≈0,P μ ≈0,S μ ≈1.

[0309] When μ1>μ2>μ3≥0, the points in the corresponding neighborhood are relatively dispersed in the three directions, and the points are distributed within an ellipsoid. At this time, L μ P μ S μ It is a decimal between 0 and 1.

[0310] Normalize the eigenvalues ​​to obtain

[0311] Shannon entropy E of normalized eigenvalues μ To measure the structural complexity of local point clouds:

[0312] Estimating local surface changes using eigenvalues: For the change in curvature, C μ The value of C changes with kn. As kn gradually changes, C... μ The value will jump, indicating a strong deviation in the direction of the normal to the neighboring surface.

[0313] O using total variance μ Estimate the structural dispersion of a local point cloud:

[0314] The anisotropy of point clouds can be used measure.

[0315] Concatenating the above vectors yields the principal component-based structural features of the point cloud: ⊙ represents splicing.

[0316] The second step is to input pcd1 into the PointNet feature extraction network to obtain the feature point set.

[0317] The third step is to concatenate the above features to obtain the feature point set H = [F] of pcd1. structure1 ⊙F structure2 ], which is the extracted point cloud feature of the keyframe.

[0318] Thus, the keyframe point cloud features are extracted.

[0319] Next, deformable attention feature aggregation is performed on the keyframe point cloud features to obtain the aggregated keyframe point cloud features. According to graph theory, a graph is composed of points and edges, denoted as G(V, E). Here, V represents each vertex in the graph, and E represents the edges between vertices. For the input point cloud (keyframe point cloud)... V = {1, 2, ..., N} represents the set of vertices. E = |V| × |V| represents the set of edges. N represents the number of vertices contained in the input point cloud.

[0320] Feature point set These are the features associated with each vertex i∈V. Here, F+9 represents the feature dimension of each vertex.

[0321] Specifically, including:

[0322]

[0323]

[0324] in,

[0325]

[0326] Where, p i This represents the coordinates of the i-th point in the input point cloud, m represents the attention head index, M represents the total number of heads (M=8 in this embodiment), k represents the index of the key feature points surrounding the query feature, and K represents the total number of surrounding feature points that each query feature point is interested in (K=4 in this embodiment). The learned Δp mik The integers are not necessarily integers, and features for non-integer points in the feature map do not exist. For non-integer feature points, the features are obtained by bilinear interpolation of the four nearest neighbor feature points, denoted as . W m This is the coefficient matrix of the m-th attention head, obtained by querying feature h′. i (p i (This is learned from the input FC layer.) ⊙ indicates concatenation. F GDA This is a characteristic of aggregation.

[0327] The aggregated keyframe point cloud features can be obtained through the above two steps.

[0328] It should be noted that the above describes the steps for extracting and aggregating single-layer features in the keyframe point cloud. The steps for extracting and aggregating multi-layer features in the keyframe point cloud based on the deformable attention mechanism are similar to those for extracting and aggregating single-layer features in the keyframe point cloud. The difference is that the feature extraction stage uses a feature extraction method that combines PCA and PointNet++.

[0329] Specifically, the steps for extracting and aggregating multi-layer features in the keyframe point cloud based on the deformable attention mechanism include:

[0330] The first step is to convert the keyframe point cloud P pc Inputting the PointNet++ feature extraction network yields a multi-scale feature point set of the point cloud.

[0331]

[0332] The third step involves identifying K feature points with strong relevance to the query feature point i based on the graph deformable attention mechanism. Here, K is set to 4 to achieve multi-scale feature aggregation of the point cloud, resulting in multi-scale aggregated features based on the graph deformable attention mechanism. The specific formula is as follows:

[0333]

[0334] in,

[0335] ξ mlik =MLP(p i ⊙p mlk ⊙Δp mlik ⊙‖Δp mlik ‖)

[0336]

[0337] The coordinates are normalized in three dimensions. The purpose of normalizing the coordinates is to align different feature layers.

[0338] function φ l () is used to normalize coordinates Rescale to the feature coordinates of layer l. The meanings of other coefficients are the same as in 3.4.2; the subscript l is used to distinguish different feature layers. F MSGDA This refers to the aggregated point cloud features.

[0339] The aggregated keyframe point cloud features are denoted as PCF = F. MSGDA .

[0340] It should be noted that the aggregated feature point cloud of this high-density point cloud has the following features: non-keyframe image features, non-keyframe point cloud features, keyframe image features after image contour enhancement processing, and aggregated keyframe point cloud features.

[0341] S730 projects the pixels of each keyframe image into a three-dimensional space, and projects the pixels of each non-keyframe image into a three-dimensional space, forming a projection point cloud with the spatial points projected from the pixels of each keyframe image and the spatial points projected from the pixels of each non-keyframe image.

[0342] That is, the projected point cloud includes the spatial point cloud corresponding to the keyframe image and the spatial point cloud corresponding to the non-keyframe image.

[0343] As described in step S220, each frame contains multiple viewpoint images from different perspectives. That is, both keyframe and non-keyframe images contain multiple viewpoint images from different perspectives.

[0344] Each point in this projected point cloud carries information including two-dimensional coordinates, three-dimensional coordinates, three-channel grayscale values, and the three primary colors of the color image. The two-dimensional coordinates of each point in the projected point cloud include the coordinates corresponding to each viewpoint of the image. For example, if an image frame contains six different viewpoints, then a pixel in that frame corresponds to six two-dimensional coordinates in three-dimensional space.

[0345] S740, acquire the color information of each keyframe point cloud, and acquire the color information of each non-keyframe point cloud, to generate a laser color point cloud, wherein the laser color point cloud includes keyframe color point clouds and non-keyframe color point clouds.

[0346] Please refer to the relevant description in step S220. When obtaining the color information of points in a keyframe point cloud, the points in the keyframe point cloud are mapped to pixels in a synchronized keyframe image (a frame image contains multiple viewpoints), and the color information of the points in the point cloud is determined based on the color information of the pixels. When obtaining the color information of points in a non-keyframe point cloud, the points in the non-keyframe point cloud are mapped to pixels in a synchronized non-keyframe image (a frame image contains multiple viewpoints), and the color information of the points in the point cloud is determined based on the color information of the pixels.

[0347] S750, based on the high-density point cloud composed of the projected point cloud and the laser color point cloud, obtain the aggregation of multi-temporal feature bird's-eye view images; wherein, the feature bird's-eye view image has planar image information and three-dimensional spatial distribution information.

[0348] The projected point cloud includes spatial point clouds projected from keyframe images and spatial point clouds projected from non-keyframe images. The laser color point cloud includes keyframe laser color point clouds and non-keyframe laser color point clouds.

[0349] The description of step S750 can be found in the description of step S240, and will not be repeated here. It should be noted that distinguishing between keyframes and non-keyframes does not affect the generation process of the high-density point cloud or the generation process of the aggregation of the multi-temporal feature bird's-eye view. The generation process of the high-density point cloud and the generation process of the aggregation of the multi-temporal feature bird's-eye view are related to each point in the projected point cloud and each point in the laser color point cloud.

[0350] It should be noted that when acquiring matching pixels, the use of image features enhanced with edge texture makes it easier to obtain matching pixels. This embodiment is based on a self-attention mechanism that can learn the relationships between pixels in images from different viewpoints, and a cross-attention mechanism that can realize the relationships between different pixels in images from different viewpoints.

[0351] The specific method is as follows: First, input the feature images of the two images to be matched, after edge enhancement at each layer. The coarse-grained feature with a side length of 1 / 8 of the original image size is denoted as... and Let the fine-grained feature with a side length of 1 / 2 of the original image size be denoted as... and Then, based on a deformable attention mechanism or Transformer, relevant information between different patches is learned. This process transforms the coarse-grained feature map... and Convert to easily matched feature maps and Two feature maps are implemented. and Coarse-grained matching can be divided into the following steps:

[0352] Calculate the score matrix representing the correlation between two feature maps. Matching probability Then it can be obtained through the following formula: The softmax function performs normalization.

[0353] For matching selection, first set a threshold θ c Matching probability Greater than θ c Image pixel pairs are selected as candidate matching pixel pairs. Then, based on the mutual nearest neighbor (MNN) strategy, the candidate matching pixel pairs are further filtered and selected to remove abnormal coarse matching pixel pairs. The coarse-grained matching of the image is expressed as follows:

[0354]

[0355] Since Transformer-based matching is between tiles, the matching is not fine enough. To achieve fine-grained matching, we perform fine-grained matching based on the coarse-grained results.

[0356] Fine-grained matching is a refinement of coarse-grained matching results. It involves refining patch-level matching results to pixel-level matching. This refined matching is based on relevance. The specific steps are as follows:

[0357] For each coarse matching point First, in the fine-grained feature map and Position it above Then, crop two sets of local windows with a size of w×w.

[0358] An attention-based matching feature extraction module is applied to a smaller window, with N cropping feature transformations within each window. f This generates local feature maps of two transformations. and

[0359] respectively and Centered on. Then, The center vector and All vectors in the vector are correlated to generate a heatmap, representing the vectors in the image. Each pixel in the w×w neighborhood centered on the pixel The matching probability is calculated. By calculating the expected value of the probability distribution, the final position on the IB with sub-pixel precision can be obtained. Record all matching results Produce the final fine-level match.

[0360] S760, based on the aggregation of the multi-time-series bird's-eye view of this feature, determines and outputs detection information; wherein, the detection information represents the object category and object location information in the target scene within the current time period.

[0361] The description of step S760 can be found in the description of step S250, and will not be repeated here.

[0362] In summary, the target detection method provided in this embodiment is more effective than the target detection method provided in the previous embodiment in improving the performance of target detection. Specifically, based on the similarity of adjacent frame images, frames with significant content changes (i.e., low similarity) are classified as image keyframes, while other images with high similarity to the image keyframes and minimal content changes are classified as image non-keyframes. Point cloud data synchronized with the image keyframes are classified as point cloud keyframes, while the rest are classified as point cloud non-keyframes. The proposed feature extraction method is mainly applied to image keyframes and point cloud keyframes. The features of non-keyframes are derived by embedding keyframe features into the 2D optical flow or 3D scene flow between keyframes and non-keyframes, thereby improving computational efficiency. The features of keyframes can be obtained by weighting the features of the current keyframe and the previous keyframe, thus achieving feature memory functionality and effectively avoiding blurring caused by rapid target movement, solving the occlusion problem to some extent. In addition, wavelet transform is introduced to denoise the image and extract edges and textures from the denoised image. The advantages of using wavelet transform lie in its ability to achieve image denoising at different scales and to extract edges and textures at different scales. Furthermore, edge texture enhancement of feature maps at each scale is beneficial for subsequent feature extraction for pixel matching from different viewpoints. This is because features enhanced by edge texture strengthen the features of pixels with drastic grayscale changes, effectively strengthening the reference point information of different images and facilitating the alignment of matching pixels from different viewpoints. In addition, image information entropy is introduced to characterize the complexity of local patch grayscale distribution, guiding image segmentation optimization. Simultaneously, the data segmentation and data arrangement methods in the multi-scale backbone network Res2Net are improved. Furthermore, a point cloud feature extraction method based on a combination of PCA and PointNet, and a point cloud feature aggregation method based on a deformable attention mechanism are proposed, which helps improve the accuracy of object detection. Moreover, a deformable attention mechanism is used to aggregate point cloud features, dynamically selecting the K most relevant feature points to the query feature point and achieving point cloud feature aggregation through a multi-head attention mechanism. Since the selected feature points are those most relevant to the query feature points, the aggregated features can better represent the point cloud, thereby improving the accuracy of target detection. Furthermore, this embodiment achieves the aggregation of different modalities and temporal information, and performs target recognition and map segmentation based on the bird's-eye view of the aggregated information. This realizes the complementary advantages of different sensors, effectively overcoming the shortcomings of a single sensor and improving the reliability of the target detection system.

[0363] Optionally, this embodiment can first train the model, program, etc., used in the object detection method provided in any of the above embodiments based on the dataset to improve the practical application effect of the object detection method. The dataset can be the nuscenes dataset, a large-scale dataset in the field of autonomous driving, containing 1000 complex driving scenarios. A target scene can be manually selected with a duration of 20 seconds. The nuscenes dataset contains 1000 driving scenarios. The entire dataset includes approximately 1.4 million camera images, 390,000 LiDAR scan point clouds, 1.4 million radar scan results, and 1.4 million target bounding boxes on 40,000 keyframes. Training the model, program, etc., used in the object detection method using the complex nuscenes dataset helps improve the practical application effect of the object detection method in complex environments (urban areas with dozens of objects in each scene).

[0364] Please see Figure 14 One embodiment of this application provides a target detection device 10, comprising:

[0365] The acquisition module 11 is used to acquire multiple frames of images and multiple frames of point clouds of the target scene, wherein one frame of image corresponds to one frame of point cloud acquired synchronously.

[0366] The point cloud generation module 12 is used to project the pixels of each frame image into a three-dimensional space, form a projected point cloud from the spatial points projected from the pixels of each frame image, and obtain the color information of the points in each frame point cloud to generate a laser color point cloud.

[0367] The acquisition module 11 is also used to acquire an aggregation of multi-temporal feature bird's-eye view images based on the high-density point cloud composed of the projected point cloud and the laser color point cloud; wherein the feature bird's-eye view image has planar image information and three-dimensional spatial distribution information.

[0368] The detection module 13 is used to determine and output detection information based on the aggregation of multi-time-series bird's-eye view images of this feature; wherein, the detection information represents the object category and object location information in the target scene within the current time period.

[0369] Optionally, the acquisition module 11 is specifically used for: acquiring the information vector of each point in the high-density point cloud; wherein the information vector includes two-dimensional coordinates, three-dimensional coordinates, three-channel grayscale values, color features, and features of the high-density point cloud, wherein the features of the high-density point cloud include structural information features of the high-density point cloud, and the two-dimensional coordinates include the two-dimensional coordinates of the point in multiple different viewpoint images contained in a frame image; after performing cylindrical voxel segmentation on the high-density point cloud, voxel encoding is performed on the segmented spatial point cloud based on the information vector of each point in the segmented voxel spatial point cloud to obtain the voxelized point cloud; asymmetric convolution is performed on the voxelized point cloud to aggregate the information vectors of different points to obtain the aggregated information point cloud; the aggregated information point cloud is compressed along the height direction to obtain a multi-temporal feature bird's-eye view; and an image feature aggregation method based on a deformable attention mechanism is used to process the multi-temporal feature bird's-eye view to obtain the aggregation of the multi-temporal feature bird's-eye view.

[0370] Optionally, the point cloud generation module 12 is specifically used to: form an initial high-density point cloud based on the projected point cloud and the laser color point cloud; obtain the depth information of the spatial points of the matching pixel projections in each frame of the initial high-density point cloud to obtain the color point cloud in the projected point cloud, wherein a frame of image contains multiple viewpoint images from different perspectives, the matching pixel is a pixel in one viewpoint image, and the matching pixel and a pixel in another viewpoint image share the same three-dimensional projection spatial point, and the depth information of the matching pixel and the pixel is the depth information of the three-dimensional projection spatial point; form a dense point cloud based on the color point cloud and the laser color point cloud; estimate the depth information of the spatial points of the non-matching pixel projections in the initial high-density point cloud based on the dense point cloud to obtain the depth information of the spatial points of the non-matching pixel projections in the initial high-density point cloud; and correct the depth information of the spatial points of the non-matching pixel projections in the initial high-density point cloud based on the dense point cloud to obtain the high-density point cloud.

[0371] Optionally, the acquisition module 11 is specifically used to match pixels in multiple different viewpoint images contained in each frame image, to obtain each pair of matching pixels between two viewpoint images, wherein a pair of matching pixels contains two matching pixels; to obtain the same three-dimensional projection space point corresponding to each pair of matching pixels in each frame image, to obtain the coordinates of the three-dimensional projection space point in the coordinate system of each image acquisition device and in the world coordinate system; to determine the depth information of the three-dimensional projection space point in each viewpoint based on the coordinates of the three-dimensional projection space point in the coordinate system of each image acquisition device and in the world coordinate system, and to use the depth information of the three-dimensional projection space point in each viewpoint as the depth information of the matching pixels in the corresponding viewpoint image.

[0372] Optionally, the target detection device 10 further includes a filtering module 14, used to filter out key frame images and non-key frame images from multiple frames; and to acquire key frame point clouds corresponding to the time sequence of the key frame images, and to acquire non-key frame point clouds corresponding to the time sequence of the non-key frame images. The point cloud generation module 12 is specifically used to project the pixels of each key frame image into three-dimensional space, and to project the pixels of each non-key frame image into three-dimensional space, so as to form a projected point cloud with the spatial points projected from the pixels of each key frame image and the spatial points projected from the pixels of each non-key frame image; to acquire the color information of each key frame point cloud, and to acquire the color information of each non-key frame point cloud, so as to generate a laser color point cloud, wherein the laser color point cloud includes key frame color point clouds and non-key frame color point clouds.

[0373] Optionally, the acquisition module 11 is further configured to acquire the denoised image and edge texture images at different scales of the keyframe image, input the denoised image into an image feature extraction network to obtain feature maps at various scales of the denoised image, superimpose the edge texture images at different scales onto the feature maps at various scales to obtain feature maps at various scales after edge texture enhancement processing; and deduce the features of the non-keyframe image based on the optical flow between the keyframe image and the non-keyframe image. For the keyframe point cloud, keyframe point cloud features are extracted, and deformable attention feature aggregation is performed on the keyframe point cloud features to obtain aggregated keyframe point cloud features; for the non-keyframe point cloud, non-keyframe point cloud features are deduced based on the scene flow between the keyframe point cloud and the non-keyframe point cloud; the features of the aggregated feature point cloud of the high-density point cloud include: non-keyframe image features, non-keyframe point cloud features, keyframe image features after image contour enhancement processing, and aggregated keyframe point cloud features.

[0374] Optionally, the acquisition module 11 is specifically used to perform wavelet decomposition on the keyframe image to obtain multiple different wavelet components; set different thresholds for different wavelet component images, and perform threshold filtering on each wavelet component according to the threshold corresponding to each wavelet component to obtain multiple denoised wavelet components; reconstruct a denoised image based on the multiple denoised wavelet components, and reconstruct edge texture images of different scales based on the multiple denoised wavelet components.

[0375] Optionally, the acquisition module 11 is specifically used to reconstruct initial edge texture images of different scales based on the denoised wavelet components; and to perform erosion and dilation operations on the initial edge texture images of each scale to obtain edge texture images of different scales.

[0376] Optionally, the target detection device 10 further includes an image processing module 15, which is used to convert the denoised image into a grayscale image, divide the grayscale image into patches of the same size, and obtain the information entropy value of each patch; according to the information entropy value of each patch, the grayscale image is segmented to obtain multiple segmented image regions; according to the information entropy value of each image region, each image region is arranged to the corresponding position in the feature extraction network, wherein image regions with information entropy values ​​greater than a preset information entropy value are arranged at positions that can pass through a first number of convolutional layers, and image regions with information entropy values ​​less than or equal to the preset information entropy value are arranged at positions that can pass through a second number of convolutional layers, wherein the first number is greater than the second number.

[0377] Optionally, the image processing module 15 is specifically used for: for each patch, according to the magnitude of the information entropy value of the pixels in the patch, to count the proportion of pixels with the same information entropy value in each group, so as to obtain the information entropy distribution statistical histogram of the patch; after performing Gaussian smoothing filtering on the information entropy distribution statistical histogram, to identify the peak points of the information entropy distribution statistical histogram; using the number of peak points as the number of clusters and the information entropy corresponding to the peak points as the cluster centers, to perform fuzzy C-means clustering on each pixel in the information entropy distribution statistical histogram, and to update the information entropy value of pixels belonging to the same category to the information entropy value of the cluster center to which the pixels of the same category belong; after the information entropy value is updated, to segment the grayscale image according to the updated information entropy value, and to obtain multiple segmented image regions.

[0378] Optionally, each frame contains multiple viewpoint images from different perspectives. Specifically, the point cloud generation module 12 projects each point in each frame's point cloud onto the viewpoint images of the corresponding frame image; when a point is projected onto an integer pixel, the color information of the projected integer pixel is obtained as the point's color information; when a point is projected onto a non-integer pixel, the color information of the non-integer pixel to which the point is projected is estimated, and the estimated color information is used as the point's color information.

[0379] Optionally, the point cloud generation module 12 is specifically used to obtain the four nearest neighbor pixels to the projected non-integer pixel, and to determine the weight coefficients of the four pixels based on the distance and grayscale distance between each of the four pixels and the projected non-integer pixel; and to substitute the weight coefficients of the four pixels and the color information of the four pixels into a Gaussian function to obtain the estimated result of the color information of the projected non-integer pixel.

[0380] Optionally, the target detection device 10 also includes a filtering module 16 for filtering out ground point clouds in each frame of point cloud.

[0381] Please see Figure 15One embodiment of this application also provides an electronic device 20, including: a processor 21, and a memory 22 communicatively connected to the processor 21. The memory 22 stores computer-executable instructions, and the processor 21 executes the computer-executable instructions stored in the memory 22 to implement the target detection method as provided in any of the preceding embodiments.

[0382] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed, cause the computer-executable instructions to be executed by a processor to implement the target detection method provided in any of the preceding embodiments.

[0383] This application also provides a computer program product, including a computer program that, when executed by a processor, is used to implement the target detection method as provided in any of the preceding embodiments.

[0384] It should be noted that the aforementioned computer-readable storage media can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc. It can also be various electronic devices containing one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0385] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising a…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0386] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0387] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and contains several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0388] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0389] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture containing instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0390] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0391] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A target detection method, characterized in that, The method includes: Acquire multiple frames of images and multiple frames of point clouds of the target scene, where each frame of image corresponds to a frame of point cloud acquired synchronously; The pixels of each frame image are projected into a three-dimensional space, and the spatial points of the projected pixels of each frame image are used to form a projected point cloud. In addition, the color information of the points in the point cloud of each frame is obtained to generate a laser color point cloud. Based on the high-density point cloud composed of the projected point cloud and the laser color point cloud, an aggregation of multi-temporal feature bird's-eye view images is obtained; wherein, the feature bird's-eye view image has planar image information and three-dimensional spatial distribution information; Based on the aggregation of the feature bird's-eye view images from multiple time series, detection information is determined and output; wherein, the detection information represents the object category and object location information in the target scene within the current time period; The aggregation of multi-temporal feature bird's-eye view images based on the high-density point cloud composed of the projected point cloud and the laser color point cloud includes: The information vector of each point in the high-density point cloud is obtained; wherein the information vector includes two-dimensional coordinates, three-dimensional coordinates, three-channel grayscale values, color features and features of the high-density point cloud, wherein the features of the high-density point cloud include features of the structural information of the high-density point cloud, and the two-dimensional coordinates include the two-dimensional coordinates of the point in multiple viewpoint images with different perspectives contained in a frame image. After the high-density point cloud is segmented into cylindrical voxels, the segmented spatial point cloud is voxel-encoded based on the information vector of each point in the segmented voxel spatial point cloud to obtain the voxelized point cloud. Asymmetric convolution is performed on the voxelized point cloud to aggregate the information vectors of different points and obtain the aggregated information point cloud. The aggregated point cloud is compressed along the height direction to obtain a multi-temporal feature bird's-eye view; An image feature aggregation method based on deformable attention mechanism processes multi-temporal feature bird's-eye view images to obtain the aggregation of multi-temporal feature bird's-eye view images.

2. The method according to claim 1, characterized in that, The process of assembling a high-density point cloud based on the projected point cloud and the laser-colored point cloud includes: An initial high-density point cloud is formed based on the projected point cloud and the laser-colored point cloud; The depth information of the spatial points of the matching pixel projection of each frame image in the initial high-density point cloud is obtained to obtain the color point cloud in the projected point cloud. Here, a frame image contains multiple view images from different perspectives. The matching pixel is a pixel in one view image, and the matching pixel and a pixel in another view image have the same three-dimensional projection spatial point. The depth information of the matching pixel and the pixel is the depth information of the three-dimensional projection spatial point. A dense point cloud is formed based on the color point cloud and the laser color point cloud; Based on the dense point cloud, the depth information of the spatial points of the non-matching pixel projection in the initial high-density point cloud is estimated to obtain the depth information of the spatial points of the non-matching pixel projection in the initial high-density point cloud. Based on the dense point cloud, the depth information of the spatial points of the non-matching pixel projection in the initial high-density point cloud is corrected to obtain the high-density point cloud. The step of obtaining the depth information of matching pixels in each frame of the image includes: For each frame of an image containing multiple viewpoints from different perspectives, the pixels in the viewpoints from different perspectives are matched to obtain each pair of matching pixels between every two viewpoints. A pair of matching pixels contains two matching pixels. Obtain the same 3D projection space point corresponding to each pair of matching pixels in each frame of the image, and obtain the coordinates of the 3D projection space point in the coordinate system of each image acquisition device and the coordinates in the world coordinate system; Based on the coordinates of the 3D projection spatial points in the coordinate system of each image acquisition device and in the world coordinate system, the depth information of the 3D projection spatial points in each viewpoint is determined, and the depth information of the 3D projection spatial points in each viewpoint is used as the depth information of the matching pixels in the corresponding viewpoint image.

3. The method according to claim 1, characterized in that, After acquiring multiple frames of images and multiple frames of point clouds of the target scene, the method further includes: The process involves filtering out key frame images and non-key frame images from multiple frames; obtaining key frame point clouds corresponding to the key frame images in time sequence; and obtaining non-key frame point clouds corresponding to the non-key frame images in time sequence. The step of projecting the pixels of each frame of an image into a three-dimensional space, and forming a projection point cloud from the spatial points projected from the pixels of each frame of the image, includes: The pixels of each keyframe image are projected into three-dimensional space, and the pixels of each non-keyframe image are projected into three-dimensional space, forming a projection point cloud with the spatial points projected from the pixels of each keyframe image and the spatial points projected from the pixels of each non-keyframe image. The step of obtaining the color information of points in each frame of the point cloud to generate a laser color point cloud includes: The color information of each keyframe point cloud and the color information of each non-keyframe point cloud are obtained to generate a laser color point cloud, wherein the laser color point cloud includes keyframe color point clouds and non-keyframe color point clouds.

4. The method according to claim 3, characterized in that, After filtering out keyframe and non-keyframe images from multiple frames, the method further includes: The denoised image and edge texture images at different scales of the keyframe image are obtained. The denoised image is input into the image feature extraction network to obtain feature maps at each scale of the denoised image. The edge texture images at different scales are superimposed on the feature maps at each scale to obtain feature maps at each scale after edge texture enhancement processing. Based on the optical flow between the keyframe image and the non-keyframe image, the features of the non-keyframe image are deduced. After acquiring the keyframe point cloud corresponding to the keyframe image in time sequence, and acquiring the non-keyframe point cloud corresponding to the non-keyframe image in time sequence, the method further includes: For the keyframe point cloud, keyframe point cloud features are extracted, and deformable attention feature aggregation is performed on the keyframe point cloud features to obtain aggregated keyframe point cloud features. For the non-keyframe point cloud, the features of the non-keyframe point cloud are deduced based on the scene flow between the keyframe point cloud and the non-keyframe point cloud. The aggregated feature point cloud of the high-density point cloud has the following features: non-keyframe image features, non-keyframe point cloud features, keyframe image features after image contour enhancement processing, and aggregated keyframe point cloud features.

5. The method according to claim 4, characterized in that, The process of acquiring the denoised image and edge texture images at different scales of the keyframe image includes: Wavelet decomposition was performed on the keyframe image to obtain multiple different wavelet components; Different thresholds are set for different wavelet component maps, and threshold filtering is performed on each wavelet component according to the threshold corresponding to each wavelet component to obtain multiple wavelet components after noise reduction. The denoised image is reconstructed from the multiple wavelet components after denoising, and edge texture images of different scales are reconstructed from the multiple wavelet components after denoising. The step of reconstructing edge texture images of different scales based on multiple wavelet components after noise reduction includes: Initial edge texture images of different scales are reconstructed based on the denoised wavelet components; Erosion and dilation operations are performed on the initial edge texture image at each scale to obtain edge texture images at different scales.

6. The method according to claim 4, characterized in that, Before inputting the denoised image into the image feature extraction network to obtain feature maps of the denoised image at various scales, the method further includes: The denoised image is converted into a grayscale image, and the grayscale image is divided into patches of the same size. The information entropy value of each patch is obtained. Based on the information entropy value of each patch, the grayscale image is segmented to obtain multiple image regions. Based on the information entropy value of each image region, each image region is arranged to the corresponding position in the feature extraction network. Image regions with information entropy values ​​greater than a preset information entropy value are arranged at positions that can pass through a first number of convolutional layers, and image regions with information entropy values ​​less than or equal to a preset information entropy value are arranged at positions that can pass through a second number of convolutional layers, where the first number is greater than the second number. The step of obtaining multiple segmented image regions after segmenting the grayscale image based on the information entropy value of each image patch includes: For each patch, based on the information entropy value of the pixels in the patch, the proportion of pixels with the same information entropy value in each group is counted to obtain the information entropy distribution statistical histogram of the patch. After applying Gaussian smoothing filtering to the statistical histogram of information entropy distribution, the peak points of the statistical histogram of information entropy distribution are identified. Using the number of peak points as the number of clusters and the information entropy corresponding to the peak points as the cluster centers, fuzzy C-means clustering is performed on each pixel in the information entropy distribution statistical histogram. The information entropy values ​​of pixels belonging to the same category are updated to the information entropy values ​​of the cluster centers to which the pixels belong. After the information entropy value is updated, the grayscale image is segmented based on the updated information entropy value to obtain multiple segmented image regions.

7. The method according to claim 1, characterized in that, Each frame contains multiple viewpoint images from different perspectives. The step of obtaining the color information of points in each frame's point cloud to generate a laser color point cloud includes: Project each point in the point cloud of each frame onto the view images of the corresponding frame image from various perspectives; When a point is projected onto an integer pixel, the color information of the integer pixel onto which the point is projected is obtained as the color information of the point. When a point is projected onto a non-integer pixel, the color information of the non-integer pixel to which the point is projected is estimated, and the estimated color information is used as the color information of the point. The color information of the non-integer pixels projected onto the estimated point includes: Obtain the four nearest neighbors of the non-integer pixel being projected, and determine the weight coefficients of the four pixels based on the distance and grayscale distance between each of the four pixels and the non-integer pixel being projected. Substitute the weight coefficients and color information of the four pixels into the Gaussian function to obtain the estimated color information of the non-integer pixels projected onto the system.

8. A target detection device, characterized in that, include: The acquisition module is used to acquire multiple frames of images and multiple frames of point clouds of the target scene, wherein one frame of image corresponds to one frame of point cloud acquired synchronously; The point cloud generation module is used to project the pixels of each frame image into a three-dimensional space, form a projected point cloud from the spatial points projected from the pixels of each frame image, and obtain the color information of the points in each frame point cloud to generate a laser color point cloud. The acquisition module is further configured to acquire an aggregation of multi-temporal feature bird's-eye view images based on the high-density point cloud composed of the projected point cloud and the laser color point cloud; wherein the feature bird's-eye view image has planar image information and three-dimensional spatial distribution information; The detection module is used to determine and output detection information based on the aggregation of the feature bird's-eye view images from multiple time series; wherein, the detection information represents the object category and object location information in the target scene within the current time period; The acquisition module is specifically used to acquire the information vector of each point in the high-density point cloud; wherein the information vector includes two-dimensional coordinates, three-dimensional coordinates, three-channel grayscale values, color features and features of the high-density point cloud, wherein the features of the high-density point cloud include features of the structural information of the high-density point cloud, and the two-dimensional coordinates include the two-dimensional coordinates of the point in multiple different viewpoint images contained in a frame image. After the high-density point cloud is segmented into cylindrical voxels, the segmented spatial point cloud is voxel-encoded based on the information vector of each point in the segmented voxel spatial point cloud to obtain the voxelized point cloud. Asymmetric convolution is performed on the voxelized point cloud to aggregate the information vectors of different points and obtain the aggregated information point cloud. The aggregated point cloud is compressed along the height direction to obtain a multi-temporal feature bird's-eye view; An image feature aggregation method based on deformable attention mechanism processes multi-temporal feature bird's-eye view images to obtain the aggregation of multi-temporal feature bird's-eye view images.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

Images