A target detection method, target detection model training method and related equipment

Abstract

The present application discloses a target detection method, a target detection model training method, an apparatus, a computer device, and a storage medium. The present application embodiment can obtain a depth image to be detected, wherein each position point of the depth image to be detected has attribute information; perform feature extraction on the depth image to be detected to obtain feature maps of different resolutions; detect the feature maps based on the attribute information, generate multiple detection frames within a preset attribute interval corresponding to each feature map, and generate a classification confidence level for each detection frame; and merge the multiple detection frames to obtain a target frame corresponding to the depth image to be detected. The present application embodiment improves the accuracy and convenience of target detection.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a target detection method, a target detection model training method, and related equipment. Background Art

[0002] With the continuous development of artificial intelligence, object detection technology has become more and more mature and widely used. Among them, object detection can be to find the object of interest in an image and determine the location of the object.

[0003] Currently, existing target detection methods generally extract features from scene data in a certain format and predict the location of the target based on the features. For example, 3D object detection based on Bird's Eye View (BEV) can voxelize the point cloud in the image from a bird's eye perspective, so that the disordered and irregular point cloud is structured to a certain extent, making it easier to use standard convolution operations to extract features from the image based on the structured point cloud, and predict the location of the object based on the extracted features. However, in high-speed driving environments, in order to ensure the safety of autonomous vehicles, the vehicle requires a large perception range, and voxelizing a large area from a bird's eye perspective will bring a huge computational burden. Therefore, this detection method cannot detect objects at a distance. For example, 3D object detection based on multi-viewpoints integrates the features of the point cloud from multiple perspectives. The fusion of multi-viewpoint features often has a very complex algorithm process and greater algorithm complexity. Therefore, this detection method is difficult to be directly applied in real-time scenarios. Summary of the Invention

[0004] The embodiments of the present application provide a target detection method, a target detection model training method, an apparatus, a computer device, and a storage medium, which can improve the accuracy and convenience of target detection.

[0005] To solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0006] The present invention provides a method for detecting a target, including:

[0007] Acquire a depth image to be detected, wherein each position point of the depth image to be detected has attribute information;

[0008] Performing feature extraction on the depth image to be detected to obtain feature maps of different resolutions;

[0009] Detecting the feature map according to the attribute information, generating a plurality of detection frames within a preset attribute interval corresponding to each feature map, and a classification confidence of each detection frame;

[0010] The multiple detection frames are merged to obtain a target frame corresponding to the depth image to be detected.

[0011] According to one aspect of the present application, a method for training a target detection model is also provided, comprising:

[0012] Acquire a sample depth image, wherein the sample depth image includes attribute information and a true frame of each position point;

[0013] Performing feature extraction on the sample depth image to obtain sample feature maps of different resolutions, wherein the sample feature maps are assigned real frames with preset attribute intervals;

[0014] Detecting each sample feature map based on the attribute information, generating a corresponding detection frame and a predicted classification confidence of each detection frame;

[0015] Calculating the true classification confidence of the detection frame and the corresponding true frame of the same category; and

[0016] The initial target detection model is trained based on the predicted classification confidence and the true classification confidence to obtain a trained target detection model.

[0017] According to one aspect of the present application, there is also provided a target detection device, comprising:

[0018] A first acquisition module is used to acquire a depth image to be detected, where each position point of the depth image to be detected has attribute information;

[0019] A first extraction module is used to extract features from the depth image to be detected to obtain feature maps of different resolutions;

[0020] a first detection module, configured to detect the feature map according to the attribute information, and generate a plurality of detection frames within a preset attribute interval corresponding to each feature map, and a classification confidence of each detection frame;

[0021] The merging module is used to merge the multiple detection frames to obtain a target frame corresponding to the depth image to be detected.

[0022] According to one aspect of the present application, a target detection model training device is also provided, comprising:

[0023] A second acquisition module is used to acquire a sample depth image, wherein the sample depth image includes attribute information and a real frame of each position point;

[0024] A second extraction module is configured to perform feature extraction on the sample depth image to obtain sample feature maps of different resolutions, wherein the sample feature maps are assigned real frames with preset attribute intervals;

[0025] A second detection module is used to detect each sample feature map based on the attribute information, generate a corresponding detection box and a predicted classification confidence of each detection box;

[0026] A calculation module, configured to calculate the true classification confidence of the detection frame and the corresponding true frame of the same category; and

[0027] The training module is used to train the initial target detection model based on the predicted classification confidence and the true classification confidence to obtain a trained target detection model.

[0028] According to one aspect of the present application, a computer device is also provided, including a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, any target detection method provided in the embodiments of the present application is executed, or when the processor calls the computer program in the memory, any target detection model training method provided in the embodiments of the present application is executed.

[0029] According to one aspect of the present application, a storage medium is also provided, which is used to store a computer program, and the computer program is loaded by a processor to execute any target detection method provided in the embodiments of the present application, or to execute any target detection model training method provided in the embodiments of the present application.

[0030] The embodiment of the present application can obtain a depth image to be detected, in which each position point of the depth image to be detected has attribute information, and perform feature extraction on the depth image to be detected to obtain feature maps of different resolutions; then the feature map can be detected according to the attribute information to generate multiple detection frames within a preset attribute interval corresponding to each feature map, as well as the classification confidence of each detection frame. At this time, the multiple detection frames can be merged to obtain a target frame corresponding to the depth image to be detected, thereby achieving efficient and accurate detection of the target frame in the depth image to be detected, and eliminating the need for multi-perspective feature fusion, thereby improving the convenience and accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 This is a schematic diagram of a target detection application scenario provided by an embodiment of the present application;

[0033] Figure 2 Schematic diagram of the target detection method provided in the embodiment of the present application;

[0034] Figure 3 is a schematic diagram of a depth image provided by an embodiment of the present application;

[0035] Figure 4 is a schematic diagram of depth image generation provided by an embodiment of the present application;

[0036] Figure 5 is a schematic diagram of the target detection architecture provided in an embodiment of the present application;

[0037] Figure 6 Schematic diagram of extracting feature maps through a meta-convolution operator provided in an embodiment of the present application;

[0038] Figure 7 This is a schematic diagram of sampling area generation provided by an embodiment of the present application;

[0039] Figure 8 This is a schematic diagram of sampling reference points and neighborhood points within the sampling area provided by an embodiment of the present application;

[0040] Figure 9 This is a schematic diagram of an embodiment of the present application for detecting a depth image to be detected to obtain multiple candidate frames, and determining a target frame from the multiple candidate frames;

[0041] Figure 10 Schematic diagram of the target detection model training architecture provided in the embodiment of the present application;

[0042] Figure 11 Schematic diagram of the target detection model training method provided in the embodiment of the present application;

[0043] Figure 12 This is a schematic diagram of the structure of the target detection model training provided in the embodiment of the present application;

[0044] Figure 13 This is a schematic diagram of an unmanned vehicle automatically detecting and avoiding obstacles provided by an embodiment of the present application;

[0045] Figure 14 is a schematic diagram of a target detection device provided in an embodiment of the present application;

[0046] Figure 15 Schematic diagram of the target detection model training device provided in an embodiment of the present application;

[0047] Figure 16 It is a structural diagram of the computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0049] In the following description, the terms "first\second" are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0051] The embodiments of the present application provide a target detection method, a target detection model training method, and related equipment, wherein the related equipment may include a target detection device, a target detection model training device, a computer device, a storage medium, and the like.

[0052] The target detection method and target detection model training method provided in the embodiments of the present application can be implemented by the terminal or the server alone, or can be implemented by the terminal and the server in collaboration. For example, the terminal alone undertakes the target detection method and target detection model training method described below, or the terminal can send a detection request for the depth image to be detected (including the depth image to be detected) to the server, and the server can execute the target detection method based on the received detection request for the depth image to be detected, and send the detection result to the terminal, that is, the target box in the depth image to be detected, which can be the position of the target object in the depth image to be detected. For another example, the terminal undertakes the target detection method described below, and the server undertakes the target detection model training method described below.

[0053] The computer device for target detection provided in the embodiments of the present application can be various types of terminal devices or servers, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, but is not limited to this. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, camera, wearable device, or vehicle-mounted terminal, etc. The vehicle-mounted terminal can include an unmanned vehicle, but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiments of the present application.

[0054] See also Figure 1 , Figure 1 A schematic diagram of a scenario for the application of the target detection method and target detection model training method provided in an embodiment of the present application. The target detection method can be applied to a target detection device, which can be specifically integrated in a terminal 10 or a server 20. The target detection model training method can be applied to target detection model training, which can also be integrated in a terminal 10 or a server 20. The present application does not impose any specific restrictions on the deployment locations of these two devices.

[0055] The following will be explained in detail using the example of integrating the target detection device in the terminal 10 and the target detection model training device in the server 20. For example, the server 20 can obtain a sample depth image, which includes attribute information and a true frame of each position point, and can perform feature extraction on the sample depth image to obtain sample feature maps of different resolutions, in which a true frame with a preset attribute interval is assigned to the sample feature map; then, each sample feature map can be detected based on the attribute information to generate a corresponding detection frame and a predicted classification confidence of each detection frame, and the true classification confidence of the detection frame and the corresponding true frame of the same category is calculated; and the initial target detection model is trained based on the predicted classification confidence and the true classification confidence to obtain a trained target detection model, thereby improving the accuracy and reliability of the target detection model training.

[0056] The server 20 can send the trained target detection model to the terminal 10. At this time, the terminal 10 can obtain the depth image to be detected, and each position point of the depth image to be detected has attribute information. Then, the target detection model can be used to extract features of the depth image to be detected to obtain feature maps of different resolutions, and the target detection model can be used to detect the feature map according to the attribute information to generate multiple detection frames within the preset attribute interval corresponding to each feature map, as well as the classification confidence of each detection frame; at this time, the multiple detection frames can be merged through the target detection model to obtain the target frame corresponding to the depth image to be detected, thereby realizing efficient and accurate detection of the target frame in the depth image to be detected, and improving the convenience of target detection.

[0057] It should be noted that Figure 1 The scenario diagram of the target detection method application shown is only an example. The target detection method application and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided by the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the application of the target detection method and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems.

[0058] It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0059] In this embodiment, the description will be made from the perspective of an object detection device, which may be integrated into a computer device such as a terminal or a server.

[0060] See also Figure 2 , Figure 2 : is a flow chart of a target detection method provided in one embodiment of the present application. The target detection method may include:

[0061] S101: Acquire a depth image to be detected, where each position point of the depth image to be detected has attribute information.

[0062] like Figure 3 As shown, a depth image containing depth information can be collected by a laser radar, and the collected depth image is used as a depth image to be detected. Figure 3 In the figure, the depth image to be detected is displayed from different perspectives, or a pre-stored depth image to be detected can be obtained from a database. The depth image to be detected may include 3D point cloud data, which may be data of the surrounding environment collected by a laser radar and represented by a set of sparse 3D spatial points. That is, each valid point on the depth image to be detected corresponds to a point in the point cloud data. Therefore, the depth image obtained by the laser radar is a representation method of the laser point cloud.

[0063] The depth image to be detected can be a tensor of a×b×c, where a can represent the number of laser radar lines, b can represent the number of sampling times of the laser radar rotating one circle, and c can represent the number of channels of input data. The values ​​of a, b and c can be flexibly set according to actual needs, for example, Figure 4 As shown, the depth image to be detected can be a tensor of 64×2650×8, that is, a=64, b=2650, c=8, 8 can represent 8 channels of input data, and these 8 channels can include: depth, reflection intensity, elongation, x-coordinate, y-coordinate, z-coordinate, azimuth, and inclination, etc.

[0064] The depth image to be detected may include a target object to be detected, which may include a vehicle, a building, a person, a plant, or an animal, the specific content of which is not limited herein. The depth image to be detected may also include attribute information of each position point in the depth image to be detected. In one embodiment, the attribute information may include at least one of depth information, spatial coordinate value, elongation, reflection intensity, azimuth, pitch angle, size of the real frame, and number of points within the real frame.

[0065] S102 : Extract features from the depth image to be detected to obtain feature maps with different resolutions.

[0066] Among them, the target detection model can be used to perform a convolution operation on the depth image to be detected to extract the feature map corresponding to the depth image to be detected, or other methods can be used to extract features of the depth image to be detected, etc., which are not limited here. The target detection model can be a trained target detection model. The specific training method of the target detection model can refer to the embodiment of the target detection model method below, and the specific content is not repeated here. Among them, the specific structure and parameters of the target detection model can be flexibly set according to actual needs. After obtaining the depth image to be detected, the depth image to be detected can be detected by the target detection model. In one embodiment, Figure 5As shown, the target detection model may include a meta-convolution operator, a detection network, and a weighted non-maximum suppression module, etc. The meta-convolution operator of the target detection model can be used to extract the feature map corresponding to the depth image to be detected, and the feature map can be detected based on the attribute information through the detection network to generate multiple detection frames within the preset attribute interval corresponding to each feature map, as well as the classification confidence of each detection frame. At this time, the multiple detection frames can be merged through the weighted non-maximum suppression module to obtain the target frame corresponding to the depth image to be detected. The target frame is the area where the detected target object is located, and can also include the category to which the target object belongs, etc., which will be described in detail below. Among them, the shape and size of the detection frame or target frame can be flexibly set according to actual needs, and it can be either a two-dimensional frame or a three-dimensional frame. For example, the detection frame or target frame can be a quadrilateral rectangular area.

[0067] In one embodiment, feature extraction is performed on the depth image to be detected to obtain feature maps of different resolutions, including: feature extraction is performed on the depth image to be detected to obtain a corresponding initial feature map; downsampling the initial feature map according to a preset sampling step size to obtain a plurality of first sampling feature maps of different resolutions; upsampling the plurality of first sampling feature maps of different resolutions to the original resolution of the sample depth image, respectively, to obtain a plurality of second sampling feature maps of different resolutions, where the second sampling feature maps of different resolutions are feature maps of different resolutions.

[0068] Specifically, first, feature extraction is performed on the depth image to be detected to obtain an initial feature map corresponding to the depth image to be detected. For example, a convolution operation can be performed on the depth image to be detected by the target detection model to extract the initial feature map corresponding to the depth image to be detected. The convolution operation can be performed on the depth image to be detected by the ordinary convolution operator of the target detection model. In order to improve the accuracy of the initial feature map extraction, the convolution operation can be performed on the depth image to be detected by the meta-convolution operator.

[0069] In one embodiment, feature extraction is performed on a depth image to be detected to obtain a corresponding initial feature map, including: sampling an area within a preset neighborhood range from the depth image to be detected to obtain at least one sampling area, where the position points of the sampling area include a sampling reference point and a plurality of neighborhood points; calculating the relative relationship between the sampling reference point and the neighborhood points within the sampling area; calculating a weight vector of each sampling point within the sampling area based on the relative relationship; extracting a feature vector of each sampling point within the sampling area; calculating an output vector of the sampling area based on the weight vector and the feature vector of each sampling point within the sampling area; and generating an initial feature map corresponding to the depth image to be detected based on the output vector of each sampling area.

[0070] For example, the convolution operation on the depth image to be detected can be performed as follows: Figure 6As shown, an area within a preset neighborhood range can be sampled from the depth image to be detected to obtain a sampling area, wherein the preset neighborhood range can be flexibly set according to actual needs, such as a square, rectangle, or any other shape. The present disclosure does not impose specific restrictions on its shape and size. For example, a 3×3 or 5×5 sampling area can be sampled. The sampling area can include multiple areas, such as Figure 7 As shown, a sliding window can be sampled from the upper left corner of the depth image to be detected to the lower right corner of the depth image to be detected to obtain multiple sampling areas.

[0071] After obtaining the sampling area, the relative relationship between the sampling reference point and the neighboring points in the sampling area can be obtained. The sampling reference point can be any point in the sampling area, and this disclosure does not limit this. For example, Figure 8 As shown, the center point of the sampling area (i.e., the sampling center point) can be used as the sampling reference point, and the points around the sampling reference point can be used as neighborhood points. Alternatively, if there are two points in the center of the sampling area, either point can be used as the reference point, and the other points can be used as neighborhood points. In one embodiment, the relative relationship includes at least one of a relative position relationship and a relative feature relationship between the sampling reference point and each position point in the sampling area, the relative position relationship includes relative Cartesian coordinates, the relative feature relationship includes an association relationship between attribute features, and the attribute features can include attribute information.

[0072] For example, Figure 6 As shown, the three-dimensional position coordinates (x i ,y i , z i ), and obtain the three-dimensional position coordinates (x j ,y j , z j ), according to the difference between the three-dimensional position coordinates of the sampling reference point and the three-dimensional position coordinates of the neighboring points, the relative Cartesian coordinates between the sampling reference point and the neighboring points are determined as (x i -x j ,y i -y j , z i -z j ), the relative Cartesian coordinate is (x i -x j ,y i -y j , z i -z j ) as the relative relationship between the sampling reference point and the neighborhood points. Or, if the eigenvector of the sampling reference point is (a1, a2, ..., a n ), the eigenvector of a neighborhood point is (b1, b2, ..., b n), the relative feature relationship can be the difference between the corresponding dimensions of two feature vectors, for example (a1-b1, a2-b2, ..., a n -b n ).

[0073] Then, the weight vector of each sampling point in the sampling area can be obtained based on the relative relationship between the sampling reference point and the neighborhood point in the sampling area. In order to improve the accuracy and convenience of obtaining the weight vector, the weight vector can be calculated by a multilayer perceptron (MLP). In one embodiment, the target detection model includes a meta-convolution operator, and the meta-convolution operator includes a multilayer perceptron. Obtaining the weight vector of each sampling point in the sampling area based on the relative relationship may include: calculating the weight vector of each sampling point in the sampling area based on the relative relationship by the multilayer perceptron. For example, the weight vector of each sampling point in the sampling area can be calculated by the multilayer perceptron based on the relative Cartesian coordinates between the sampling reference point and the neighborhood point in the sampling area. After the corresponding weight vectors are calculated for multiple sampling areas, a weight matrix can be generated based on the weight vectors corresponding to the multiple sampling areas. The embodiment of the present application realizes the calculation of the relative Cartesian coordinates of the sampling reference point and the neighborhood point in three-dimensional space by the meta-convolution operator, and inputs the relative Cartesian coordinates into a globally shared multilayer perceptron to obtain the weight vector of each neighborhood point.

[0074] It should be noted that the calculation here involves calculating the relative relationship of the sampling reference point to all points within the sampling area. For each sampling reference point within the sampling area, its relative Cartesian coordinates are obtained. A multilayer perceptron then calculates the weight vector for that point based on these relative Cartesian coordinates. This allows the calculation of the weight vector for each sampling point within the sampling area. In addition to learning the weight vector based on the relative Cartesian coordinates of the sampling reference point and its neighboring points within the sampling area, the meta-convolution operator can also leverage additional information, such as the depth difference between sampling points or the feature vectors of those points.

[0075] After obtaining the sampling area, the feature vector of each sampling point in the sampling area can be extracted. For example, the pixel features of each sampling point in the sampling area can be extracted as a feature vector. This feature vector can include feature information of eight channels, such as depth, reflection intensity, elongation, x-coordinate, y-coordinate, z-coordinate, azimuth, and pitch angle; it can also be the features of the network's intermediate layer. It should be noted that the multi-layer perceptron of the meta-convolution operator can output a weight vector of the same length as the feature vector. Such a weight vector exists at each sampling point in the sampling area. In this way, the meta-convolution operator can only learn weight vectors of the same length as the feature vector, reducing the memory burden. At this point, the initial feature map corresponding to the depth image to be detected can be obtained based on the weight vectors and feature vectors of each sampling point in the sampling area.

[0076] In one embodiment, based on the weight vector and feature vector of each sampling point in the sampling area, obtaining the initial feature map corresponding to the depth image to be detected includes: performing vector operations on the weight vector and feature vector of each sampling point in the sampling area to obtain an intermediate vector corresponding to each position point; performing a fusion operation on multiple intermediate vectors to obtain a fusion vector; performing a convolution operation on the fusion vector to obtain an output vector, and determining the feature map corresponding to the depth image to be detected based on the output vector.

[0077] Specifically, after obtaining the weight vector and the eigenvector, a weighted product operation can be performed. For example, the weight vector and the eigenvector of each sampling point in the sampling area can be subjected to a vector operation. The vector operation can be a vector multiplication (i.e., element-wise product). j and the eigenvector f j Multiply them to get the intermediate vector, such as Figure 6 O1 to O9 in the window. Since the multilayer perceptron through the meta-convolution operator can output a weight vector of the same length as the feature vector, each sampling point in the sampling area has such a weight vector. Therefore, the feature vector at each sampling point in the window area and its corresponding weight vector can be multiplied element by element to obtain the intermediate vector.

[0078] Each sampling area can correspond to multiple intermediate vectors, and then a product aggregation operation can be performed. Specifically, multiple intermediate vectors can be fused to obtain a fused vector. In one embodiment, the fusion operation is a splicing operation or a pooling operation. For example, multiple intermediate vectors can be spliced ​​together using a meta-convolution operator (for example, they can be spliced ​​end to end in sequence). It should be noted that the product aggregation method can also be other methods, such as adding the output eigenvalues ​​(i.e., intermediate vectors) of all sampling points, or performing maximum pooling, etc.

[0079] At this time, the output vector can be obtained based on the fusion vector, for example, Figure 6As shown, the fused vector can be fed into a 1×1 convolution to obtain the final convolution result, and the convolution result is used as the output vector, which can be considered as the output vector of the sampling reference point. Each sampling area can obtain a corresponding output vector, and the initial feature map corresponding to the depth image to be detected is determined based on the output vector. For example, multiple output vectors corresponding to multiple sampling areas can be used as the initial feature map corresponding to the depth image to be detected.

[0080] This embodiment can better utilize the three-dimensional spatial information (such as relative Cartesian coordinates) on the depth image through the meta-convolution operator. For example, adding the three-dimensional coordinate relationship between the sampling reference point and the neighborhood point in the Cartesian coordinate system (i.e., relative Cartesian coordinates) is conducive to better extraction of local geometric features. The meta-convolution operator replaces the learnable parameters with feature vectors embedded with the geometric relationship between points, which can not only better extract local geometric information, but also improve the model's perception of the shape and geometric structure of local objects. That is, the meta-convolution operator uses the relative spatial coordinates between points to provide a geometric prior for the weight learning of each point, and generates a separate weight vector for each point, and performs subsequent feature fusion between points, thereby enhancing the perception of local geometric feature structures under the traditional convolution paradigm. At the same time, the accuracy of three-dimensional target detection is significantly improved while increasing the amount of parameters that need to be learned very little, which brings a significant accuracy improvement to the subsequent three-dimensional target detection based on depth images, and the accuracy improvement for small objects such as pedestrians is particularly obvious. It should be understood that any operation in the model that requires convolution can use the meta-convolution operator of this application. The input of the meta-convolution operator can be the original image input into the model, or the feature map output by the intermediate layer of the network (which can be called the intermediate feature map), so as to generate the features of the original image or the intermediate feature map. The original image can be a training set image or an image to be tested; it can be a depth image or other image. This application does not impose any specific restrictions on the input image of the meta-convolution operator.

[0081] After obtaining the initial feature map, the initial feature map can be downsampled according to a preset sampling step size to obtain multiple first sampling feature maps of different resolutions, wherein the preset sampling step size can be flexibly set according to actual needs. For example, the backbone network consists of multiple stages, each stage will perform upsampling or downsampling, and each stage of the backbone network can be composed of multiple residual networks, that is, the residual network in the backbone network can be used to perform upsampling or downsampling operations. For example, the initial feature map can be downsampled to 1 / 16 in succession according to a sampling step size of 2 through the backbone network of the initial target detection model (for example, downsampled according to step sizes Stride1, Stride2, Stride4, Stride8, and Stride16), to obtain 5 first sampling feature maps of different resolutions: Res1, Res2, Res3, Res4, and Res5. Res is a different residual block. Then, the multiple first sampling feature maps of different resolutions can be upsampled to the original resolution of the depth image to be detected to obtain multiple second sampling feature maps, and the second sampling feature maps of different resolutions are feature maps of different resolutions. It should be noted that for the depth image collected by the lidar, such as a 64×2650×8 tensor, since the height of the depth image is only 64, upsampling or downsampling can be performed only in the width direction.

[0082] S103: Detect the feature map according to the attribute information, generate multiple detection frames within the preset attribute interval corresponding to each feature map, and the classification confidence of each detection frame.

[0083] For example, Figure 9 As shown, the target detection model can be used to detect the feature map according to the attribute information to obtain multiple detection frames and the classification confidence corresponding to each detection frame. The classification confidence can be the overlap between the detection frame and the corresponding real frame of the same category. The overlap can be the intersection-over-union ratio of the detection frame and the corresponding real frame of the same category. For example, different categories can include vehicles or people.

[0084] In one embodiment, a corresponding detection frame is generated for each position point of the image to be detected, and feature maps of different resolutions are used to generate detection frames within a preset attribute range. The preset attribute range includes any one or more of a preset depth range, reflection intensity range, azimuth angle range, pitch angle range, frame size range, density range of points within the frame, and elongation range of points within the frame.

[0085] In one embodiment, the classification confidence is the degree of overlap between the detection frame and the corresponding ground-truth frame of the same category. For example, the classification confidence of a vehicle is the degree of overlap between the vehicle's detection frame and the corresponding ground-truth frame of the vehicle. For another example, the classification confidence of a pedestrian is the degree of overlap between the pedestrian's detection frame and the corresponding ground-truth frame of the pedestrian.

[0086] In one embodiment, the overlap is the intersection-over-union ratio (IoU) of the detection frame and the corresponding true frame of the same category, and the IoU is the ratio of the intersection area of ​​the two frames to the union area. For example, VFL can be used to predict the IoU training. Compared with other tasks (such as traditional 2D target detection), VFL predicts IoU. The effect of VFL on this task is very good. The reason is that: this task can be a dense prediction (each point will generate a frame), and there will be a large number of low regression quality frames (low IoU). There are many high-scoring frames in these frames. In the process of non-maximum suppression, high-quality frames are easily eliminated by these low-quality (but high-scoring, which is the classification confidence) frames. Therefore, VFL is used to predict an IoU as the classification confidence. In this way, the regression quality of the frame and its score are more consistent, and high-quality frames are more likely to be retained in non-maximum suppression.

[0087] S104: Merge multiple detection frames according to the classification confidence of each detection frame to obtain a target frame corresponding to the depth image to be detected.

[0088] In one embodiment, the classification confidence includes overlap, and multiple detection frames are merged according to the classification confidence of each detection frame to obtain a target frame corresponding to the depth image to be detected, including: filtering out detection frames with an overlap greater than a first threshold from multiple detection frames to obtain a first candidate frame; filtering out a candidate frame with the highest overlap from the first candidate frame to obtain a second candidate frame; filtering out a candidate frame with an overlap between the first candidate frame and the second candidate frame greater than a second threshold to obtain a third candidate frame; weighted processing is performed on the second candidate frame and the third candidate frame to obtain a new generated frame; the second candidate frame and the third candidate frame in the first candidate frame are eliminated, and the operation of filtering out the candidate frame with the highest overlap from the first candidate frame is returned to execute, until all the candidate frames contained in the first candidate frame are processed, at which time all the new generated frames are the target frames corresponding to the depth image to be detected.

[0089] Weighted non-maximum suppression can be used to remove redundant frames from the generated multiple detection frames and obtain target frames with better quality. Weighted non-maximum suppression further improves the quality of the detected candidate frames. Since a detection frame can be predicted for each foreground point in the depth image to be detected during the generation of multiple detection frames, a large number of detection frames will be generated, and the error between the detection frames can be a zero-mean distribution. Therefore, a large number of detection frames can be weighted averaged according to their weight coefficients to generate a higher quality target frame. Specifically, first, a detection frame with an overlap (i.e., classification confidence) greater than a first threshold can be screened out from the multiple detection frames to obtain a first candidate frame. The first threshold can be flexibly set according to actual needs, and the specific value is not limited here. If the first threshold of the detection frame is greater than the first threshold, it means that the overlap of the detection frame is high. Therefore, the candidate frame with an overlap greater than the first threshold can be screened out from the multiple detection frames for subsequent detection operations, while the candidate frame with an overlap less than or equal to the first threshold is eliminated. Then, the candidate frame with the highest overlap can be screened out from the detection frames with an overlap greater than the first threshold, and the detection frame with the highest overlap is used as the second candidate frame. The second candidate frame may include one or more frames, and the overlap degree may be an intersection-over-union ratio. For example, a second candidate frame having an intersection-over-union ratio greater than 0.5 with the first candidate frame may be screened out.

[0090] Then, the candidate frame whose overlap with the second candidate frame is greater than the second threshold can be filtered out from the first candidate frame to obtain the third candidate frame, wherein the second threshold can be flexibly set according to actual needs, and the specific value is not limited here. The second candidate frame and the third candidate frame are weighted to obtain a new generated frame, and the second candidate frame and the third candidate frame in the first candidate frame are removed. The operation of filtering out the candidate frame with the highest overlap from the first candidate frame is returned until all the candidate frames contained in the first candidate frame are processed. At this time, all the new generated frames are the target frames corresponding to the depth image to be detected. The target frame can be the area where the target object to be detected is located, and can also include the category to which the target object belongs.

[0091] It should be noted that for weighted non-maximum suppression (WNMS): the key point is that WNMS can fuse multiple detection frames to obtain a frame with better quality. The reasons why WNMS works better in this embodiment may include: (1) dense prediction (i.e., each point generates a frame) will generate a large number of detection frames, and fusing multiple detection frames can greatly reduce the error. (2) The error of the output of the target detection model relative to the true value is usually a zero-mean distribution, which provides a theoretical basis for using weighted averaging to reduce the error.

[0092] In one embodiment, weighted processing of the second candidate box and the third candidate box may include: determining the weight coefficient of each first candidate box, and performing weighted processing according to the weight coefficient of the second candidate box and the third candidate box. In one embodiment, the weight coefficient is determined according to at least one of the following parameters: classification confidence, regression confidence, distance relationship between boxes, feature relationship between boxes. Weighted processing refers to weighting the coordinate parameters of multiple boxes. Assuming that the coordinate parameters are expressed as center point coordinates and length, width and height, for example (x, y, z, w, h, l), the coordinate parameter value of the newly generated box is obtained by weighting each x, y, z, w, h, l parameter of the weighted box separately. Of course, the coordinate parameters can also be expressed in other ways, such as using multiple vertex coordinates, and this application is not limited to this.

[0093] For example, the classification confidence of the second candidate box can be used as the weight coefficient of the second candidate box, and the classification confidence of the third candidate box can be used as the weight coefficient of the third candidate box. The center position coordinates and width and height of the newly generated box = the weight coefficient of the second candidate box * the center position coordinates and width and height of the second candidate box + the weight coefficient of the third candidate box * the center position coordinates and width and height of the third candidate box.

[0094] For example, through weighted non-maximum suppression, the second candidate box (also called a high-score box) and the third candidate box (for example, a box with an intersection-over-union ratio greater than 0.5) and their weight coefficients can be weighted averaged to obtain a new higher-quality target box. The process can be expressed as follows:

[0095]

[0096] Among them, b0 can represent the second candidate box, b k Can represent the third candidate box, It can represent the newly generated box in the depth image to be detected, s k It can represent the weight coefficient of the kth box, Can represent indicator functions.

[0097] The embodiment of the present application can obtain a depth image to be detected, in which each position point of the depth image to be detected has attribute information, and perform feature extraction on the depth image to be detected to obtain feature maps of different resolutions; then the feature map can be detected according to the attribute information to generate multiple detection frames within a preset attribute interval corresponding to each feature map, as well as the classification confidence of each detection frame. At this time, the multiple detection frames can be merged to obtain a target frame corresponding to the depth image to be detected, thereby achieving efficient and accurate detection of the target frame in the depth image to be detected, and eliminating the need for multi-perspective feature fusion, thereby improving the convenience and accuracy of target detection.

[0098] In this embodiment, the object detection model training device will be described from the perspective of the object detection model training device, which can be integrated into a computer device such as a terminal or a server. Figure 10 As shown, the initial target detection model may include a meta-convolution operator and a detection network, etc. The detection network may include a backbone network (also referred to as a backbone network), a depth-based feature pyramid, and a prediction module, etc. The sample depth image can be used to perform corresponding processing in sequence through the meta-convolution operator, the backbone network, the depth-based feature pyramid, and the prediction module of the initial target detection model to train the initial target detection model (i.e., the target detection model before training) to obtain a target detection model, which will be described in detail below.

[0099] See also Figure 11 , Figure 11 : is a flowchart of a target detection model training method provided in one embodiment of the present application. The target detection model training method may include:

[0100] S201: Acquire a sample depth image, where the sample depth image includes attribute information and a ground truth box of each position point.

[0101] For example, a pre-stored depth image can be obtained from a database and used as a sample depth image. Alternatively, a depth image can be acquired by a lidar and used as a sample depth image. The sample depth image may include multiple images. The sample depth image is similar to the depth image to be detected described above and is not described in detail here.

[0102] In addition, the sample depth image may include attribute information and a true bounding box for each location point, as well as the true classification confidence of the true bounding box. The true bounding box may be the area where the target object is located in the sample depth image, and may also include the category to which the target object belongs. The shape and size of the true bounding box can be flexibly set according to actual needs. For example, the true bounding box can be a quadrilateral rectangular area. The attribute information may include at least one of depth information, spatial coordinate values, elongation, reflection intensity, azimuth, pitch angle, the size of the true bounding box, and the number of points within the true bounding box.

[0103] S202 , extracting features from the sample depth image to obtain sample feature maps of different resolutions, wherein the sample feature maps are allocated with real frames having preset attribute intervals.

[0104] For example, Figure 12 As shown, the sample depth image can be input into the initial target detection model to perform feature extraction on the sample depth image through the initial target detection model to obtain sample feature maps of different resolutions.

[0105] In one embodiment, the sample depth image is annotated with ground truth frames, and the ground truth frames in the sample feature map are mapped by: determining the attribute interval within which the attribute information of each ground truth frame lies; obtaining a mapping relationship between different attribute intervals and different sample feature maps; and assigning each ground truth frame to the corresponding sample feature map based on the mapping relationship. The preset attribute intervals include any one or more of a preset depth interval, a reflection intensity interval, an azimuth interval, a pitch angle interval, a frame size interval, a density interval of points within the frame, and an elongation interval of points within the frame.

[0106] For example, the attribute information is depth information as an example for detailed description, for example, Figure 12 As shown, target detection labels can be assigned based on a depth-based feature pyramid, that is, the real frame can be assigned to the corresponding sample feature maps of different resolutions based on the depth information, so that it can be assigned to the sample feature maps of different resolutions for prediction based on the depth information of the target object. Among them, assigning the real frame to the corresponding sample feature map can be to use the coordinates of the real frame as the label of the sample feature map. Specifically, in order to improve the convenience of mapping the real frame in the sample feature map, the target depth interval where the depth information of each position point in the sample depth image is located can be determined, and optionally the target depth interval can be determined based on the depth information of the frame where each position point is located. For example, when the depth information of the frame where the position point A is located is 6m, the target depth interval where it is located is [0m, 15m), and the mapping relationship between the pre-set different depth intervals and different feature maps is obtained. The mapping relationship can be flexibly set according to actual needs. Then, based on the mapping relationship between different depth intervals and different sample feature maps, the feature map matching the target depth interval can be determined as the sample feature map, and the real frame can be assigned to the sample feature map so that prediction can be made through the sample feature map with the real frame label.

[0107] Due to the nature of perspective projection, target objects (such as objects) appear larger when closer and smaller when farther away in depth images captured by a LiDAR. This algorithm not only considers scale variations but also considers that point clouds at different distances have different physical properties, such as density and reflection intensity. Therefore, it can assign target objects to feature maps of different resolutions based on their depth information for training and prediction. For example, target objects at [0m, 15m), [15m, 30m), and [30m, +inf) can be assigned to feature maps sampled with steps of 4, 2, and 1, respectively, for prediction. The depth-based feature pyramid alleviates the scale variation problem in depth images. In addition to alleviating the scale variation problem, it also considers the different physical properties of points at different depths, thereby improving the effectiveness and reliability of subsequent predictions of sample depth images based on the sample feature maps.

[0108] It should be noted that objects of different depth ranges can be assigned to sample feature maps of different levels through Pyramid. But in fact, classification can be based on many characteristics, such as the size of the object, reflectivity, etc. For example, in 2D detection, it can be assigned according to the size of the real detection frame. This embodiment preferably allocates based on depth information because the depth information can not only reflect the size of the object, but also reflect the different physical properties of the point cloud data at different distances. For example, the reflectivity generally decreases with increasing depth, and the point cloud density also gradually decreases.

[0109] In one embodiment, performing feature extraction on a sample depth image to obtain sample feature maps of different resolutions may include: performing feature extraction on the sample depth image to obtain an initial feature map corresponding to the sample depth image; downsampling the initial feature map according to a preset sampling step size to obtain multiple first sampling feature maps of different resolutions; and upsampling the multiple first sampling feature maps of different resolutions to the original resolution of the sample depth image to obtain multiple second sampling feature maps, where the second sampling feature maps are sample feature maps.

[0110] Specifically, first, feature extraction is performed on the sample depth image to obtain an initial feature map corresponding to the sample depth image. For example, a convolution operation can be performed on the sample depth image using the initial target detection model to extract the initial feature map corresponding to the sample depth image. The convolution operation can be performed on the sample depth image using a normal convolution operator of the initial target detection model. To improve the accuracy of target feature extraction, a meta-convolution operator can be used to convolve the sample depth image.

[0111] In one embodiment, feature extraction is performed on the sample depth image to obtain an initial feature map corresponding to the sample depth image, which may include: sampling a preset neighborhood range from the sample depth image to obtain at least one sampling area, where the position points of the sampling area include a sampling reference point and multiple neighborhood points; obtaining the relative relationship between the sampling reference point and each neighborhood point in the sampling area; obtaining a weight vector of each position point in the sampling area based on the relative relationship; extracting a feature vector of each position point in the sampling area; calculating an output vector of the sampling area based on the weight vector and feature vector of each position point in the sampling area; and generating an initial feature map corresponding to the sample depth image based on the output vector of each sampling area.

[0112] Specifically, the meta-convolution operator can be used to perform sampling, weight acquisition, weighted product, and product aggregation operations on the sample depth image to extract the initial feature map corresponding to the sample depth image. For example, the convolution operation on the sample depth image can be performed as follows: Figure 6As shown in FIG, an area within a preset neighborhood range can be sampled from a depth image to obtain a sampling area. After obtaining the sampling area, the relative relationship between the sampling reference point and the neighborhood point within the sampling area can be obtained, for example, Figure 8 As shown, the center of the sampling area can be used as the sampling reference point, and the points around the sampling reference point can be used as neighborhood points.

[0113] Then, a weight vector for each sampling point within the sampling area can be obtained based on the relative relationship between the sampling reference point and the neighboring points within the sampling area. To improve the accuracy and convenience of obtaining the weight vector, the weight vector can be calculated using a multi-layer perceptron. In one embodiment, the initial target detection model includes a meta-convolution operator, and the meta-convolution operator includes a multi-layer perceptron. Obtaining the weight vector for each sampling point within the sampling area based on the relative relationship can include: calculating the weight vector for each sampling point within the sampling area based on the relative relationship using the multi-layer perceptron. For example, the weight vector for each sampling point within the sampling area can be calculated using the multi-layer perceptron based on the relative Cartesian coordinates between the sampling reference point and the neighboring points within the sampling area. After corresponding weight vectors are calculated for multiple sampling areas, a weight matrix can be generated based on the corresponding weight vectors for the multiple sampling areas. It should be noted that for the sampling reference points within the sampling area, the relative Cartesian coordinates of the sampling reference points themselves can be obtained, and the weight vector for the sampling reference points can be calculated using the multi-layer perceptron based on the relative Cartesian coordinates. Thus, the weight vector for each sampling point within the sampling area can be calculated. After obtaining the sampling area, the feature vector of each sampling point in the sampling area can be extracted. For example, the pixel features of each sampling point in the sampling area can be extracted as a feature vector. This feature vector can include feature information of eight channels, such as depth, reflection intensity, elongation, x-coordinate, y-coordinate, z-coordinate, azimuth, and pitch angle. It can also be the features of the network's intermediate layer. It should be noted that the multi-layer perceptron of the meta-convolution operator can output a weight vector of the same length as the feature vector. Such a weight vector exists at each sampling point in the sampling area. In this way, the meta-convolution operator can only learn weight vectors of the same length as the feature vector, reducing the memory burden. At this point, the initial feature map corresponding to the sample depth image can be obtained based on the sample weight vectors and sample feature vectors of each sampling point in the sample sampling area.

[0114] In one embodiment, calculating the output vector of the sampling area based on the weight vector and the eigenvector of each sampling point in the sampling area may include: performing vector operations on the weight vector and the eigenvector of each sampling point in the sampling area to obtain an intermediate vector corresponding to each position point; performing a fusion operation on multiple intermediate vectors to obtain a fusion vector; and performing a convolution operation on the fusion vector to obtain an output vector.

[0115] Specifically, after obtaining the weight vector and the eigenvector, a weighted product operation can be performed. For example, the weight vector and the eigenvector of each sampling point in the sampling area can be subjected to a vector operation. The vector operation can be a vector multiplication (i.e., element-wise product). j and the eigenvector f j Multiply them to get the intermediate vector, such as Figure 6 O1 to O9 in the window. Since the multilayer perceptron through the meta-convolution operator can output a weight vector of the same length as the feature vector, each sampling point in the sampling area has such a weight vector. Therefore, the feature vector at each sampling point in the window area and its corresponding weight vector can be multiplied element by element to obtain the intermediate vector.

[0116] Each sampling region can correspond to multiple intermediate vectors, which can then be subjected to a product aggregation operation. Specifically, multiple intermediate vectors can be fused to obtain a fused vector. The fusion operation can be a concatenation operation or a pooling operation. For example, multiple intermediate vectors can be concatenated together using a meta-convolution operator (e.g., they can be concatenated end-to-end in sequence). It should be noted that the product aggregation method can also be other methods, such as adding the output eigenvalues ​​(i.e., intermediate vectors) of all sampling points or performing maximum pooling.

[0117] At this time, the output vector can be obtained based on the fusion vector, for example, Figure 6 As shown, the fusion vector can be sent to a 1×1 convolution to obtain the final convolution result, and the convolution result can be used as the output vector. The initial feature map corresponding to the sample depth image is determined based on the output vector. For example, multiple output vectors corresponding to multiple sampling areas can be used as the initial feature map corresponding to the sample depth image.

[0118] This embodiment can better utilize the three-dimensional spatial information (such as relative Cartesian coordinates) on the sample depth image through the meta-convolution operator, which is conducive to better extraction of local geometric features, so that the meta-convolution operator replaces the learnable parameters with feature vectors embedded with the geometric relationship between points. It can not only better extract local geometric information and improve the model's perception of local object shape and geometric structure, but also significantly improve the three-dimensional target detection accuracy with very little increase in the number of parameters that need to be learned, which brings a significant improvement in the accuracy of subsequent three-dimensional target detection based on depth images.

[0119] After obtaining the initial feature map, the initial feature map can be downsampled according to the preset sampling step to obtain multiple first sampling feature maps with different resolutions, wherein the preset sampling step can be flexibly set according to actual needs, for example, Figure 12As shown, the backbone network (also called the backbone network) consists of multiple stages, each stage will perform upsampling or downsampling, and each stage of the backbone network can be composed of multiple basic blocks (BasicBlock) in the residual network, that is, the residual network in the backbone network can be used to perform upsampling or downsampling operations. First, the initial feature map can be downsampled to 1 / 16 in succession according to a sampling step of 2 through the backbone network of the initial target detection model (that is, downsampled according to Stride1, Stride2, Stride4, Stride8 and Stride16), and 5 first sampling feature maps of different resolutions can be obtained: Res1, Res2, Res3, Res4 and Res5. Then, the multiple first sampling feature maps of different resolutions can be upsampled to the original resolution of the depth image to obtain multiple second sampling feature maps, such as Figure 12 In this example, five second-sample feature maps are obtained: Agg1, Agg2, Agg3, Agg4, and Agg5. Agg represents different aggregation blocks. It should be noted that for depth images acquired by LiDAR, such as a 64×2650×8 tensor, since the depth image has a height of only 64, upsampling or downsampling can only be performed in the width direction.

[0120] S203: Detect each sample feature map based on the attribute information, and generate a corresponding detection frame and a predicted classification confidence of each detection frame.

[0121] After obtaining the sample feature map corresponding to the sample depth image, the sample depth image can be predicted based on the sample feature map through the initial target detection model to obtain the detection frame and the predicted classification confidence of each detection frame, wherein the obtained detection frame can include multiple ones, which can be either two-dimensional frames or three-dimensional frames.

[0122] In one embodiment, the predicted classification confidence is the predicted overlap between the detection box and the corresponding real box of the same category, and the real classification confidence is the real overlap between the detection box and the corresponding real box of the same category.

[0123] In one embodiment, the predicted overlap is the predicted intersection-over-union ratio between the detection box and the corresponding real box of the same category, and the actual overlap is the actual intersection-over-union ratio between the detection box and the corresponding real box of the same category.

[0124] For example, the intersection-of-union (IoU) ratio between each generated detection box and its corresponding ground-truth box can be directly predicted as the predicted overlap, and the confidence of the predicted classification can be determined based on the predicted overlap. For example, the predicted overlap can be used as the confidence of the predicted classification. That is, the IoU ratio of the current detection box and its corresponding ground-truth box is predicted in the regression branch, and the predicted IoU ratio value is used as the confidence of the predicted classification. Since the IoU ratio is a physical quantity that directly reflects the regression accuracy, the prediction of the IoU ratio can make the classification accuracy as consistent as possible with the regression accuracy.

[0125] In one embodiment, the method for generating the predicted classification confidence of each detection frame may include: obtaining the intersection area and union area between the detection frame and the true frame, calculating the ratio between the intersection area and the union area, and obtaining the intersection-union ratio between the detection frame and the true frame; determining the predicted overlap between the detection frame and the true frame based on the intersection-union ratio, and setting the predicted overlap as the predicted classification confidence corresponding to the detection frame.

[0126] Specifically, we can obtain the intersection area A∩B and the union area A∪B between the detection box A and the true box B, calculate the ratio between the intersection area and the union area, and obtain the intersection-union ratio IOU between the detection box and the true box = (A∩B) / (A∪B).

[0127] At this time, the predicted overlap between the detection frame and the true frame can be determined based on the intersection-in-union ratio. For example, the intersection-in-union ratio can be used as the predicted overlap between the detection frame and the true frame, or the intersection-in-union ratio can be weighted to obtain the predicted overlap, and the predicted overlap can be set as the predicted classification confidence corresponding to the detection frame.

[0128] It should be noted that a core issue in detection tasks is the inconsistency between regression quality and classification quality. This problem also exists in 3D detection based on depth images collected by LiDAR. This inconsistency causes high-precision boxes to be eliminated due to low confidence in the subsequent non-maximum suppression. Therefore, the intersection-of-union (IOU) of the current detection box and its corresponding ground-truth box can be predicted on the regression branch, and the predicted IUU value is used as the predicted classification confidence. Since IUU is a physical quantity that directly reflects regression accuracy, predicting IUU can ensure that the classification accuracy is as consistent as possible with the regression accuracy. IUU prediction alleviates the problem of inconsistency between the confidence of the detection box and its regression quality.

[0129] S204: Calculate the true classification confidence between the detection frame and the corresponding true frame of the same category.

[0130] For example, the true classification confidence between the vehicle detection frame and the real frame of the corresponding vehicle type can be calculated through manual labeling or other models, or the true classification confidence between the pedestrian detection frame and the real frame of the corresponding pedestrian type can be calculated.

[0131] S205: Train the initial target detection model based on the predicted classification confidence and the true classification confidence to obtain a trained target detection model.

[0132] After obtaining the predicted classification confidence and the true classification confidence, the initial target detection model can be trained based on the predicted classification confidence and the true classification confidence. For example, the initial target detection model can be iteratively trained based on the predicted classification confidence of the detection box corresponding to multiple sample depth images and the true classification confidence of the true box until the predicted classification confidence and the true classification confidence converge, so that the parameters of the initial detection model are adjusted to appropriate values ​​to obtain the target detection model.

[0133] In one embodiment, the initial target detection model is trained based on the predicted classification confidence and the true classification confidence to obtain the trained target detection model, which may include: calculating the classification loss value between the predicted classification confidence and the true classification confidence through a classification loss function; calculating the regression loss value between the detection box and the true box of the same category through a regression loss function; determining the total loss value based on the classification loss value and the regression loss value; and training the initial target detection model based on the total loss value to obtain the trained target detection model.

[0134] For example, the classification loss value L between the predicted classification confidence and the true classification confidence can be calculated by the classification loss function iou :

[0135]

[0136] The prediction loss of the intersection-over-union (IoU) can be used as the classification loss, where q can represent the true IoU (i.e., the true classification confidence), and p can represent the predicted IoU (i.e., the predicted classification confidence). The specific values ​​of α and γ can be flexibly set according to actual needs and are not limited here.

[0137] And, calculating the regression loss value between the detection box and the real box through the regression loss function. In one embodiment, calculating the regression loss value between the detection box and the real box through the regression loss function may include: obtaining the area parameters and position parameters of the detection box; calculating the regression loss value between the detection box and the real box based on the area parameters, position parameters and the real box through the regression loss function.

[0138] The region parameters may include the center coordinates of the detection box and the length, width, and height of the detection box. The position parameters may include the sine and cosine values ​​of the detection box orientation. For example, SmoothL1 can be used as the regression loss function:

[0139]

[0140] Among them, Lreg It can represent the regression loss value, q1 can represent the detection box, and p1 can represent the ground truth box. Q contains the center coordinates, length, width, height, and sine and cosine values ​​of the detection box. The regression target is the offset from the current point to its ground truth box, including the length, width, height, and orientation of the ground truth box.

[0141] After obtaining the classification loss value and regression loss value, the total loss value can be determined based on the classification loss value and regression loss value.

[0142] In one embodiment, determining the total loss value based on the classification loss value and the regression loss value may include: adding the classification loss value and the regression loss value to obtain the total loss value; or determining a first weight value corresponding to the classification loss value and a second weight value corresponding to the regression loss value, and adding the product between the classification loss value and the first weight value and the product between the regression loss value and the second weight value to obtain the total loss value.

[0143] For example, total loss value = classification loss value L iou + regression loss value L reg , or, total loss value = first weight value * classification loss value L iou + second weight value * regression loss value L reg , wherein the specific values ​​of the first weight value and the second weight value can be flexibly set according to actual needs.

[0144] At this time, the initial target detection model can be trained according to the total loss value, and the initial target detection model can be iteratively trained according to the total loss value corresponding to multiple sample depth images until the total loss value converges, so that the parameters of the initial detection model are adjusted to appropriate values ​​to obtain the target detection model, so that the depth image to be detected can be detected through the target detection model according to the above-mentioned target detection method.

[0145] It should be noted that in addition to training the target detection model according to the regression branch and the classification branch in the above manner, the regression confidence can also be predicted through the regression branch of the target detection model, and the initial target detection model can be trained according to the predicted regression confidence and the true regression confidence to obtain the trained target detection model.

[0146] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the target detection method above, and will not be repeated here.

[0147] The embodiment of the present application can obtain a sample depth image, which includes attribute information and a real frame of each position point, perform feature extraction on the sample depth image, and obtain sample feature maps of different resolutions, in which a real frame with a preset attribute interval is assigned to the sample feature map; detect each sample feature map based on the attribute information, generate a corresponding detection frame and a predicted classification confidence of each detection frame; calculate the real classification confidence of the detection frame and the corresponding real frame of the same category; and train the initial target detection model based on the predicted classification confidence and the real classification confidence to obtain a trained target detection model, thereby improving the accuracy and reliability of the target detection model training.

[0148] Based on the methods described in the above embodiments, the following describes an exemplary application of the embodiments of the present application in a practical application scenario, such as an unmanned vehicle application scenario. This embodiment is described in detail using the example of integrating a target detection device and a target detection model training device into an unmanned vehicle. The unmanned vehicle may be equipped with a laser radar for acquiring depth images, a target detection model, and the like.

[0149] 1. Model training.

[0150] First, the autonomous vehicle can train an initial target detection model using the aforementioned target detection model training method to obtain a target detection model. For example, the autonomous vehicle can use a lidar to collect multiple depth images and annotate the two-dimensional or three-dimensional true bounding boxes (LRBs) of target objects such as people, vehicles, buildings, plants, and animals, along with the true classification confidence of these LRBs, on the depth images containing the depth information of each location. The multiple depth images can then be used as sample depth images.

[0151] The unmanned vehicle can then perform operations such as sampling, weight acquisition, weighted product, and product aggregation on the sample depth image using the meta-convolution operator of the initial target detection model to extract a feature map corresponding to the sample depth image. For example, the meta-convolution operator can be used to sample an area within a preset neighborhood from the sample depth image to obtain the sampling area, and the relative Cartesian coordinates between the sampling reference point and the neighborhood points within the sampling area can be obtained. The multi-layer perceptron of the meta-convolution operator can be used to obtain the weight vector of each sampling point in the sampling area based on the relative Cartesian coordinates, and the feature vector of each sampling point in the sampling area can be extracted. At this time, the feature map corresponding to the sample depth image can be obtained based on the weight vector and feature vector of each sampling point in the sampling area.

[0152] Secondly, the unmanned vehicle can downsample the feature map according to a preset sampling step size to obtain multiple first sampling feature maps of different resolutions, and upsample the multiple first sampling feature maps of different resolutions to the original resolution of the sample depth image to obtain multiple second sampling feature maps; based on the depth information of each position point in the sample depth image, the real frame is assigned to the corresponding second sampling feature map, and the second sampling feature map after the real frame is assigned is used as the target feature map corresponding to the sample depth image, so that it can be assigned to the feature maps of different resolutions for prediction according to the depth information of the target object.

[0153] At this point, the autonomous vehicle can use the initial target detection model to predict the sample depth image based on the target feature map, obtaining at least one 2D or 3D detection box. The predicted IoU ratio between the detection box and the ground-truth box is then calculated and used as the confidence level of the predicted classification for the detection box.

[0154] Finally, the unmanned vehicle can train the initial target detection model based on the predicted classification confidence and the true classification confidence to obtain a target detection model. For example, the classification loss value between the predicted classification confidence and the true classification confidence (i.e., the classification loss value between the predicted intersection-over-union ratio and the true intersection-over-union ratio) can be calculated through a classification loss function, and the regression loss value between the detection box and the true box can be calculated through a regression loss function. The total loss value is determined based on the classification loss value and the regression loss value. The initial target detection model is iteratively trained based on the total loss value corresponding to multiple sample depth images until the total loss value converges, so that the parameters of the initial detection model are adjusted to appropriate values, and the target detection model is obtained. The depth image to be detected is detected by the target detection model, thereby improving the accuracy and reliability of the target detection model training.

[0155] 2. Application of the model.

[0156] Unmanned vehicles can use lidar to collect depth images as depth images to be detected, extract feature maps corresponding to the depth images to be detected through the meta-convolution operator of the target detection model, and detect the depth images to be detected based on the feature maps to obtain multiple two-dimensional or three-dimensional candidate boxes (i.e., detection boxes) and the classification confidence corresponding to each candidate box.

[0157] Then, the unmanned vehicle can use the weighted non-maximum suppression algorithm to filter out detection frames with a degree of overlap greater than a first threshold from multiple detection frames to obtain a first candidate frame; filter out the candidate frame with the highest degree of overlap from the first candidate frame to obtain a second candidate frame, and filter out the candidate frame with a degree of overlap between the first candidate frame and the second candidate frame greater than a second threshold to obtain a third candidate frame; perform weighted processing on the second candidate frame and the third candidate frame to obtain a new generated frame; remove the second candidate frame and the third candidate frame from the first candidate frame, and return to execute the operation of filtering out the candidate frame with the highest degree of overlap from the first candidate frame, until all the candidate frames contained in the first candidate frame are processed. At this time, all new generated frames are the target frames corresponding to the depth image to be detected, and the target frame is the location of the detected target object, which improves the accuracy and convenience of target detection.

[0158] For example, Figure 13 As shown, the unmanned vehicle can quickly and accurately detect obstacles (i.e., target objects) around the driving section based on the collected depth images, and avoid obstacles in a timely manner. In the autonomous driving scenario, the unmanned vehicle can obtain depth images through the lidar, and directly and efficiently detect the precise position of objects in the environment from the depth images, and realize the unmanned vehicle's large-scale perception of the environment, thereby improving the safety of autonomous driving.

[0159] To facilitate better implementation of the target detection method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above target detection method. The meanings of the terms are the same as those in the above target detection method, and the specific implementation details can be referred to the description in the method embodiment.

[0160] See also Figure 14 , Figure 14 This is a structural diagram of a target detection device 300 provided in an embodiment of the present application, wherein the target detection device 300 may include a first acquisition module 301, a first extraction module 302, a first detection module 303, and a merging module 304, etc.

[0161] The first acquisition module 301 is used to acquire a depth image to be detected, and each position point of the depth image to be detected has attribute information.

[0162] The first extraction module 302 is used to extract features from the depth image to be detected to obtain feature maps with different resolutions.

[0163] The first detection module 303 is used to detect the feature map according to the attribute information, generate multiple detection boxes within the preset attribute interval corresponding to each feature map, and the classification confidence of each detection box.

[0164] The merging module 304 is used to merge multiple detection frames based on the classification confidence of each detection frame to obtain a target frame corresponding to the depth image to be detected.

[0165] In one embodiment, the attribute information includes at least one of depth information, spatial coordinate value, elongation, reflection intensity, azimuth angle, pitch angle, size of the real frame, and number of points in the real frame.

[0166] In one embodiment, a corresponding detection frame is generated for each position point, and feature maps of different resolutions are used to generate detection frames within a preset attribute range. The preset attribute range includes any one or more of a preset depth range, reflection intensity range, azimuth angle range, pitch angle range, frame size range, density range of points within the frame, and elongation range of points within the frame.

[0167] In one embodiment, the classification confidence is the degree of overlap between the detection box and the corresponding ground-truth box of the same category.

[0168] In one embodiment, the overlap is the intersection-over-union ratio between the detection box and the corresponding ground-truth box of the same category.

[0169] In one embodiment, the classification confidence includes overlap, and the merging module 304 can be specifically used to: filter out detection frames with an overlap greater than a first threshold from multiple detection frames to obtain a first candidate frame; filter out the candidate frame with the highest overlap from the first candidate frame to obtain a second candidate frame; filter out the candidate frame with an overlap between the first candidate frame and the second candidate frame greater than a second threshold to obtain a third candidate frame; perform weighted processing on the second candidate frame and the third candidate frame to obtain a new generated frame; remove the second candidate frame and the third candidate frame from the first candidate frame, and return to execute the operation of filtering out the candidate frame with the highest overlap from the first candidate frame, until all the candidate frames contained in the first candidate frame are processed, at which time all the new generated frames are the target frames corresponding to the depth image to be detected.

[0170] In one embodiment, the merging module 304 may be specifically configured to determine a weight coefficient of each first candidate box, and perform weighted processing according to the weight coefficients of the second candidate box and the third candidate box.

[0171] In one embodiment, the weight coefficient is determined according to at least one of the following parameters: classification confidence, regression confidence, distance relationship between boxes, and feature relationship between boxes.

[0172] In one embodiment, the first extraction module 302 can be specifically used to: perform feature extraction on the depth image to be detected to obtain a corresponding initial feature map; downsample the initial feature map according to a preset sampling step size to obtain multiple first sampling feature maps with different resolutions; upsample the multiple first sampling feature maps with different resolutions to the original resolution of the sample depth image to obtain multiple second sampling feature maps with different resolutions.

[0173] In one embodiment, the first extraction module 302 can be specifically used to: sample an area within a preset neighborhood range from the depth image to be detected to obtain at least one sampling area, where the position points of the sampling area include a sampling reference point and multiple neighborhood points; calculate the relative relationship between the sampling reference point and the neighborhood point in the sampling area; calculate the weight vector of each sampling point in the sampling area based on the relative relationship; extract the feature vector of each sampling point in the sampling area; calculate the output vector of the sampling area based on the weight vector and feature vector of each sampling point in the sampling area; and generate an initial feature map corresponding to the depth image to be detected based on the output vector of each sampling area.

[0174] In one embodiment, the first extraction module 302 can be specifically used to: perform vector operations on the weight vectors and feature vectors of each sampling point in the sampling area to obtain an intermediate vector corresponding to each position point; perform a fusion operation on multiple intermediate vectors to obtain a fusion vector; and perform a convolution operation on the fusion vector to obtain an output vector.

[0175] In one embodiment, the relative relationship includes at least one of a relative position relationship and a relative characteristic relationship between the sampling reference point and each position point in the sampling area;

[0176] Relative positional relationships include relative Cartesian coordinates;

[0177] Relative feature relationships include the association relationships between attribute features;

[0178] The weight vector of each position point in the sampling area is calculated by a multi-layer perceptron;

[0179] The fusion operation is a concatenation operation or a pooling operation.

[0180] In the embodiment of the present application, the first acquisition module 301 can acquire a depth image to be detected, where each position point of the depth image to be detected has attribute information, and the first extraction module 302 can perform feature extraction on the depth image to be detected to obtain feature maps of different resolutions; then the first detection module 303 can detect the feature map according to the attribute information, and generate multiple detection frames within a preset attribute interval corresponding to each feature map, as well as the classification confidence of each detection frame. At this time, the merging module 304 can merge the multiple detection frames to obtain a target frame corresponding to the depth image to be detected, thereby achieving efficient and accurate detection of the target frame in the depth image to be detected, and eliminating the need for multi-perspective feature fusion, thereby improving the convenience and accuracy of target detection.

[0181] To facilitate better implementation of the target detection model training method provided in the embodiments of the present application, the embodiments of the present application also provide a device based on the target detection model training method. The meanings of the terms herein are the same as those in the target detection model training method described above, and the specific implementation details can be referred to the description in the method embodiment.

[0182] See also Figure 15 , Figure 15 A structural diagram of a target detection model training device 400 provided in an embodiment of the present application, wherein the target detection model training device 400 may include a second acquisition module 401, a second extraction module 402, a second detection module 403, a calculation module 404, and a training module 405, etc.

[0183] The second acquisition module 401 is used to acquire a sample depth image, where the sample depth image includes attribute information and a real frame of each position point.

[0184] The second extraction module 402 is configured to perform feature extraction on the sample depth image to obtain sample feature maps of different resolutions, wherein the sample feature maps are allocated with real frames of preset attribute intervals.

[0185] The second detection module 403 is used to detect each sample feature map based on the attribute information, and generate a corresponding detection box and a predicted classification confidence of each detection box.

[0186] The calculation module 404 is used to calculate the true classification confidence of the detection box and the corresponding true box of the same category.

[0187] The training module 405 is used to train the initial target detection model based on the predicted classification confidence and the true classification confidence to obtain a trained target detection model.

[0188] In one embodiment, the sample depth image is annotated with a real frame, and the real frame in the sample feature map is mapped in the following manner: determining the attribute interval where the attribute information of each real frame is located; obtaining the mapping relationship between different attribute intervals and different sample feature maps; and assigning each real frame to the corresponding sample feature map according to the mapping relationship.

[0189] In one embodiment, the predicted classification confidence is the predicted overlap between the detection box and the corresponding real box of the same category, and the real classification confidence is the real overlap between the detection box and the corresponding real box of the same category.

[0190] In one embodiment, the predicted overlap is the predicted intersection-over-union (IoU) ratio between the detection box and the corresponding ground-truth box of the same category, and the true overlap is the true IoU ratio between the detection box and the corresponding ground-truth box of the same category.

[0191] In one embodiment, the training module 405 can be specifically used to: calculate the classification loss value between the predicted classification confidence and the true classification confidence through the classification loss function; calculate the regression loss value between the detection box and the true box of the same category through the regression loss function; determine the total loss value based on the classification loss value and the regression loss value; train the initial target detection model based on the total loss value to obtain a trained target detection model.

[0192] In one embodiment, the training module 405 can be specifically used to: obtain the region parameters and position parameters of the detection box; and calculate the regression loss value between the detection box and the real box of the same category based on the region parameters, position parameters and reference box through a regression loss function.

[0193] In one embodiment, the training module 405 can be specifically used to: accumulate the classification loss value and the regression loss value to obtain a total loss value; or, determine a first weight value corresponding to the classification loss value and a second weight value corresponding to the regression loss value, and accumulate the product between the classification loss value and the first weight value and the product between the regression loss value and the second weight value to obtain a total loss value.

[0194] In one embodiment, the second extraction module 402 can be specifically used to: perform feature extraction on the sample depth image to obtain an initial feature map corresponding to the sample depth image; downsample the initial feature map according to a preset sampling step size to obtain multiple first sampling feature maps of different resolutions; upsample the multiple first sampling feature maps of different resolutions to the original resolution of the sample depth image to obtain multiple second sampling feature maps, where the second sampling feature maps are the sample feature maps.

[0195] In one embodiment, the second extraction module 402 can be specifically used to: sample a preset neighborhood range from the sample depth image to obtain at least one sampling area, where the position points of the sampling area include a sampling reference point and multiple neighborhood points; obtain the relative relationship between the sampling reference point and each neighborhood point in the sampling area; obtain the weight vector of each position point in the sampling area based on the relative relationship; extract the feature vector of each position point in the sampling area; calculate the output vector of the sampling area based on the weight vector and feature vector of each position point in the sampling area; and generate an initial feature map corresponding to the sample depth image based on the output vector of each sampling area.

[0196] In one embodiment, the second extraction module 402 can be specifically used to: perform vector operations on the weight vectors and feature vectors of each sampling point in the sampling area to obtain an intermediate vector corresponding to each position point; perform a fusion operation on multiple intermediate vectors to obtain a fusion vector; and perform a convolution operation on the fusion vector to obtain an output vector.

[0197] In an embodiment of the present application, a second acquisition module 401 can acquire a sample depth image, where the sample depth image includes attribute information and a real frame at each position point. A second extraction module 402 performs feature extraction on the sample depth image to obtain sample feature maps of different resolutions, where a real frame with a preset attribute interval is allocated to the sample feature map. A second detection module 403 detects each sample feature map based on the attribute information to generate a corresponding detection frame and a predicted classification confidence of each detection frame. A calculation module 404 calculates the real classification confidence of the detection frame and the corresponding real frame of the same category. A training module 405 trains the initial target detection model based on the predicted classification confidence and the real classification confidence to obtain a trained target detection model, thereby improving the accuracy and reliability of the target detection model training.

[0198] The embodiment of the present application also provides a computer device, which may be a terminal or a server, etc. Figure 16 , which shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0199] The computer device may include one or more processing core processors 501, one or more computer readable storage media memories 502, a power supply 503, an input unit 504 and other components. Those skilled in the art will understand that Figure 16 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0200] Processor 501 is the control center of the computer device. It connects the various components of the entire computer device using various interfaces and lines. By running or executing software programs and / or modules stored in memory 502 and accessing data stored in memory 502, it performs various functions of the computer device and processes data, thereby providing overall monitoring of the computer device. Optionally, processor 501 may include one or more processing cores. Preferably, processor 501 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 501.

[0201] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0202] The computer device also includes a power supply 503 for supplying power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 503 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0203] The computer device may further include an input unit 504 , which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0204] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the computer device will load the executable files corresponding to one or more application processes into the memory 502 according to the following instructions, and the processor 501 will run the application stored in the memory 502 to implement various functions as follows:

[0205] Training the target detection model: obtaining a sample depth image, which includes attribute information and a true frame of each position point; performing feature extraction on the sample depth image to obtain sample feature maps of different resolutions, in which a true frame with a preset attribute interval is assigned; detecting each sample feature map based on the attribute information to generate the corresponding detection frame and the predicted classification confidence of each detection frame; calculating the true classification confidence of the detection frame and the corresponding true frame of the same category; and training the initial target detection model based on the predicted classification confidence and the true classification confidence to obtain a trained target detection model.

[0206] After obtaining the target detection model, the depth image to be detected can be obtained, and each position point of the depth image to be detected has attribute information; feature extraction is performed on the depth image to be detected to obtain feature maps of different resolutions; the feature map is detected according to the attribute information, and multiple detection frames within the preset attribute interval corresponding to each feature map and the classification confidence of each detection frame are generated; the classification confidence of each detection frame is used to merge multiple detection frames to obtain the target frame corresponding to the depth image to be detected.

[0207] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the target detection method above, and will not be repeated here.

[0208] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the above embodiments.

[0209] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by computer instructions, or by controlling related hardware through computer instructions. The computer instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium having a computer program stored therein. The computer program may include computer instructions, and the computer program can be loaded by a processor to execute any of the target detection methods provided in the embodiments of the present application, as well as any of the target detection model training methods provided in the embodiments of the present application.

[0210] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0211] The storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0212] Since the instructions stored in the storage medium can execute any target detection method provided in the embodiments of the present application, and execute any target detection model training method provided in the embodiments of the present application, the beneficial effects that can be achieved by any target detection method and target detection model training method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0213] The above is a detailed introduction to a target detection method, a target detection model training method, a target detection device, a target detection model training device, a computer device and a storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A target detection method, characterized in that: include: Acquire a depth image to be detected, wherein each position point of the depth image to be detected has attribute information; Performing feature extraction on the depth image to be detected to obtain feature maps of different resolutions; Detecting the feature map according to the attribute information, generating a plurality of detection frames within a preset attribute interval corresponding to each feature map, and a classification confidence of each detection frame; Merging the multiple detection frames to obtain a target frame corresponding to the depth image to be detected; The feature extraction of the depth image to be detected to obtain feature maps of different resolutions includes: Sampling an area within a preset neighborhood range from the depth image to be detected to obtain at least one sampling area, wherein the position points of the sampling area include a sampling reference point and a plurality of neighborhood points; Calculate the relative relationship between the sampling reference point and the neighboring points in the sampling area; Calculating a weight vector of each sampling point in the sampling area based on the relative relationship using a multilayer perceptron; Extracting the feature vector of each sampling point in the sampling area; Perform vector operations on the weight vectors and feature vectors of each sampling point in the sampling area to obtain an intermediate vector corresponding to each position point; A fusion operation is performed on the multiple intermediate vectors to obtain an output vector of the sampling area.

2. The method according to claim 1, characterized in that The attribute information includes at least one of depth information, spatial coordinate value, elongation, reflection intensity, azimuth angle, pitch angle, size of the real frame, and number of points in the real frame.

3. The method according to claim 1, wherein A corresponding detection frame is generated for each position point, and feature maps of different resolutions are used to generate detection frames within a preset attribute range. The preset attribute range includes any one or more of a preset depth range, reflection intensity range, azimuth angle range, pitch angle range, frame size range, density range of points within the frame, and elongation range of points within the frame.

4. The method according to claim 1, wherein The classification confidence is the degree of overlap between the detection box and the corresponding real box of the same category.

5. The method according to claim 4, wherein The overlap is the intersection-over-union ratio between the detection frame and the corresponding ground-truth frame of the same category.

6. The method according to claim 1, characterized in that The classification confidence includes the overlap, and the merging of the multiple detection frames to obtain the target frame corresponding to the depth image to be detected includes: Filtering out detection frames whose overlap is greater than a first threshold from the multiple detection frames to obtain a first candidate frame; Filtering the candidate frame with the highest overlap from the first candidate frames to obtain a second candidate frame; Filtering out, from the first candidate frames, candidate frames whose overlap with the second candidate frame is greater than a second threshold, to obtain a third candidate frame; Performing weighted processing on the second candidate box and the third candidate box to obtain a new generated box; Eliminate the second candidate frame and the third candidate frame from the first candidate frame, and return to the operation of filtering out the candidate frame with the highest overlap from the first candidate frame until all the candidate frames contained in the first candidate frame are processed. At this time, all newly generated frames are the target frames corresponding to the depth image to be detected.

7. The method according to claim 6, characterized in that Performing weighted processing on the second candidate box and the third candidate box, including: A weight coefficient of each first candidate box is determined, and weighted processing is performed according to the weight coefficients of the second candidate box and the third candidate box.

8. The method according to claim 7, wherein the weight coefficient is determined based on at least one of the following parameters: classification confidence, regression confidence, distance relationship between boxes, and feature relationship between boxes.

9. The method according to claim 1, characterized in that The feature extraction of the depth image to be detected to obtain feature maps of different resolutions includes: Performing feature extraction on the depth image to be detected to obtain a corresponding initial feature map; Downsampling the initial feature map according to a preset sampling step size to obtain a plurality of first sampling feature maps with different resolutions; The multiple first sampling feature maps with different resolutions are respectively upsampled to the original resolution of the depth image to be detected to obtain multiple second sampling feature maps with different resolutions.

10. The method according to claim 9, characterized in that Perform feature extraction on the depth image to be detected to obtain a corresponding initial feature map, including: An initial feature map corresponding to the depth image to be detected is generated according to the output vectors of each sampling area.

11. The method according to claim 10, characterized in that The fusing operation on the multiple intermediate vectors includes: Performing a fusion operation on the multiple intermediate vectors to obtain a fused vector; A convolution operation is performed on the fused vector to obtain the output vector.

12. The method according to claim 10, characterized in that The relative relationship includes at least one of a relative position relationship and a relative characteristic relationship between the sampling reference point and each position point in the sampling area; The relative position relationship includes relative Cartesian coordinates; The relative feature relationship includes the association relationship between the attribute features; The weight vector of each position point in the sampling area is calculated by a multi-layer perceptron; The fusion operation is a splicing operation or a pooling operation; The sampling reference point is the center point of the sampling area.

13. A target detection model training method, characterized in that: include: Acquire a sample depth image, wherein the sample depth image includes attribute information and a true frame of each position point; Performing feature extraction on the sample depth image to obtain sample feature maps of different resolutions, wherein the sample feature maps are assigned real frames with preset attribute intervals; Detecting each sample feature map based on the attribute information, generating a corresponding detection frame and a predicted classification confidence of each detection frame; Calculate the true classification confidence of the detection frame and the corresponding true frame of the same category; as well as Training the initial target detection model based on the predicted classification confidence and the true classification confidence to obtain a trained target detection model; The feature extraction of the sample depth image to obtain sample feature maps of different resolutions includes: Sampling an area within a preset neighborhood range from the sample depth image to obtain at least one sampling area, wherein the position points of the sampling area include a sampling reference point and a plurality of neighborhood points; Calculate the relative relationship between the sampling reference point and the neighboring points in the sampling area; Calculating a weight vector of each sampling point in the sampling area based on the relative relationship using a multilayer perceptron; Extracting the feature vector of each sampling point in the sampling area; Perform vector operations on the weight vectors and feature vectors of each sampling point in the sampling area to obtain an intermediate vector corresponding to each position point; A fusion operation is performed on the multiple intermediate vectors to obtain an output vector of the sampling area.

14. The method according to claim 13, characterized in that The sample depth image is annotated with a true box, and the true box in the sample feature map is mapped in the following way: Determine the attribute interval where the attribute information of each real frame is located; Obtain the mapping relationship between different attribute intervals and different sample feature maps; Each real frame is assigned to the corresponding sample feature map according to the mapping relationship.

15. The method according to claim 13, characterized in that The predicted classification confidence is the predicted overlap between the detection box and the corresponding real box of the same category, and the real classification confidence is the real overlap between the detection box and the corresponding real box of the same category.

16. The method according to claim 15, characterized in that The predicted overlap is the predicted intersection-over-union ratio of the detection box and the corresponding real box of the same category, and the real overlap is the real intersection-over-union ratio of the detection box and the corresponding real box of the same category.

17. The method according to claim 13, wherein The initial target detection model is trained based on the predicted classification confidence and the true classification confidence to obtain a trained target detection model, including: Calculating a classification loss value between the predicted classification confidence and the true classification confidence by a classification loss function; Calculate the regression loss value between the detection frame and the real frame of the same category through the regression loss function; Determine a total loss value based on the classification loss value and the regression loss value; The initial target detection model is trained according to the total loss value to obtain a trained target detection model.

18. The method according to claim 17, characterized in that Calculating the regression loss value between the detection frame and the real frame of the same category by using the regression loss function includes: Obtaining the region parameters and position parameters of the detection frame; A regression loss value between the detection box and the real box of the same category is calculated based on the region parameter, the position parameter and the reference box through a regression loss function.

19. The method according to claim 17, wherein Determining the total loss value according to the classification loss value and the regression loss value includes: Accumulate the classification loss value and the regression loss value to obtain a total loss value; or, Determine a first weight value corresponding to the classification loss value and a second weight value corresponding to the regression loss value, and add the product between the classification loss value and the first weight value and the product between the regression loss value and the second weight value to obtain a total loss value.

20. The method according to any one of claims 13 to 19, characterized in that Feature extraction is performed on the sample depth image to obtain sample feature maps of different resolutions, including: Performing feature extraction on the sample depth image to obtain an initial feature map corresponding to the sample depth image; Downsampling the initial feature map according to a preset sampling step size to obtain a plurality of first sampling feature maps with different resolutions; The multiple first sampling feature maps with different resolutions are respectively upsampled to the original resolution of the sample depth image to obtain multiple second sampling feature maps, where the second sampling feature maps are the sample feature maps.

21. The method according to claim 20, characterized in that The extracting features from the sample depth image to obtain an initial feature map corresponding to the sample depth image includes: An initial feature map corresponding to the sample depth image is generated according to the output vector of each sampling area.

22. The method according to claim 20, characterized in that The fusing operation on the multiple intermediate vectors to obtain the output vector of the sampling area includes: Performing a fusion operation on the multiple intermediate vectors to obtain a fused vector; A convolution operation is performed on the fused vector to obtain an output vector.

23. A target detection device, characterized in that: include: A first acquisition module is used to acquire a depth image to be detected, where each position point of the depth image to be detected has attribute information; A first extraction module is used to extract features from the depth image to be detected to obtain feature maps of different resolutions; a first detection module, configured to detect the feature map according to the attribute information, and generate a plurality of detection frames within a preset attribute interval corresponding to each feature map, and a classification confidence of each detection frame; A merging module, configured to merge the multiple detection frames to obtain a target frame corresponding to the depth image to be detected; Wherein, the first extraction module is further used for: Sampling an area within a preset neighborhood range from the depth image to be detected to obtain at least one sampling area, wherein the position points of the sampling area include a sampling reference point and a plurality of neighborhood points; Calculate the relative relationship between the sampling reference point and the neighboring points in the sampling area; Calculating a weight vector of each sampling point in the sampling area based on the relative relationship using a multilayer perceptron; Extracting the feature vector of each sampling point in the sampling area; Perform vector operations on the weight vectors and feature vectors of each sampling point in the sampling area to obtain an intermediate vector corresponding to each position point; A fusion operation is performed on the multiple intermediate vectors to obtain an output vector of the sampling area.

24. A target detection model training device, characterized in that: include: A second acquisition module is used to acquire a sample depth image, wherein the sample depth image includes attribute information and a real frame of each position point; A second extraction module is configured to perform feature extraction on the sample depth image to obtain sample feature maps of different resolutions, wherein the sample feature maps are assigned real frames with preset attribute intervals; A second detection module is used to detect each sample feature map based on the attribute information, generate a corresponding detection box and a predicted classification confidence of each detection box; A calculation module is used to calculate the true classification confidence of the detection frame and the corresponding true frame of the same category; as well as A training module, configured to train the initial target detection model based on the predicted classification confidence and the true classification confidence to obtain a trained target detection model; Wherein, the second extraction module is further used for: Sampling an area within a preset neighborhood range from the sample depth image to obtain at least one sampling area, wherein the position points of the sampling area include a sampling reference point and a plurality of neighborhood points; Calculate the relative relationship between the sampling reference point and the neighboring points in the sampling area; Calculating a weight vector of each sampling point in the sampling area based on the relative relationship using a multilayer perceptron; Extracting the feature vector of each sampling point in the sampling area; Perform vector operations on the weight vectors and feature vectors of each sampling point in the sampling area to obtain an intermediate vector corresponding to each position point; A fusion operation is performed on the multiple intermediate vectors to obtain an output vector of the sampling area.

25. A computer device, characterized in that: It includes a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, it executes the target detection method according to any one of claims 1 to 12, or executes the target detection model training method according to any one of claims 13 to 22.

26. A storage medium, characterized in that The storage medium is used to store a computer program, which is loaded by a processor to execute the target detection method described in any one of claims 1 to 12, or to execute the target detection model training method described in any one of claims 13 to 22.

Images