Model training method, target detection method, electronic equipment, medium and product

Abstract

The invention discloses a model training method, a target detection method, electronic equipment, a medium and a product, and the method comprises the steps: obtaining a training sample which comprises point cloud data and a target image which are obtained through shooting a preset region; point cloud in the point cloud data is projected to the target image, a plurality of projection points are obtained, and each projection point is a pixel point in the target image; the pixel value of a pixel point belonging to the projection point in the target image is set as a depth truth value, the pixel value of a pixel point not belonging to the projection point in the target image is set as a preset value, a depth truth value image is obtained, and the depth truth value is obtained through calculation according to the point cloud corresponding to the pixel point; the depth truth value image and the target image are adopted to train a perception model, a target perception model is obtained, the target perception model is used for target detection, and through the mode, the accuracy of the target perception model can be improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence, and in particular relates to a model training method, a target detection method, an electronic device, a medium and a product. Background Art

[0002] With the rapid development of autonomous driving technology, object detection is an important part of autonomous driving technology, and its accuracy is crucial to the safety of autonomous driving. Existing object detection models mainly rely on the depth truth value provided by point clouds for supervision to detect objects in images; however, in actual applications, there is a problem that the point cloud density is insufficient, resulting in less depth truth data, which in turn affects the accuracy of the object detection model's detection results. Summary of the invention

[0003] The embodiments of the present application provide a model training method, a target detection method, an electronic device, a medium and a product, which can interpolate based on existing point cloud data to obtain more point cloud data, thereby increasing the depth truth data to improve the accuracy of the target detection model.

[0004] In a first aspect, an embodiment of the present application provides a model training method, the method comprising:

[0005] Acquire training samples, where the training samples include point cloud data and target images obtained by photographing a preset area;

[0006] Projecting the point cloud in the point cloud data onto the target image to obtain a plurality of projection points, each of which is a pixel point in the target image;

[0007] Setting the pixel values ​​of the pixel points in the target image that belong to the projection point as the depth true value, and setting the pixel values ​​of the pixel points in the target image that do not belong to the projection point as the preset value, to obtain a depth true value image, wherein the depth true value is obtained based on the point cloud computing corresponding to the pixel points;

[0008] The perception model is trained using the true depth image and the target image to obtain a target perception model, and the target perception model is used for target detection.

[0009] In a second aspect, an embodiment of the present application provides a target detection method, the method comprising:

[0010] Acquire the image to be detected;

[0011] The image to be detected is input into a target perception model to obtain a detection target, and the target perception model is obtained by using the model training method described in the first aspect.

[0012] In a third aspect, an embodiment of the present application provides a model training device, the device comprising:

[0013] An acquisition module, used to acquire training samples, wherein the training samples include point cloud data and target images obtained by photographing a preset area;

[0014] A projection module, used for projecting the point cloud in the point cloud data onto the target image to obtain a plurality of projection points, each of which is a pixel point in the target image;

[0015] An assignment module, used to set the pixel values ​​of the pixel points in the target image that belong to the projection point as the depth true value, and set the pixel values ​​of the pixel points in the target image that do not belong to the projection point as preset values, to obtain a depth true value image, wherein the depth true value is obtained based on the point cloud computing corresponding to the pixel points;

[0016] A training module is used to train a perception model using the depth truth image and the target image to obtain a target perception model, and the target perception model is used for target detection.

[0017] In a fourth aspect, an embodiment of the present application provides a target detection device, the device comprising:

[0018] An image acquisition module, used for acquiring an image to be detected;

[0019] The target detection module is used to input the image to be detected into the target perception model to obtain the detection target, and the target perception model is obtained using the model training method described in the first aspect.

[0020] In a fifth aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory storing computer program instructions;

[0021] When the processor executes the computer program instructions, it implements the model training method as described in the first aspect and / or the target detection method as described in the second aspect.

[0022] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the model training method as described in the first aspect and / or the target detection method as described in the second aspect are implemented.

[0023] In the seventh aspect, the embodiments of the present application may be implemented by providing a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the model training method as described in the first aspect and / or the target detection method as described in the second aspect.

[0024] The model training method, target detection method, electronic device, medium and product proposed in the present application obtain training samples, project the point cloud data in the training samples to the target image, obtain multiple projection points, set the pixel values ​​of the pixel points belonging to the projection points in the target image as the depth true value, and set the pixel values ​​of the pixel points that do not belong to the projection points as preset values, fully utilize the depth information of the point cloud data to construct a depth true value image, train the perception model through the depth true value image and the target image, better guide the perception model to learn to obtain the target detection model, thereby improving the detection accuracy of the target detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 It is a flow chart of the model training method provided in the embodiment of the present application;

[0027] Figure 2 is a schematic diagram of pixel distribution around a candidate point provided in an embodiment of the present application;

[0028] Figure 3 It is a flowchart of the training of the Bevdepth model provided in the embodiment of the present application;

[0029] Figure 4 This is a flow chart of a target detection method provided in an embodiment of the present application.

[0030] Figure 5 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0031] Figure 6 is a schematic diagram of the structure of a target detection device provided in an embodiment of the present application;

[0032] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0034] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0035] In each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the disclosed embodiment will be obtained.

[0036] In order to solve the problems of the prior art, the embodiments of the present application provide a model training method, device, electronic device and storage medium. The model training method provided by the embodiments of the present application is first introduced below.

[0037] Figure 1 FIG. 1 is a flow chart of a model training method provided by an embodiment of the present application. Figure 1 As shown, the model training method provided in the embodiment of the present application is applied to an electronic device and includes the following steps 101 to 104, wherein:

[0038] Step 101: Acquire training samples, where the training samples include point cloud data and target images obtained by photographing a preset area.

[0039] The preset area can be selected according to actual needs. For example, the preset area is an area in a scene where the autonomous driving bird's eye view (Bird's Eye View, BEV) is applied, which can be a traffic intersection, a road surface, etc.

[0040] Point cloud data is a collection of a large number of three-dimensional points, each of which represents a position in space. For example, point cloud P can use (x, y, z) to represent the specific position of the point in three-dimensional space. Point cloud data can be three-dimensional point cloud data obtained by scanning a preset area through a laser radar (LiDAR), or it can be point cloud data obtained by using a stereo vision method to shoot a preset area from different angles with multiple cameras and using the parallax principle to calculate depth information, or it can be point cloud data obtained by analyzing pattern deformation to infer the three-dimensional shape of the target image surface, and there is no limitation here.

[0041] The target image is a two-dimensional image captured by a camera of the same preset area. The camera may be a camera on a camera or a camera mounted on a vehicle, and is not limited here.

[0042] It should be noted that in order to ensure the relevance of the point cloud data and the target image, when acquiring the point cloud data and the target image, it is necessary to ensure that the point cloud data and the target image are data of the same area, and to ensure that the time of the device equipped with the camera and the 3D scanning device are synchronized.

[0043] Step 102: project the point cloud in the point cloud data onto the target image to obtain a plurality of projection points, each of which is a pixel point in the target image.

[0044] Since the point cloud data is acquired by the lidar and the target image is taken by the camera, the lidar and the camera have two different coordinate systems. Therefore, the point cloud data is converted from the lidar coordinate system to the camera coordinate system through the camera's extrinsic parameter matrix and intrinsic and extrinsic parameters and projected into the target image.

[0045] The camera's extrinsic parameter matrix mainly includes information such as rotation and flattening. Through the extrinsic parameter matrix, each point in the point cloud can be accurately mapped from a coordinate system to the camera coordinate system. The camera's intrinsic parameter matrix mainly includes parameters such as focal length and principal point position. Through the intrinsic parameter matrix, the coordinates of three-dimensional points can be converted into two-dimensional image coordinates. The camera's extrinsic parameter matrix and intrinsic parameter matrix are both pre-calibrated.

[0046] Step 103, setting the pixel values ​​of the pixel points in the target image that belong to the projection point to the depth true value, and setting the pixel values ​​of the pixel points in the target image that do not belong to the projection point to the preset value, to obtain a depth true value image, wherein the depth true value is obtained based on the point cloud computing corresponding to the pixel point.

[0047] When the point cloud data is converted from the lidar coordinate system to the camera coordinate system, the depth true value of each point cloud data is obtained. The depth true value represents the vertical distance from each pixel to the camera plane. The depth true value of each point cloud data is used as the pixel value of the corresponding pixel in the target image, and the pixel values ​​of the pixels in the target image that do not belong to the projection points are set to preset values. The preset value can be 0 or -1, etc., which is a specific value indicating no depth information. There is no limitation here. The depth true value image is obtained by assigning the pixel value of each pixel in the target image.

[0048] In another embodiment of the present application, a two-dimensional image with the same size as the target image is created, and the pixel value of the two-dimensional image is an initial value, which may be 0 or a maximum depth value, and is not limited here. The pixels in the two-dimensional image correspond one-to-one to the pixels in the target image, and the depth true value of the pixel belonging to the projection point in the target image is used as the pixel value of the corresponding pixel in the two-dimensional image. The pixel points in the two-dimensional image whose pixel values ​​are the initial values ​​are screened, and the pixel values ​​of these pixel points are set to preset values. For example, the preset value may be 0, or may be a specific value such as -1 indicating no depth information, and is not limited here. The pixel value of each pixel in the depth two-dimensional image is assigned to obtain a depth true value image.

[0049] Step 104: Use the true depth image and the target image to train a perception model to obtain a target perception model, and the target perception model is used for target detection.

[0050] The perception model includes but is not limited to the BEV target detection model, the 3D target detection model and other models that use depth estimation for detection, such as the BEVDet model, the BEVDepth model, etc.; the target image is used as the input of the perception model, and the depth truth image is used to supervise the perception model to guide the perception model to learn the target detection model, thereby improving the detection accuracy of the target detection model.

[0051] Exemplarily, the perception model can be a BEVDet model, which includes an image view encoder, a view converter, a BEV encoder, and a detection head, wherein the image view encoder is used to extract image features from two-dimensional image data, the view converter is used to convert two-dimensional image features into three-dimensional features, the BEV encoder is used to extract key information in the three-dimensional features to obtain BEV features, and the detection head is used to perform 3D object detection on the image based on the BEV features.

[0052] The target image is input into the BEVDet model, and the depth estimation value image is obtained through the BEVDet model. The loss function value of the BEVDet model is calculated according to the true depth image and the depth estimation value image to adjust the BEVDet model to obtain a BEVDet model with more accurate target detection results.

[0053] In this embodiment, by obtaining training samples, the point cloud data in the training samples are projected onto the target image to obtain multiple projection points, the pixel values ​​of the pixel points in the target image belonging to the projection points are set to the depth true values, and the pixel values ​​of the pixel points that do not belong to the projection points are set to preset values, the depth information of the point cloud data is fully utilized to construct a depth true value image, and the perception model is trained by the depth true value image and the target image, so as to better guide the perception model to learn and obtain the target detection model, thereby improving the detection accuracy of the target detection model.

[0054] In another embodiment of the present application, interpolation processing may also be performed on pixel points that are not projection points to obtain more point cloud data. Specifically, after step S102 and before step S103, the following is further included:

[0055] For each pixel point in the target image, execute: if the pixel point does not belong to the first projection point or the second projection point, interpolate according to the point cloud corresponding to the first projection point to obtain a third point cloud, and use the pixel point as the projection point corresponding to the third point cloud.

[0056] In the above steps, the multiple projection points obtained are divided into a first projection point and a second projection point, the first projection point is a projection point corresponding to a first point cloud belonging to a foreground class in the point cloud data, and the second projection point is a projection point corresponding to a second point cloud belonging to a background class in the point cloud data; specifically, before projecting the point cloud data to the target image, target detection is performed on the point cloud data in advance to obtain multiple 3D detection frames, each detection frame includes a single target object, the point cloud within the 3D detection frame is called a point cloud of the foreground class, and the point cloud outside the 3D detection frame is called a point cloud of the background class, all point clouds in the point cloud data are projected to the target image, the projection point obtained by projecting the point cloud of the foreground class is the first projection point, and the projection point obtained by projecting the point cloud of the background class is the second projection point.

[0057] Exemplarily, the point cloud data includes point clouds A, B, C, and D; target detection is performed on the point cloud data to obtain a detection box A1 including point cloud A, and a detection box B1 including point clouds B and C; point clouds A, B, C, and D are projected onto the target image to obtain corresponding projection points a, b, c, and d; wherein point clouds A, B, and C are point clouds of the foreground class, point cloud D is a point cloud of the background class, projection points a, b, and c are first projection points, and projection point d is a second projection point.

[0058] The pixel points in the target image are traversed, and for the pixel points that do not belong to the first projection point and the second projection point, that is, the pixel points that are not projected by the point cloud data, the pixel points that are not projected by the point cloud data are interpolated by the interpolation method according to the point cloud corresponding to the first projection point, so as to obtain the point cloud corresponding to the pixel point, that is, the third point cloud. Since the third point cloud is obtained by interpolation of the point cloud of the foreground class corresponding to the first projection point, the third point cloud also belongs to the point cloud of the foreground class; wherein the interpolation method includes but is not limited to the neighbor interpolation method, the linear interpolation method, the cubic spline interpolation method, etc.

[0059] Exemplarily, the target image includes pixel points A, B, and C, the first projection point includes pixel point A, and the second projection point includes pixel point C. It is determined whether pixel point A belongs to the first projection point or the second projection point. If pixel point A belongs to the first projection point, pixel point A is not processed. It is determined whether pixel point B belongs to the first projection point or the second projection point. If pixel point B does not belong to the first projection point or the second projection point, pixel point B is interpolated according to the point cloud a corresponding to the first projection point A to obtain the point cloud b corresponding to pixel point B. The pixel point C is continued to be determined. If pixel point C belongs to the second projection point, pixel point C is not processed.

[0060] In this embodiment, the pixel points in the target image are traversed, and the pixel points that do not belong to the point cloud projection are interpolated to obtain point clouds corresponding to more pixel points, so as to increase the point cloud density and obtain more depth truth data, which is beneficial to subsequent model training and improves the detection accuracy of the target perception model.

[0061] In one embodiment of the present application, all point clouds in the point cloud data may be projected to the target image. In another embodiment of the present application, the point cloud data may be first screened, and the screened point clouds may be projected to the target image. Specifically, in step S102, the point clouds in the point cloud data are projected to the target image to obtain a plurality of projection points, including:

[0062] Detecting the target object in the point cloud data to obtain at least one detection frame, wherein each detection frame does not overlap with another;

[0063] For each of the detection frames, the following steps are performed: screening the point clouds in the detection frame to remove point clouds whose distance from the ground is less than a first preset distance, to obtain a plurality of the first point clouds;

[0064] According to the external parameter matrix and the internal parameter matrix of the camera, the first point cloud is projected onto the target image to obtain a plurality of the first projection points, and the camera is used to acquire the target image;

[0065] According to the external parameter matrix and the internal parameter matrix, the second point cloud is projected onto the target image to obtain a plurality of second projection points, where the second point cloud is a point cloud outside the detection box in the point cloud data.

[0066] In the above steps, target detection is performed on the point cloud data to obtain multiple 3D detection frames, each of which includes a single target object. The target object can be a vehicle, a pedestrian, an animal, etc. The obtained 3D detection frames are screened and the overlapping detection frames are removed to obtain at least one non-overlapping detection frame.

[0067] Specifically, a first set is obtained based on multiple 3D detection frames obtained by performing target detection on point cloud data, and each 3D detection frame in the first set is labeled to mark the center point of each 3D detection frame, the length, width, height, heading angle and category of each detection frame. The three-dimensional intersection-over-union (3D_IOU) between each detection frame is calculated based on the labeled information of each detection frame. If the three-dimensional intersection-over-union (3D_IOU) of the detection frame and other detection frames is 0, the detection frame is stored in the second set. There is at least one detection frame in the second set, and each detection frame in the second set does not overlap.

[0068] For example, the first set can be expressed as Box = {box1, box2, box3}, box1 represents the first detection box, box2 represents the second detection box, box3 represents the third detection box, and the annotated first detection box can be expressed as box1 = {x1, y1, w1, h1, l1, sita1}, where (x1, y1, z1) represents the center point of the first detection box, (w1, h1, l1) represents the length, width and height of the first detection box, and (sita1) is the heading angle. The representation methods of box2 and box3 are the same as those of box1. Similarly, calculate the 3D intersection and union ratio of box1 and box2, box1 and box3, and box2 and box3. If the 3D intersection and union ratio of box1 and box2, and box1 and box3 are both 0, it means that the first detection box does not overlap with the second detection box and the third detection box, and box1 is stored in the second set. If the 3D intersection and union ratio of box2 and box3 is not 0, it means that there is an overlapping part between the second detection box and the third detection box, and the second detection box and the third detection box are not stored in the second set. The obtained second set can be expressed as Box2={box1}.

[0069] Since some point clouds are close to the ground, they may be affected by factors such as ground reflection, resulting in inaccurate depth information. Therefore, the point cloud in each detection box in the second set is screened, and the point cloud whose distance to the ground is less than the first preset distance is removed to obtain multiple first point clouds. The first preset distance can be set according to actual conditions.

[0070] Exemplarily, the first preset distance is set to "(zh) / (2+th)", where z and h are the information marked by the detection box, th is the preset ground height threshold, which is set to 0.2m here, and can also be set according to actual conditions. The second set includes the first detection box, and the point cloud in the first detection box constitutes the point set P={{x1,y1,z1}…{xN,yN,zN}}. For any point p={x1,y1,z1} in the point set P, if z1 is less than the first preset distance, the point is retained; if it is greater than the first preset distance, the point is removed. After screening, the final point set P1 is obtained. The point set P1 includes multiple point clouds whose distance from the ground is less than the first preset distance, that is, the point set P1 includes multiple first point clouds.

[0071] Obtain the external parameter matrix and the internal parameter matrix of the camera that shoots the target image, and project the first point cloud and the point cloud outside the detection box in the point cloud data, i.e., the second point cloud, to the target image through the internal parameter matrix and the external parameter matrix; the pixel point projected by the first point cloud is the first projection point, and the pixel point projected by the second point cloud is the second projection point.

[0072] In this embodiment, by screening the detection frames to obtain non-overlapping detection frames, confusion between targets can be effectively avoided, thereby improving the accuracy and reliability of target detection. The point cloud within the detection frame is then screened to remove point clouds whose distance from the ground is less than a preset value to reduce the influence of factors such as ground reflection, further improving the accuracy of the data, and projecting the screened point cloud and the point cloud outside the detection frame to the target image to achieve accurate mapping from three-dimensional space to two-dimensional image.

[0073] In one embodiment of the present application, the first point cloud is projected onto the target image according to the external parameter matrix and the internal parameter matrix of the camera to obtain a plurality of the first projection points, including:

[0074] According to the external parameter matrix and the internal parameter matrix of the camera, each of the first point clouds is processed as follows:

[0075] Performing coordinate transformation on the first point cloud according to the external parameter matrix and the internal parameter matrix to obtain a first value, a second value, and a third value, wherein the first value is a pixel position of the first point cloud projected to the target image in a first direction, the second value is a pixel position of the first point cloud projected to the target image in a second direction, the first direction is perpendicular to the second direction, and the third value is a true depth value corresponding to the first point cloud;

[0076] When the first value, the second value, and the third value satisfy a preset condition, the projection point corresponding to the first point cloud is determined to be a first projection point determined according to the first value and the second value.

[0077] In the above steps, since the point cloud data is obtained by a laser radar and the target image is obtained by a camera, the laser radar and the camera have two different coordinate systems. Therefore, by obtaining the intrinsic parameter matrix and the extrinsic parameter matrix of the camera, the point cloud is converted from the coordinate system of the laser radar to the coordinate system of the camera to project the point cloud to the two-dimensional target image. After converting the coordinates of the first point cloud into the coordinate representation in the camera coordinate system, the pixel position of the first point cloud projected to the target image in the first direction, i.e., the first value, the pixel position in the second direction, i.e., the second value, and the depth true value, i.e., the third value, are obtained. The first value is divided by the third value to obtain the pixel value of the first point cloud projected in the first direction of the target image. The first value is divided by the third value to obtain the pixel value of the first point cloud projected in the second direction of the target image. The first direction and the second direction are perpendicular to each other, for example, the first direction is the x-axis and the second direction is the y-axis.

[0078] A point cloud whose true depth value is greater than zero and whose first value and second value are both located in the first direction and the second direction of the target image (that is, when the point cloud is projected to the target image, it is located within the size range of the target image) is determined to be a point cloud projected to the target image, and the point projected by the point cloud is determined to be the first projection point.

[0079] Exemplarily, the point cloud p is converted into the coordinate system of the camera by the following formula:

[0080] Pi=M*RT*p

[0081] Among them, Pi = {Px, Py, Pz} represents the point cloud p in the camera coordinate system, M represents the 3*4 external parameter matrix of the camera, RT represents the 3*3 external parameter matrix of the camera, and p = {x, y, z} represents the point cloud p.

[0082] The size range of the x-axis of the target image is (0, x2), and the size range of the y-axis is (0, y2). When Pz is greater than 0, and Px is less than or equal to x2, and Py is less than or equal to y2, it is determined that point p can be projected to the target image, and the point p'(Px, Py) where point p is projected to the target image is the first projection point.

[0083] In this embodiment, by utilizing the camera's intrinsic parameter matrix and extrinsic parameter matrix, accurate coordinate transformation is achieved, and target objects in the image are more accurately identified and classified, thereby improving the performance of the target detection model. By setting preset conditions, point clouds that do not meet the conditions are effectively filtered out, reducing the amount of data that needs to be processed, thereby reducing the consumption of computing resources and improving data processing efficiency.

[0084] In one embodiment of the present application, if the pixel point does not belong to the first projection point or the second projection point, interpolation is performed according to the point cloud corresponding to the first projection point to obtain a third point cloud, including:

[0085] If the pixel point does not belong to the first projection point or the second projection point, obtaining a plurality of candidate pixel points within a preset range of the pixel point in the target image;

[0086] Dividing the multiple candidate pixel points to obtain multiple symmetrically distributed areas;

[0087] For each of the regions, the following steps are performed: filtering the pixels in the region and retaining a first pixel, where the first pixel is a pixel belonging to the first projection point; if there are multiple first pixels, retaining the first pixel closest to the pixel;

[0088] In the case that each of the regions includes a first pixel point, interpolation processing is performed according to a point cloud corresponding to the first pixel point included in each of the regions to obtain a third point cloud corresponding to the pixel point.

[0089] In the above steps, the pixel points in the target image are traversed, and the pixel points in the target image that do not belong to the first projection point or the second projection point are screened. The point is called the target pixel point, and the point within the preset range of the target pixel point is obtained as the candidate pixel point. The multiple candidate pixel points are divided into multiple symmetrically distributed areas, and each area contains the same number of candidate pixel points. The preset range can be set according to actual conditions.

[0090] Specifically, for each target pixel, N candidate pixels around the target pixel in the target image are obtained (N can be set according to actual conditions), the N candidate pixels are divided into M areas with symmetrical distribution (M can be set according to actual conditions), and the candidate pixels in each area are queried to determine whether it is the first projection point. If there is an area where no candidate pixel is the first projection point, the interpolation processing based on the first pixel is stopped, and the next target pixel in the target image is obtained; if there are multiple candidate pixels in an area as the first projection point, the candidate pixel closest to the target pixel is screened out to participate in the subsequent interpolation processing. After the above processing, if there is a candidate pixel in each of the M areas to participate in the subsequent interpolation processing, the bilinear interpolation method is used to perform interpolation processing according to the corresponding point cloud information of the M candidate pixels to obtain the point cloud corresponding to the target pixel, that is, the third point cloud. In this case, the target pixel has a corresponding relationship with the third point cloud, and N and M are both positive integers.

[0091] For example, Figure 2As shown in the figure, there is a target pixel point P (x, y), and 24 candidate pixel points near the query point P are divided into four symmetrical areas. It is found whether there is a first projection point in the four areas. Taking area 1 as an example, it is queried whether point M (x-1, y) is the first projection point, and whether the other five pixel points in area 1 except point M are the first projection points. When there are multiple points in area 1 that are the first projection points, the pixel point closest to point P is retained as the first pixel point. The four areas are traversed. If there is a first pixel point in each area, point P is interpolated according to the first pixel points in the four areas to obtain the point cloud corresponding to point P.

[0092] In this embodiment, by interpolating candidate points in multiple symmetrically distributed areas, the depth information of the current pixel can be estimated more accurately. Compared with interpolation processing in a single direction, multi-directional interpolation can provide more comprehensive point cloud information, thereby obtaining more accurate depth information and improving the accuracy of depth estimation.

[0093] In one embodiment of the present application, when each of the regions includes a first pixel point, interpolation processing is performed according to a point cloud corresponding to the first pixel point included in each of the regions to obtain a third point cloud corresponding to the pixel point, including:

[0094] In the case where each of the regions includes a first pixel point, if the distance between any two of the first pixel points included in each region is less than a second preset distance, interpolation processing is performed based on the point cloud corresponding to the first pixel points included in each region to obtain a third point cloud corresponding to the pixel point, and the third point cloud belongs to the foreground class.

[0095] In the above steps, if each area of ​​the target pixel point includes a first pixel point, the distance between any two first pixel points is calculated. When the distance between any two first pixel points is less than the second preset distance, it means that the target pixel point can be interpolated according to the first pixel points in multiple areas to obtain a point cloud corresponding to the target pixel point; if the distance between two first pixel points is greater than or equal to the second preset distance, the target pixel point is not interpolated.

[0096] It should be noted that the second preset distance can be set according to actual conditions, and since the first pixel point is obtained by projecting the point cloud of the foreground class, and the third point cloud corresponding to the target pixel point is obtained by interpolation based on the point cloud corresponding to the first pixel point, the third point cloud also belongs to the point cloud of the foreground class.

[0097] Exemplarily, the area around the candidate pixel point P is divided into four symmetrical areas, each area has a first pixel point, and four points P1 (x1, y1, z1), P2 (x2, y2, z2), P3 (x3, y3, z3), and P4 (x4, y4, z4) are obtained. The distances between P1P2, P1P3, P1P4, P2P3, P2P4, and P3P4 are calculated. When the above distances are all less than the second preset distance, point P is interpolated according to the point clouds corresponding to P1, P2, P3, and P4 to obtain the point cloud information corresponding to point P.

[0098] In this embodiment, by setting a second preset distance, it is ensured that the distance between the first pixel points in each area is close enough to generate more accurate point cloud data and reduce the interpolation error caused by excessive distance. By setting a second preset distance, pixel points with a long distance are excluded, thereby reducing noise interference in the interpolation process and improving data reliability.

[0099] In another embodiment of the present application, the correspondence between the first point cloud and the first projection point can be stored in the first dictionary in the form of a key-value pair, with the coordinates of the projection point as an identifier and the point cloud information of the first point cloud as a value. For example, the first dictionary Project1 can be represented as

[0100] Project1={Px*N+Py,{x1,y1,z1,Pz,Type}...}, Px and Py represent the pixel positions of point cloud p projected to the target image in the X and Y directions, N represents the total pixel value of the X axis in the target image, {x1,y1,z1,Pz,Type} represents the three-dimensional point cloud information of point cloud p, Type represents the category of point cloud p, and the category of point cloud p is the same as the category of the 3D detection box to which the point cloud belongs.

[0101] The pixel values ​​of the point cloud p projected to the target image in the x-axis and y-axis directions can be calculated according to the following formula:

[0102] x1=Px / Pz

[0103] y1=Py / Pz

[0104] x1 represents the pixel value of point cloud p projected to the X-axis of the target image, y1 represents the pixel value of point cloud p projected to the Y-axis of the target image, Px represents the pixel position of point cloud p projected to the X-axis direction of the target image, Py represents the pixel position of point cloud p projected to the Y-axis direction of the target image, and Pz represents the true depth value of point cloud p.

[0105] Traverse the pixel points in the target image to find whether they exist in the first dictionary, take the pixel points that do not exist in the first dictionary as the target pixel points, interpolate each target pixel point according to the above interpolation rule, obtain the third point cloud corresponding to the target pixel point, store the correspondence between the target pixel point and the third point cloud in the second dictionary, the expression form of the second dictionary is the same as that of the first dictionary; combine the first dictionary and the second dictionary to obtain the third dictionary.

[0106] The first point cloud in the third dictionary and the point cloud belonging to the background class are projected to the target image, the depth true value of the point is used as the pixel value of the corresponding pixel point, and the pixel value of the pixel point that is not projected in the target image is set to the preset value to obtain the depth true value image.

[0107] In one embodiment of the present application, the perception model may be trained using the target image and the depth truth image. Specifically, step S104, using the depth truth image and the target image to train the perception model to obtain the target perception model, includes:

[0108] Inputting the target image into a perception model to obtain a depth estimation value image of the target image by the perception model;

[0109] Determining a loss function value of the perception model according to the true depth image and the estimated depth image;

[0110] The loss function value is used to adjust the model parameters of the perception model to obtain the target perception model.

[0111] In the above steps, the target image is used as the input of the perception model to obtain the depth value of each pixel in the target image estimated by the perception model, the estimated depth value of each pixel is used as the pixel value of the pixel, and a depth estimation value image is generated. According to the previously generated depth true value image and the depth estimation value image output by the target model, a universal loss function is used to calculate the loss function value of the perception model, and the calculated loss function value is used to adjust the parameters of the perception model through a back propagation algorithm until the loss function value meets the preset loss function value to obtain the target perception model; the preset loss function value can be set according to the actual situation, and the universal loss function can be the loss function that comes with the perception model, or it can be a loss function set according to the actual situation, which is not limited here.

[0112] The perception model may be a BEV target detection model, a 3D target detection model, or other models that use depth estimation for target detection, which is not limited here.

[0113] For example, Figure 3As shown, the perception model is the Bevdepth model. The general training process of the Bevdepth model can be described as follows: after the Bevdepth model receives the target image, the Bevdepth model extracts the two-dimensional features of the target image, performs depth estimation on the target image, estimates the pixel value of each pixel in the target image, that is, the depth value, and obtains the depth estimation value image. According to the extracted two-dimensional features and the depth estimation value image, a three-dimensional bird's-eye view is generated, and the three-dimensional features in the bird's-eye view feature image are further extracted. Target detection is performed based on the extracted three-dimensional features to obtain predicted target detection results, and the detection loss function value is calculated based on the previously marked 3D detection box and the predicted target detection result. The parameters of the Bevdepth model are adjusted through the back propagation method until the preset requirements are met.

[0114] Based on the technical solution of this embodiment, the training process of the Bevdepth model can be described as: after the Bevdepth model receives the target image, the Bevdepth model extracts the two-dimensional features of the target image, performs depth estimation on the target image, estimates the pixel value of each pixel in the target image, that is, the depth value, and obtains the depth estimation value image. The depth true value image obtained by the above implementation method and the depth estimation value image generated by the Bevdepth model are used to calculate the depth estimation loss function value of the Bevdepth model. Based on the depth estimation loss function value, the depth estimation of the Bevdepth model is supervised and adjusted through the back propagation method to improve the accuracy of the depth estimation of the Bevdepth model, thereby training a Bevdepth model with higher detection accuracy, that is, the target perception model. The subsequent training steps are the same as the general Bevdepth model training method mentioned above, and will not be repeated here.

[0115] In this embodiment, the depth estimation value image output by the perception model is compared with the true depth value image and the model is optimized, which can significantly improve the accuracy of the perception model in the depth estimation task, thereby improving the accuracy of the perception model for target detection.

[0116] In one embodiment of the present application, determining the loss function value of the perception model according to the true depth image and the estimated depth image includes:

[0117] According to the true depth image, a second set and a third set are obtained, wherein the second set includes a set of pixel values ​​of all pixel points in the point cloud data obtained by projecting the point cloud belonging to the background class, and the third set includes a set of pixel values ​​of all pixel points in the point cloud data obtained by projecting the point cloud belonging to the foreground class;

[0118] According to the depth estimation value image of the target image, a fourth set and a fifth set are obtained, wherein the fourth set includes a set of depth estimation values ​​of all pixel points obtained by projecting the point cloud belonging to the background class in the point cloud data, and the fifth set includes a set of depth estimation values ​​of all pixel points obtained by projecting the point cloud belonging to the foreground class in the point cloud data;

[0119] Based on the second set and the fourth set, a preset initial loss function is used for calculation to obtain a first loss sub-function value;

[0120] Based on the third set and the fifth set, the initial loss function is used to perform calculation to obtain a second loss sub-function value;

[0121] The sum of the first loss sub-function value and the second loss sub-function value is used as the loss function value of the perception model.

[0122] In the above steps, according to the type of point cloud projected to the pixel points, the pixel points in the image are classified to obtain a second set, a third set, a fourth set and a fifth set, wherein the second set is a set of pixel values ​​of pixel points obtained by projecting the point cloud belonging to the background class in the depth true value image, the third set is a set of pixel values ​​of pixel points obtained by projecting the point cloud belonging to the foreground class in the depth true value image, the fourth set is a set of depth estimation values ​​of pixel points obtained by projecting the point cloud belonging to the background class in the depth estimation value image, and the fifth set is a set of depth estimation values ​​of pixel points obtained by projecting the point cloud belonging to the foreground class in the depth estimation value image.

[0123] According to the second set and the fourth set, the preset initial loss function is used to calculate the value of the loss function for background class target detection in the perception model, that is, the first loss sub-function value. According to the third set and the fifth set, the preset initial loss function is used to calculate the value of the loss function for foreground class target detection in the perception model, that is, the second loss sub-function value. The preset initial function can be a mean square error function, a mean absolute error function, or other general loss functions, which are not limited here. The sum of the first loss sub-function value and the second loss function value is used as the loss function value of the perception model.

[0124] In this embodiment, the loss function is divided into a foreground loss function and a background loss function, so that the model can focus more on the depth estimation of the foreground target, and the weights of the first loss sub-function and the second loss sub-function can be adjusted so that the model can pay more attention to the category that needs to be detected.

[0125] In one embodiment of the present application, based on the second set and the fourth set, a preset initial loss function is used for calculation to obtain a first loss sub-function value, including:

[0126] Based on the second set and the fourth set, a preset initial loss function is used to perform calculation to obtain a first loss value;

[0127] Multiplying the first loss value by a first coefficient to obtain a first loss sub-function value, wherein the first coefficient is a ratio of a fourth value to a fifth value, the fourth value is the number of pixels in the depth true value image obtained by projecting a point cloud belonging to a background class, and the fifth value is the number of pixels in the depth true value image that do not belong to the point cloud projection;

[0128] and / or,

[0129] The step of calculating based on the third set and the fifth set using the initial loss function to obtain a second loss sub-function value includes:

[0130] Based on the third set and the fifth set, using the initial loss function to perform calculation to obtain a second loss value;

[0131] Multiply the second loss value by the second coefficient to obtain a second loss sub-function value, wherein the second coefficient is the ratio of the sixth value to the fifth value, and the sixth value is the number of pixels obtained by projecting the point cloud belonging to the foreground class in the depth truth image.

[0132] In the above steps, the second set is taken as the true value, the fourth set is taken as the predicted value, and the preset initial loss function is used for calculation to obtain the first loss value. The number of pixel points obtained by the point cloud projection belonging to the background class in the depth true value image is counted as the fourth value, and the number of pixel points obtained by the point cloud projection that do not belong to the depth true value image is counted as the fifth value. The ratio of the fourth value to the fifth value is taken as the first coefficient. The first loss value is multiplied by the first coefficient to obtain the loss function value of the background class, that is, the first loss sub-function value.

[0133] Similarly, the third set is taken as the true value and the fifth set as the predicted value, and the preset initial loss function is used for calculation to obtain the second loss value. The number of pixel points obtained by the point cloud projection belonging to the foreground class in the depth truth image is counted as the sixth value, and the ratio of the sixth value to the fifth value is taken as the second coefficient. The second loss value is multiplied by the second coefficient to obtain the loss function value of the foreground class, that is, the second loss sub-function value.

[0134] Exemplarily, the first loss sub-function value and the second loss sub-function value may be expressed by the following formula:

[0135] Loss_B=Num2 / Num3*loss_base(pred_B,Gt_B)

[0136] Loss_F=Num1 / Num3*loss_base(pred_F,Gt_F)

[0137] Among them, Loss_B represents the first loss sub-function value, Loss_F represents the second loss sub-function value, loss_base represents the preset initial loss function, Num1 represents the sixth value, Num2 represents the fourth value, Num3 represents the fifth value, pred_B represents the fourth set, Gt_B represents the second set, pred_F represents the fifth set, and Gt_F represents the third set.

[0138] In this embodiment, the weighted loss function can be used to balance the impact of different categories on the total loss function value, prevent the model from being biased towards a larger number of categories, and add dynamic weights to the foreground loss function and the background loss function respectively, so that the model can focus more on the categories that need to be detected.

[0139] The following is an example of the model training method provided in the embodiments of the present application.

[0140] Step 0: Obtain point cloud data and target images based on the same target, as well as annotated 3D detection frames.

[0141] Among them, the 3D detection box is to detect the target object (such as a moving object, a person, a vehicle, an animal, etc.) in the point cloud data, and obtain at least one 3D detection box. Each 3D detection box contains a single target, and the parameters of each 3D detection box are marked, expressed as box = {x, y, z, l, w, h, sita}, (x, y, z,) represents the center point of the 3D detection box, l, w, h represent the length, width, and height of the 3D detection box respectively, sita represents the heading angle, and the category of the 3D detection box is marked.

[0142] Step 1: When there are multiple 3D detection frames, the multiple 3D detection frames are screened to exclude overlapping 3D detection frames to obtain at least one target 3D detection frame.

[0143] The set of 3D detection boxes is represented as Box = {box1, box2…boxN}, and the 3D intersection and union ratio between each detection box in the Box is calculated. For example, if the 3D intersection and union ratio of box1 and other detection boxes are all 0, then box1 is marked as True, otherwise box1 is marked as False, and the same is true for other detection boxes in the Box; get Box_State = {Ture, False…Ture}; traverse the Box, if the corresponding value of the 3D detection box in the Box in Box_State is Ture, then the detection box is used as the target 3D detection box, and the point clouds in all target 3D detection boxes are extracted to obtain the point set P = = {{x1, y1, z1}…{xN, yN, zN}}.

[0144] Step 2: Filter all point cloud data in the target detection frame to remove point cloud data that is closer to the ground, and obtain a point set P1, which includes at least one target point cloud data.

[0145] Set the ground threshold to "zh / 2+th". For any point p={x1,y1,z1} in the point set P, if z1<(zh / 2+th), the point will not be retained. If z1>(zh / 2+th), the point p will be stored in the point set P1, where z and h are the values ​​of the 3D detection box, th is the ground height threshold, which is generally set to 0.2m, and the point set P1 is obtained.

[0146] Step 3: According to the camera's intrinsic parameter matrix and extrinsic parameter matrix, the point cloud in the point set P1 is projected to a two-dimensional image of the same size as the target image, and the dictionary Project is constructed based on the correspondence between the point cloud and the projection points.

[0147] Through the following formula, any point p={x2,y2,z2,1} in the point set P1 is projected to the target image:

[0148] Pi=M*RT*p

[0149] Among them, RT represents the 3*4 external parameter matrix; M represents the 3*3 internal parameter matrix, and Pi={Px,Py,Pz} represents the point p in the camera coordinate system.

[0150] Pz represents the distance from point p to the camera, that is, the depth value. The depth value is used as the pixel value of the corresponding pixel point. Px represents the pixel position of point p projected to the x-axis of the two-dimensional image. Py represents the pixel position of point p projected to the y-axis of the two-dimensional image. The projection point p'(Px, Py) is obtained, and the ratio of Px to Pz is used as the pixel value of point m projected to the x-axis of the target image, and the ratio of Py to Pz is used as the pixel value of point m projected to the y-axis of the target image.

[0151] The point cloud data with Pz less than or equal to 0 and Px and Py greater than the target image size range are deleted to obtain the point set P2.

[0152] According to the correspondence between the point cloud and the projection point in the point set P2, a dictionary Project is constructed. Project can be expressed as {Px*N+Py,{x1,y1,z1,Pz,Type}...}, where N is the total pixel value in the x direction of the target image, Px*N+Py is used as the identifier, and {x1,y1,z1,Pz,Type} is the value, where Type is the category corresponding to the point cloud. The category of the point cloud is the same as the category of the 3D target detection box to which the point cloud belongs.

[0153] Step 4: Traverse each pixel in the target image, interpolate the pixels that do not have a corresponding relationship in the dictionary Project, and obtain a new dictionary Project1.

[0154] According to the coordinate position of the pixel point, it is determined whether the pixel point has a corresponding relationship in the dictionary Project, and the pixel point that does not have a corresponding relationship is called the first pixel point, such as Figure 2 As shown, there is a first pixel point P(x,y) in the target image, query the 24 pixel points around point P, divide the 24 pixel points into 4 areas, take area 1 as an example, query whether point P(x-1,y) is the first projection point, and query whether the other five pixel points in area 1 except point P are the first projection points. When there are multiple points in area 1 that are the first projection points, retain the pixel point closest to point P as the first pixel point, traverse the four areas, if there is a first pixel point in each area, get the point set P_arround around point P=={{x1,y1,z1,Pz1,T1}…{xn,yn,zn,Pz,Tn}}, when the length of P_arround is 4 and the point set P_arro When the distance between any two points in und is less than the interpolation threshold, the interpolation threshold can be set according to the distance of the point cloud. Then, although there is no point cloud directly projected to the pixel point, the expression P(x,y) exists in four directions. The point cloud information {x1,y1,z1,Pz,Type} corresponding to these four points in the dictionary Project can be interpolated to obtain the point cloud P'{x',y',z',Pz',T'} corresponding to the pixel point P. In this case, the first pixel point P(x,y) forms a corresponding relationship with the point cloud P'{x',y',z',Pz',T'}. Repeat the above steps for all pixel points in the target image that do not have a corresponding relationship in the dictionary Project to obtain a new dictionary.

[0155] Project'={x*N+y,{x',y',z',Pz',T'}...}, and a new dictionary Project1 is obtained by Project+Project'.

[0156] Step 5: All point clouds outside the 3D detection box and the point clouds in the dictionary Project are projected to the target image, and the pixel value of each pixel is defined according to the point cloud type to obtain the depth truth image.

[0157] The point cloud within the 3D detection box is called the foreground point cloud, and the point cloud outside the 3D detection box is called the background point cloud. The pixel value of the pixel point in the target image obtained by the foreground point cloud and the background point cloud projection is set as the depth value of the point, and the pixel value of the pixel point without point cloud projection in the target image is set to 0 to obtain the depth true value image.

[0158] Step 6: Input the target image into the perception model to obtain the depth estimation value image. Calculate the loss function value of the perception model through the true depth image and the depth estimation value image to train the Bevdepth model to obtain a Bevdepth model with better detection effect.

[0159] The target image is input into the Bevdepth model, and the Bevdepth model outputs a depth estimation image. The depth estimation image has the same resolution as the target image, and the pixel value of each pixel in the depth estimation image represents the depth estimation value of the pixel.

[0160] According to the point cloud type corresponding to each type of pixel, the pixels in the depth estimation image and the depth true value image are classified, and the number of each type of pixel in the depth true value image is counted. The pixels obtained by projecting the foreground point cloud are called the first type of pixels, and the total number of the first type of pixels is recorded as Num1. The pixels obtained by projecting the background point cloud are called the second type of pixels, and the total number of the second type of pixels is recorded as Num2. The pixels without point cloud projection are called the third type of pixels, and the total number of the third type of pixels is recorded as Num3.

[0161] The calculation method of the loss function value can be expressed as follows:

[0162] Loss_B=Num2 / Num3*loss_base(pred_B,Gt_B)

[0163] Loss_F=Num1 / Num3*loss_base(pred_F,Gt_F)

[0164] Loss = Loss_B + Loss_F

[0165] Among them, Loss_B represents the background loss function value, Loss_F represents the foreground loss function value, Loss represents the total loss function value, loss_base represents the mean square error function, pred_B represents the set of depth estimation values ​​of the point cloud corresponding to the pixel points in the depth estimation value image as the background point cloud, Gt_B represents the set of depth true values ​​of the point cloud corresponding to the pixel points in the depth truth image as the background point cloud, pred_F represents the set of depth estimation values ​​of the point cloud corresponding to the pixel points in the depth estimation value image as the foreground point cloud, and Gt_F represents the set of depth true values ​​of the point cloud corresponding to the pixel points in the depth truth image as the foreground point cloud.

[0166] After calculating the loss function value, adjust the parameters of the Bevdepth model through the back propagation algorithm, and repeat the above steps until the loss function value meets the preset loss function value to obtain the final Bevdepth model.

[0167] The model training method provided in the embodiment of the present application effectively solves the problem of inaccurate depth estimation caused by sparse point clouds by performing depth interpolation on the point clouds within the 3D detection box; and different interpolation thresholds can be set for targets at different distances according to actual needs, thereby improving the accuracy of depth estimation of near and distant targets; and the perception model is trained by dynamically adjusting the loss function, so that the perception model pays more attention to the target category that needs to be detected, avoids background interference, and further improves the accuracy of depth estimation of the target perception model.

[0168] Based on the target perception model obtained through training, the embodiment of the present application provides a target detection method. Figure 4 A flow chart of a target detection method provided by an exemplary embodiment of the present application is shown. The target detection method provided by the embodiment of the present application is applied to an electronic device, such as Figure 4 As shown, the target detection method may include the following steps:

[0169] Step 301, obtaining an image to be detected;

[0170] Step 302: input the image to be detected into a target perception model to obtain a detection target, and the target perception model is obtained by using the model training method described in any of the above embodiments.

[0171] In the above steps, the target detection model in the computer device can perform target detection on the image to be detected; further, the computer device can perform subsequent operations based on the obtained target detection result, such as analyzing the target object in the target image based on the target detection result.

[0172] In this embodiment, the target perception model can quickly identify the target in the image to be detected, and obtain more accurate target detection results.

[0173] Figure 5 The structure diagram of the model training device provided in the embodiment of the present application is shown. Figure 5 As shown, the model training device 400 includes:

[0174] An acquisition module 401 is used to acquire a training sample, wherein the training sample includes point cloud data and a target image obtained by photographing a preset area;

[0175] A projection module 402 is used to project the point cloud in the point cloud data onto the target image to obtain a plurality of projection points, each of which is a pixel point in the target image;

[0176] An assignment module 403 is used to set the pixel values ​​of the pixel points in the target image that belong to the projection point as the depth true value, and set the pixel values ​​of the pixel points in the target image that do not belong to the projection point as the preset value, so as to obtain a depth true value image, wherein the depth true value is obtained based on the point cloud computing corresponding to the pixel points;

[0177] The training module 404 is used to train the perception model using the true depth image and the target image to obtain a target perception model, and the target perception model is used for target detection.

[0178] In an embodiment of the present application, the plurality of projection points include a first projection point and a second projection point, the first projection point being a projection point corresponding to a first point cloud belonging to a foreground class in the point cloud data, and the second projection point being a projection point corresponding to a second point cloud belonging to a background class in the point cloud data;

[0179] The model training device 400 also includes:

[0180] An interpolation module is used to execute, for each pixel point in the target image: if the pixel point does not belong to the first projection point or the second projection point, interpolate according to the point cloud corresponding to the first projection point to obtain a third point cloud, and use the pixel point as the projection point corresponding to the third point cloud.

[0181] In one embodiment of the present application, the projection module 402 includes:

[0182] A detection submodule, configured to detect a target object in the point cloud data to obtain at least one detection frame, wherein each detection frame does not overlap with another;

[0183] A screening submodule is used to perform, for each of the detection frames: screening the point clouds in the detection frame to remove point clouds whose distance from the ground is less than a first preset distance, to obtain a plurality of the first point clouds;

[0184] A first projection submodule, used for projecting the first point cloud onto the target image according to the external parameter matrix and the internal parameter matrix of the camera to obtain a plurality of the first projection points, and the camera is used for acquiring the target image;

[0185] The second projection submodule is used to project the second point cloud to the target image according to the external parameter matrix and the internal parameter matrix to obtain multiple second projection points, where the second point cloud is the point cloud outside the detection box in the point cloud data.

[0186] In an embodiment of the present application, the first projection submodule includes:

[0187] A coordinate conversion unit is used to process each of the first point clouds as follows according to the external parameter matrix and the internal parameter matrix of the camera:

[0188] Performing coordinate transformation on the first point cloud according to the external parameter matrix and the internal parameter matrix to obtain a first value, a second value, and a third value, wherein the first value is a pixel position of the first point cloud projected to the target image in a first direction, the second value is a pixel position of the first point cloud projected to the target image in a second direction, the first direction is perpendicular to the second direction, and the third value is a true depth value corresponding to the first point cloud;

[0189] A projection determination unit is used to determine that the projection point corresponding to the first point cloud is a first projection point determined according to the first value and the second value when the first value, the second value and the third value meet a preset condition.

[0190] In one embodiment of the present application, the interpolation module includes:

[0191] a candidate pixel point determination unit, configured to obtain a plurality of candidate pixel points within a preset range of the pixel point in the target image if the pixel point does not belong to the first projection point or the second projection point;

[0192] A region division unit, used for dividing the plurality of candidate pixels into a plurality of symmetrically distributed regions;

[0193] A pixel point screening unit is used to perform, for each of the regions: screening the pixel points in the region, retaining a first pixel point, where the first pixel point is a pixel point belonging to the first projection point; if there are multiple first pixel points, retaining a first pixel point closest to the pixel point;

[0194] The interpolation unit is used to perform interpolation processing according to the point cloud corresponding to the first pixel point included in each of the regions when each of the regions includes the first pixel point, so as to obtain a third point cloud corresponding to the pixel point.

[0195] In an embodiment of the present application, the interpolation unit includes:

[0196] An interpolation subunit is used to perform interpolation processing on the point cloud corresponding to the first pixel points included in each of the regions to obtain a third point cloud corresponding to the pixel points if the distance between any two of the first pixel points included in each of the regions is less than a second preset distance, when each of the regions includes a first pixel point, and the third point cloud belongs to the foreground class.

[0197] In one embodiment of the present application, the training module 404 includes:

[0198] A depth estimation submodule, used to input the target image into a perception model to obtain a depth estimation value image of the target image by the perception model;

[0199] A loss function value calculation submodule, used to determine the loss function value of the perception model according to the true depth image and the estimated depth image;

[0200] The model adjustment submodule is used to adjust the model parameters of the perception model using the loss function value to obtain the target perception model.

[0201] In one embodiment of the present application, the loss function value calculation submodule includes:

[0202] a first pixel classification unit, configured to obtain a second set and a third set according to the true depth image, wherein the second set includes a set of pixel values ​​of all pixels obtained by projecting a point cloud belonging to a background class in the point cloud data, and the third set includes a set of pixel values ​​of all pixels obtained by projecting a point cloud belonging to a foreground class in the point cloud data;

[0203] a second pixel classification unit, configured to obtain a fourth set and a fifth set according to the depth estimation value image of the target image, wherein the fourth set includes a set of depth estimation values ​​of all pixels obtained by projection of the point cloud belonging to the background class in the point cloud data, and the fifth set includes a set of depth estimation values ​​of all pixels obtained by projection of the point cloud belonging to the foreground class in the point cloud data;

[0204] A first loss sub-function value calculation unit, configured to calculate, based on the second set and the fourth set, using a preset initial loss function to obtain a first loss sub-function value;

[0205] A second loss sub-function value calculation unit, configured to calculate based on the third set and the fifth set using the initial loss function to obtain a second loss sub-function value;

[0206] A loss function value calculation unit is used to use the sum of the first loss sub-function value and the second loss sub-function value as the loss function value of the perception model.

[0207] In one embodiment of the present application, the first loss sub-function value calculation unit includes:

[0208] A first loss value calculation subunit, configured to calculate based on the second set and the fourth set using a preset initial loss function to obtain a first loss value;

[0209] A first loss function value calculation subunit, configured to multiply the first loss value by a first coefficient to obtain a first loss sub-function value, wherein the first coefficient is a ratio of a fourth value to a fifth value, the fourth value is the number of pixels obtained by projection of a point cloud belonging to a background class in the depth true value image, and the fifth value is the number of pixels not obtained by projection of a point cloud in the depth true value image;

[0210] and / or,

[0211] The second loss sub-function value calculation unit includes:

[0212] A second loss value calculation subunit, configured to calculate based on the third set and the fifth set using the initial loss function to obtain a second loss value;

[0213] The second loss function value calculation subunit is used to multiply the second loss value by a second coefficient to obtain a second loss sub-function value, wherein the second coefficient is the ratio of the sixth value to the fifth value, and the sixth value is the number of pixels obtained by projecting the point cloud belonging to the foreground class in the depth true value image.

[0214] The model training device 400 provided in the embodiment of the present application can implement each process implemented in the aforementioned model training method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0215] Figure 6 FIG. 1 shows a structural diagram of a target detection device provided in an embodiment of the present application. Figure 6 As shown, the target detection device 500 includes:

[0216] An image acquisition module 501 is used to acquire an image to be detected;

[0217] The target detection module 502 is used to input the image to be detected into the target perception model to obtain a detection target. The target perception model is obtained by using any one of the model training methods in the above embodiments.

[0218] The target detection device 500 provided in the embodiment of the present application can implement each process implemented in the aforementioned target detection method embodiment and achieve the same technical effect. To avoid repetition, it will not be described here.

[0219] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0220] The electronic device may include a processor 601 and a memory 602 storing computer program instructions.

[0221] Specifically, the processor 601 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0222] The memory 602 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 602 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 602 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 602 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 602 is a non-volatile solid-state memory.

[0223] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect or the second aspect of the present disclosure.

[0224] The processor 601 implements any one of the model training methods and target detection methods in the above embodiments by reading and executing computer program instructions stored in the memory 602.

[0225] In one example, the electronic device may further include a communication interface 603 and a bus 610. Figure 7 As shown, the processor 601, the memory 602, and the communication interface 603 are connected via a bus 610 and communicate with each other.

[0226] The communication interface 603 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0227] Bus 610 includes hardware, software or both, and the parts of model training method and target detection method or verification equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 610 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0228] In addition, in combination with the model training method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the model training methods and target detection methods in the above embodiment is implemented.

[0229] In addition, the embodiments of the present application may be implemented by providing a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device implements any one of the model training methods and target detection methods in the above embodiments.

[0230] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiment, several specific steps are described as examples. However, the method process of the present application is not limited to the specific steps described, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0231] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0232] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0233] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0234] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire training samples, where the training samples include point cloud data and target images obtained by photographing a preset area; Projecting the point cloud in the point cloud data onto the target image to obtain a plurality of projection points, each of which is a pixel point in the target image; Setting the pixel values ​​of the pixel points in the target image that belong to the projection point as the depth true value, and setting the pixel values ​​of the pixel points in the target image that do not belong to the projection point as the preset value, to obtain a depth true value image, wherein the depth true value is obtained based on the point cloud computing corresponding to the pixel points; The perception model is trained using the true depth image and the target image to obtain a target perception model, and the target perception model is used for target detection.

2. The model training method according to claim 1, characterized in that: The multiple projection points include a first projection point and a second projection point, the first projection point is a projection point corresponding to a first point cloud belonging to a foreground class in the point cloud data, and the second projection point is a projection point corresponding to a second point cloud belonging to a background class in the point cloud data; After projecting the point cloud in the point cloud data onto the target image to obtain a plurality of projection points, setting the pixel values ​​of the pixel points in the target image belonging to the projection points to the true depth values, and setting the pixel values ​​of the pixel points in the target image not belonging to the projection points to the preset values, before obtaining the true depth value image, the method further includes: For each pixel point in the target image, execute: if the pixel point does not belong to the first projection point or the second projection point, interpolate according to the point cloud corresponding to the first projection point to obtain a third point cloud, and use the pixel point as the projection point corresponding to the third point cloud.

3. The model training method according to claim 2, characterized in that: The step of projecting the point cloud in the point cloud data onto the target image to obtain a plurality of projection points comprises: Detecting the target object in the point cloud data to obtain at least one detection frame, wherein each detection frame does not overlap with another; For each of the detection frames, the following steps are performed: screening the point clouds in the detection frame to remove point clouds whose distance from the ground is less than a first preset distance, to obtain a plurality of the first point clouds; According to the external parameter matrix and the internal parameter matrix of the camera, the first point cloud is projected onto the target image to obtain a plurality of the first projection points, and the camera is used to acquire the target image; According to the external parameter matrix and the internal parameter matrix, the second point cloud is projected onto the target image to obtain a plurality of second projection points, where the second point cloud is a point cloud outside the detection box in the point cloud data.

4. The model training method according to claim 3, characterized in that: The step of projecting the first point cloud onto the target image according to the external parameter matrix and the internal parameter matrix of the camera to obtain a plurality of the first projection points includes: According to the external parameter matrix and the internal parameter matrix of the camera, each of the first point clouds is processed as follows: Performing coordinate transformation on the first point cloud according to the external parameter matrix and the internal parameter matrix to obtain a first value, a second value, and a third value, wherein the first value is a pixel position of the first point cloud projected to the target image in a first direction, the second value is a pixel position of the first point cloud projected to the target image in a second direction, the first direction is perpendicular to the second direction, and the third value is a true depth value corresponding to the first point cloud; When the first value, the second value, and the third value satisfy a preset condition, the projection point corresponding to the first point cloud is determined to be a first projection point determined according to the first value and the second value.

5. The model training method according to claim 2, characterized in that: If the pixel point does not belong to the first projection point or the second projection point, interpolating according to the point cloud corresponding to the first projection point to obtain a third point cloud includes: If the pixel point does not belong to the first projection point or the second projection point, obtaining a plurality of candidate pixel points within a preset range of the pixel point in the target image; Dividing the multiple candidate pixel points to obtain multiple symmetrically distributed areas; For each of the regions, the following steps are performed: filtering the pixels in the region and retaining a first pixel, where the first pixel is a pixel belonging to the first projection point; if there are multiple first pixels, retaining the first pixel closest to the pixel; In the case that each of the regions includes a first pixel point, interpolation processing is performed according to a point cloud corresponding to the first pixel point included in each of the regions to obtain a third point cloud corresponding to the pixel point.

6. The model training method according to claim 5, characterized in that: In the case where each of the regions includes a first pixel point, interpolation processing is performed according to a point cloud corresponding to the first pixel point included in each of the regions to obtain a third point cloud corresponding to the pixel point, including: In the case where each of the regions includes a first pixel point, if the distance between any two of the first pixel points included in each region is less than a second preset distance, interpolation processing is performed based on the point cloud corresponding to the first pixel points included in each region to obtain a third point cloud corresponding to the pixel point, and the third point cloud belongs to the foreground class.

7. The model training method according to claim 1, characterized in that: The step of training the perception model using the true depth image and the target image to obtain a target perception model includes: Inputting the target image into a perception model to obtain a depth estimation value image of the target image by the perception model; Determining a loss function value of the perception model according to the true depth image and the estimated depth image; The loss function value is used to adjust the model parameters of the perception model to obtain the target perception model.

8. The model training method according to claim 7, characterized in that: The step of determining the loss function value of the perception model according to the true depth image and the estimated depth image comprises: According to the true depth image, a second set and a third set are obtained, wherein the second set includes a set of pixel values ​​of all pixel points in the point cloud data obtained by projecting the point cloud belonging to the background class, and the third set includes a set of pixel values ​​of all pixel points in the point cloud data obtained by projecting the point cloud belonging to the foreground class; According to the depth estimation value image of the target image, a fourth set and a fifth set are obtained, wherein the fourth set includes a set of depth estimation values ​​of all pixel points obtained by projecting the point cloud belonging to the background class in the point cloud data, and the fifth set includes a set of depth estimation values ​​of all pixel points obtained by projecting the point cloud belonging to the foreground class in the point cloud data; Based on the second set and the fourth set, a preset initial loss function is used for calculation to obtain a first loss sub-function value; Based on the third set and the fifth set, the initial loss function is used to perform calculation to obtain a second loss sub-function value; The sum of the first loss sub-function value and the second loss sub-function value is used as the loss function value of the perception model.

9. The model training method according to claim 8, characterized in that: Based on the second set and the fourth set, a preset initial loss function is used for calculation to obtain a first loss sub-function value, including: Based on the second set and the fourth set, a preset initial loss function is used to perform calculation to obtain a first loss value; Multiplying the first loss value by a first coefficient to obtain a first loss sub-function value, wherein the first coefficient is a ratio of a fourth value to a fifth value, the fourth value is the number of pixels in the depth true value image obtained by projecting a point cloud belonging to a background class, and the fifth value is the number of pixels in the depth true value image that do not belong to the point cloud projection; and / or, Based on the third set and the fifth set, the initial loss function is used for calculation to obtain a second loss sub-function value, including: Based on the third set and the fifth set, using the initial loss function to perform calculation to obtain a second loss value; Multiply the second loss value by the second coefficient to obtain a second loss sub-function value, wherein the second coefficient is the ratio of the sixth value to the fifth value, and the sixth value is the number of pixels obtained by projecting the point cloud belonging to the foreground class in the depth truth image.

10. A target detection method, characterized in that: The method comprises: Acquire the image to be detected; The image to be detected is input into a target perception model to obtain a detection target, and the target perception model is obtained by using the model training method described in any one of claims 1 to 9.

11. An electronic device, characterized in that: include: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the model training method as described in any one of claims 1-9, and / or the target detection method as described in claim 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the model training method as described in any one of claims 1 to 9, and / or the target detection method as described in claim 10.

13. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the model training method as described in any one of claims 1 to 9, and / or the target detection method as described in claim 10.

Images