A target detection sample omission determination method, device and equipment

Abstract

The embodiment of the application discloses a target detection sample omission determination method, device and equipment, the method comprises the following steps: obtaining a first detection target set of a target image detected by a camera on a vehicle; determining a depth image of the target image in a camera coordinate system; determining a point cloud of a measured object in a vehicle coordinate system according to the depth image; determining a second detection target set, the detection targets in the second detection target set are obtained by clustering the point cloud; determining the positions of the detection targets in the second detection target set in a pixel coordinate system to obtain a third detection target set; respectively determining the boundaries of the detection targets in the first detection target set and the third detection target set to obtain a first boundary set and a second boundary set; determining the boundaries contained in the second boundary set and not contained in the first boundary set to obtain a third boundary set, and determining the result of the target detection sample omission; so as to improve the efficiency of determining the target detection sample omission.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer, in particular to a target detection sample omission determination method, device and equipment. BACKGROUND

[0002] In the process of automatic driving, the image detected by the camera on the vehicle is usually subjected to target detection, mainly by using a target detection model to realize target detection. The result obtained by the target detection model may have the case of missing detection target, for example, the detection result misses one or more detection targets existing in the image. At this time, one or more missed detection targets and the label of the detection target are usually determined manually, and the image, the detection target and the corresponding label are added to the training data of the target detection model to improve the target detection capability of the target detection model.

[0003] However, the manual determination of the missed detection target is low in efficiency, and there is an urgent need for a target detection sample omission determination method to improve the efficiency of determining the target detection sample omission. SUMMARY

[0004] Therefore, the present application provides a target detection sample omission determination method, device and equipment to improve the efficiency of determining the target detection sample omission.

[0005] In a first aspect, the present application provides a target detection sample omission determination method, which comprises:

[0006] obtaining a first detection target set of a target image detected by a camera on a vehicle;

[0007] determining a depth image of the target image in the camera coordinate system, wherein the gray value of a point in the depth image is the distance between the measured object and the camera;

[0008] determining a point cloud of the measured object in the vehicle coordinate system according to the depth image;

[0009] determining a second detection target set, wherein the detection targets in the second detection target set are obtained by clustering the point cloud, and the detection targets in the second detection target set correspond one-to-one to the classes of the point cloud;

[0010] determining the positions of the detection targets in the second detection target set in the pixel coordinate system to obtain a third detection target set;

[0011] respectively determining the boundaries of the detection targets in the first detection target set and the third detection target set to obtain a first boundary set and a second boundary set;

[0012] determining the boundaries contained in the second boundary set and not contained in the first boundary set to obtain a third boundary set;

[0013] According to the third boundary set, a result of missing of the target detection sample is determined.

[0014] In a possible implementation, after the point cloud of the measured object in the vehicle coordinate system is determined according to the depth image, the method further includes:

[0015] inputting the target image into the target region boundary detection model to obtain a two-dimensional boundary of the target region in the target image;

[0016] obtaining a three-dimensional boundary of the target region in the vehicle coordinate system according to the two-dimensional boundary of the target region;

[0017] obtaining the target point cloud according to the three-dimensional boundary of the target region;

[0018] The second detection target set is determined, the detection targets in the second detection target set are obtained by clustering the point cloud, and the detection targets in the second detection target set and the classes of the point cloud are in one-to-one correspondence.

[0019] The second detection target set is determined, the detection targets in the second detection target set are obtained by clustering the target point cloud, and the detection targets in the second detection target set and the classes of the target point cloud are in one-to-one correspondence.

[0020] In a possible implementation, the second detection target set is determined, the detection targets in the second detection target set are obtained by clustering the point cloud, and the detection targets in the second detection target set and the classes of the point cloud are in one-to-one correspondence, including:

[0021] The second detection target set is determined, the detection targets in the second detection target set are obtained by clustering the point cloud using the K-Means clustering algorithm, and the detection targets in the second detection target set and the classes of the point cloud are in one-to-one correspondence.

[0022] In a possible implementation, the boundaries of the detection targets in the first detection target set and the third detection target set are respectively determined to obtain the first boundary set and the second boundary set, including:

[0023] The rectangular boundaries of the detection targets in the first detection target set and the third detection target set are respectively determined to obtain the first boundary set and the second boundary set.

[0024] In a second aspect, the present application provides a device for determining missing of a target detection sample, and the device includes:

[0025] The two-dimensional data processing unit is configured to: acquire a first detection target set of a target image detected by a camera on a vehicle; and determine a depth image of the target image in a camera coordinate system, wherein a gray value of a point in the depth image is a distance between a measured object and the camera.

[0026] The three-dimensional data processing unit is configured to determine a point cloud of the measured object in a vehicle coordinate system according to the depth image; determine a second detection target set, the detection targets in the second detection target set being obtained by clustering the point cloud, and the detection targets in the second detection target set corresponding to the classes of the point cloud in a one-to-one manner; determine the positions of the detection targets in the second detection target set in a pixel coordinate system to obtain a third detection target set.

[0027] The two-dimensional data processing unit is configured to respectively determine the boundaries of the detection targets in the first detection target set and the third detection target set to obtain a first boundary set and a second boundary set; determine the boundaries contained in the second boundary set and not contained in the first boundary set to obtain a third boundary set; and determine the result of the missing of the target detection sample according to the third boundary set.

[0028] In a possible implementation, the two-dimensional data processing unit is further configured to input the target image into a target region boundary detection model to obtain a two-dimensional boundary of the target region in the target image; obtain a three-dimensional boundary of the target region in the vehicle coordinate system according to the two-dimensional boundary of the target region; and obtain a target point cloud according to the three-dimensional boundary of the target region.

[0029] The three-dimensional data processing unit is specifically configured to determine the second detection target set, the detection targets in the second detection target set being obtained by clustering the target point cloud, and the detection targets in the second detection target set corresponding to the classes of the target point cloud in a one-to-one manner.

[0030] In a possible implementation, the three-dimensional data processing unit is specifically configured to determine the second detection target set, the detection targets in the second detection target set being obtained by clustering the point cloud using a K-Means clustering algorithm, and the detection targets in the second detection target set corresponding to the classes of the point cloud in a one-to-one manner.

[0031] In a possible implementation, the two-dimensional data processing unit is configured to respectively determine the rectangular boundaries of the detection targets in the first detection target set and the third detection target set to obtain the first boundary set and the second boundary set.

[0032] In a third aspect, the present application provides a device for confirming missing of a target detection sample, and the device comprises a processor and a memory, wherein the memory stores a code, and the processor is configured to call the code stored in the memory to execute the method according to any one of the preceding aspects.

[0033] In a fourth aspect, the present application provides a computer readable storage medium, and the computer readable storage medium is configured to store a computer program, and the computer program is configured to execute the method according to any one of the preceding aspects. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a flowchart of a method for determining missing target detection samples provided by an embodiment of the present application;

[0035] Figure 2 is a structural schematic diagram of a device for determining missing target detection samples provided by an embodiment of the present application;

[0036] Figure 3 is a structural schematic diagram of an apparatus for determining missing target detection samples provided by an embodiment of the present application. DETAILED DESCRIPTION

[0037] Currently, one or more missed detection targets and labels of the detection targets are usually determined manually, and the image, the detection targets and the corresponding labels are added to the training data of the target detection model to improve the target detection capability of the target detection model. However, the manual determination of the missed detection targets is low in efficiency, and there is an urgent need for a determination method for missing target detection samples to improve the efficiency of determining missing target detection samples.

[0038] Based on this, in the embodiments of the present application provided by the applicant, a first detection target set of a target image detected by a camera on a vehicle is obtained; a depth image of the target image in a camera coordinate system is determined, and a gray value of a point in the depth image is a distance between a measured object and the camera; a point cloud of the measured object in a vehicle coordinate system is determined according to the depth image; a second detection target set is determined, and a detection target in the second detection target set is obtained by clustering the point cloud, and the detection target in the second detection target set and the point cloud are one-to-one corresponding; a position of the detection target in the second detection target set in a pixel coordinate system is determined to obtain a third detection target set; boundaries of the detection targets in the first detection target set and the third detection target set are respectively determined to obtain a first boundary set and a second boundary set; a boundary contained in the second boundary set and not contained in the first boundary set is determined to obtain a third boundary set; and a result of missing target detection samples is determined according to the third boundary set.

[0039] According to the depth image of the target image, the point cloud obtained from the depth image is clustered to obtain the detection target; the boundary of the detection target obtained by clustering the point cloud is compared with the boundary of the detection target in the first detection target set, and the determination of the missing target detection samples is automatically realized, which can improve the detection efficiency.

[0040] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, the method, device and apparatus for determining missing target detection samples provided by the embodiments of the present application are described below with reference to the accompanying drawings.

[0041] While the exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be carried out in various forms and should not be limited by the embodiments set forth herein. Other embodiments obtained by those skilled in the art based on the embodiments in the present application without making creative contributions are within the scope of the present application.

[0042] In the claims and specification of the present application and the accompanying drawings, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0043] The present application provides a target detection sample omission determination method.

[0044] Please refer to Figure 1 , Figure 1 The target detection sample omission determination method provided by the embodiments of the present application.

[0045] As Figure 1 shown, the target detection sample omission determination method in the embodiments of the present application includes S101-S108.

[0046] S101, obtaining a first detection target set of a target image detected by a camera on a vehicle.

[0047] The target image is obtained by detecting the measured object by the camera, and the camera is located on the vehicle.

[0048] The first detection target set includes one or more detection targets.

[0049] S102, determining a depth image of the target image in the camera coordinate system, and the gray value of a point in the depth image is the distance between the measured object and the camera.

[0050] The depth image can also be called a distance image, which refers to an image in which the distance from the camera to each point in the scene is taken as the pixel value, and which can reflect the depth distance of the photographed space.

[0051] Each pixel point in the depth image represents the distance of the measured object from the camera plane.

[0052] S103, determining a point cloud of the measured object in the vehicle coordinate system according to the depth image.

[0053] The conversion of the depth image into the point cloud is essentially a coordinate system transformation, and according to the extrinsic relationship matrix of the camera with respect to the vehicle, the depth image can be converted into the point cloud in the vehicle coordinate system.

[0054] The rotation matrix and the translation matrix jointly describe the conversion relationship between the vehicle coordinate system and the camera coordinate system; the rotation matrix describes the direction of the coordinate axis of the vehicle coordinate system relative to the coordinate axis of the camera coordinate system, and the translation matrix describes the position of the origin of the camera coordinate system in the vehicle coordinate system.

[0055] S104, a second detection target set is determined, the detection targets in the second detection target set are obtained by clustering the point cloud, and the detection targets in the second detection target set correspond one-to-one to the classes of the point cloud.

[0056] The clustering of the point cloud refers to clustering the points in the point cloud, and dividing the points in the point cloud into different classes, each class corresponding to a detection target.

[0057] S105, the positions of the detection targets in the second detection target set in the pixel coordinate system are determined, and a third detection target set is obtained.

[0058] The detection targets in the second detection target set are detection targets in a three-dimensional coordinate system, and the detection targets in the third detection target set are detection targets in a pixel coordinate system (two-dimensional coordinate system).

[0059] S106, the boundaries of the detection targets in the first detection target set and the third detection target set are respectively determined, and a first boundary set and a second boundary set are obtained.

[0060] One or more boundaries in the first boundary set are the boundaries of one or more detection targets in the first detection target set.

[0061] One or more boundaries in the second boundary set are the boundaries of one or more detection targets in the third detection target set.

[0062] S107, a third boundary set is obtained by determining the boundaries contained in the second boundary set and not contained in the first boundary set.

[0063] The third boundary set can contain one or more boundaries, or can not contain boundaries.

[0064] S108, the result of missing target detection samples is determined according to the third boundary set.

[0065] When the third boundary set does not contain boundaries, the first detection target set has no missing target detection samples; when the third boundary set contains one or more boundaries, the first detection target set has missing target detection samples, and the missing detection targets are the detection targets corresponding to the one or more boundaries in the third boundary set.

[0066] According to the depth image of the target image, and clustering the point cloud obtained from the depth image, a detection target is obtained; and the boundary of the detection target obtained by clustering the point cloud is compared with the boundary of the detection target in the first detection target set, so that the determination of the missing target detection sample is automatically realized, and the detection efficiency is improved.

[0067] The application further provides another method for determining missing target detection samples.

[0068] The method for determining missing target detection samples provided by the embodiments of the application comprises S201-S211.

[0069] S201, obtaining a first detection target set of a target image, the target image being detected by a camera.

[0070] According to the target image, a two-dimensional position coordinate of the measured object in a pixel coordinate system and a pixel value of a corresponding pixel point of the measured object can be obtained.

[0071] The first detection target set can be obtained by using an original target detection model.

[0072] The original target detection model is used to realize target detection, that is, a process of detecting a detection target from an image.

[0073] S202, determining a depth image of the target image in a camera coordinate system, the gray value of a point in the depth image being the distance between the measured object and the camera.

[0074] In some possible implementation manners, the depth image can be obtained by inputting the target image into a depth network.

[0075] S203, determining a point cloud of the measured object in a vehicle coordinate system according to the depth image.

[0076] S204, inputting the target image into a target region boundary detection model to obtain a two-dimensional boundary of a target region in the target image.

[0077] The target region can be one or more regions in the target image, and the boundary of the target region is the one or more boundaries.

[0078] The two-dimensional boundary of the target region is a two-dimensional boundary in a pixel coordinate system.

[0079] For example, the target image is an image detected by a camera on a vehicle, and the target region is a region corresponding to a road on which the vehicle travels, or a region corresponding to part or all of the road in the target image. At this time, the target region boundary detection model can be a road boundary detection model.

[0080] S205, obtaining a three-dimensional boundary of the target region in the vehicle coordinate system according to the two-dimensional boundary of the target region.

[0081] In S205, the two-dimensional boundary in the pixel coordinate system is converted into the three-dimensional vehicle coordinate system.

[0082] S206, obtaining a target point cloud according to the three-dimensional boundary of the target region, data in the target point cloud being data of the point cloud within the three-dimensional boundary of the target region.

[0083] That is, the point cloud is filtered by using the three-dimensional boundary of the target region.

[0084] Within the three-dimensional boundary of the target region means within a three-dimensional region constituted by the three-dimensional boundary.

[0085] S207, determining a second detection target set, a detection target in the second detection target set being obtained by clustering the target point cloud, and the detection target in the second detection target set corresponding to a class of the target point cloud in one-to-one correspondence.

[0086] Clustering the target point cloud means clustering points in the point cloud, and dividing the points in the target point cloud into different classes, each class corresponding to a detection target.

[0087] In some possible cases, the target point cloud is clustered according to distance, which can be completed by using a common clustering algorithm, for example, a K-Means clustering algorithm is used to complete clustering of the target point cloud.

[0088] The detection target in the second detection target set refers to a detection target in the vehicle coordinate system.

[0089] S208, determining a position of the detection target in the second detection target set in the pixel coordinate system, and obtaining a third detection target set.

[0090] The detection target in the third detection target set refers to a detection target in the pixel coordinate system.

[0091] S209, respectively determining boundaries of the detection targets in the first detection target set and the third detection target set, and obtaining a first boundary set and a second boundary set.

[0092] The boundary in the first boundary set is a set of the detection targets in the first target set, and can include one or more boundaries.

[0093] The boundary in the second boundary set is a set of the detection targets in the third target set, and can include one or more boundaries.

[0094] In some possible implementations, the boundaries of the detection targets in the first detection target set and the third detection target set are determined respectively to obtain a first boundary set and a second boundary set, specifically including: determining the rectangular boundaries of the detection targets in the first detection target set and the third detection target set respectively to obtain a first boundary set and a second boundary set.

[0095] That is, the shape of the boundary is determined to be a rectangle.

[0096] The first boundary set includes one or more boundaries, and each boundary corresponds to a bounding box.

[0097] In a possible implementation, the bounding box is displayed in a rectangular box in a pixel coordinate system.

[0098] S210 : Determine boundaries included in the second boundary set and not included in the first boundary set to obtain a third boundary set.

[0099] The third boundary set may include one or more boundaries, or may not include any boundaries.

[0100] S211 : Determine the result of target detection sample omission according to the third boundary set.

[0101] When the third boundary set does not contain any boundary, it indicates that the target detection results obtained by the original target detection model do not miss any target detection samples.

[0102] When the third boundary set contains a or boundary, it indicates that the target detection result obtained by the original target detection model has no target detection samples missed.

[0103] In some possible cases, when the third boundary set contains one or boundaries, it indicates that there is an omission. The label of the missed detection target can also be determined, and the target image, the missed detection target and the label of the missed detection target are added to the training data, and the target detection model is trained based on the training data to improve the accuracy of target detection.

[0104] By utilizing the solution of this embodiment, it is possible to automatically determine whether target detection samples are missing, which improves efficiency compared to manual determination of whether there are any omissions.

[0105] The embodiment of the present application also provides a device for determining missing target detection samples.

[0106] See also Figure 2 , Figure 2 It is a structural diagram of a device for determining target detection sample omission provided in an embodiment of the present application.

[0107] like Figure 2As shown, the target detection sample omission determination apparatus in the embodiment of the application comprises:

[0108] The two-dimensional data processing unit 201 is configured to acquire a first detection target set of a target image detected by a camera on a vehicle, determine a depth image of the target image in a camera coordinate system, and set a gray value of a point in the depth image as a distance between a measured object and the camera.

[0109] The three-dimensional data processing unit 202 is configured to determine a point cloud of the measured object in a vehicle coordinate system according to the depth image, determine a second detection target set, the detection targets in the second detection target set being obtained by clustering the point cloud, the detection targets in the second detection target set corresponding to the classes of the point cloud in a one-to-one manner, determine positions of the detection targets in the second detection target set in a pixel coordinate system, and obtain a third detection target set.

[0110] The two-dimensional data processing unit 201 is further configured to determine boundaries of the detection targets in the first detection target set and the third detection target set respectively, obtain a first boundary set and a second boundary set, determine boundaries contained in the second boundary set and not contained in the first boundary set, obtain a third boundary set, and determine a result of target detection sample omission according to the third boundary set.

[0111] The units included in the above target detection sample omission determination apparatus can achieve the same technical effects as the target detection sample omission determination method in the above embodiments, and thus, details are not repeated here.

[0112] The embodiment of the application further provides a target detection sample omission determination apparatus.

[0113] Please refer to Figure 3 , Figure 3 which is a structural schematic diagram of the target detection sample omission determination apparatus provided by the embodiment of the application.

[0114] As Figure 3 shown, the target detection sample omission determination apparatus in the embodiment of the application comprises a processor 301 and a memory 302, wherein the memory 302 stores codes, and the processor 301 is configured to call the codes stored in the memory to execute the method of any one of the above embodiments.

[0115] The units included in the above target detection sample omission determination apparatus can achieve the same technical effects as the target detection sample omission determination method in the above embodiments, and thus, details are not repeated here.

[0116] In the embodiments of the present application, a computer readable storage medium for storing a computer program for executing the above-mentioned target detection sample omission determination method and achieving the same technical effects is also provided, and details are not repeated here. The computer readable storage medium is, for example, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

[0117] The above description of disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for determining target detection sample omission, characterized in that: The method comprises: Acquire a first detection target set of a target image detected by a camera on a vehicle; the first detection target set includes one or more detection targets; Determine the depth image of the target image in the camera coordinate system. The grayscale value of the midpoint of the depth image is the distance between the measured object and the camera. Determine the point cloud of the object being measured in the vehicle coordinate system based on the depth image; Input the target image to the target area boundary detection model to obtain the two-dimensional boundary of the target area in the target image; According to the two-dimensional boundary of the target area, the three-dimensional boundary of the target area in the vehicle coordinate system is obtained; According to the three-dimensional boundary of the target area, the target point cloud is obtained; Determine a second detection target set, where the detection targets in the second detection target set are obtained by clustering the target point cloud, and the detection targets in the second detection target set correspond to the classes of the target point cloud in a one-to-one manner; Determine positions of detection targets in the second detection target set in a pixel coordinate system to obtain a third detection target set; Determining boundaries of detection targets in the first detection target set and the third detection target set, respectively, to obtain a first boundary set and a second boundary set; one or more boundaries in the first boundary set are boundaries of one or more detection targets in the first detection target set; one or more boundaries in the second boundary set are boundaries of one or more detection targets in the third detection target set; Determine the boundaries included in the second boundary set and not included in the first boundary set to obtain a third boundary set; According to the third boundary set, the result of missing target detection samples is determined.

2. The method according to claim 1, characterized in that The determining of the second detection target set, where the detection targets in the second detection target set are obtained by clustering the point cloud, and the detection targets in the second detection target set correspond to the classes of the point cloud in a one-to-one manner, includes: A second detection target set is determined, where the detection targets in the second detection target set are obtained by clustering the point cloud using a K-Means clustering algorithm, and the detection targets in the second detection target set correspond one-to-one to the classes of the point cloud.

3. The method according to claim 1, characterized in that The determining the boundaries of the detection targets in the first detection target set and the third detection target set respectively to obtain the first boundary set and the second boundary set includes: Rectangular boundaries of the detection targets in the first detection target set and the third detection target set are determined respectively to obtain a first boundary set and a second boundary set.

4. A device for determining target detection sample omission, characterized in that: The device comprises: A two-dimensional data processing unit is configured to obtain a first detection target set of a target image detected by a camera on a vehicle; the first detection target set includes one or more detection targets; determine a depth image of the target image in the camera coordinate system, wherein the grayscale value of a point in the depth image is the distance between the detected object and the camera; The two-dimensional data processing unit is further used to input the target image into the target area boundary detection model to obtain the two-dimensional boundary of the target area in the target image; obtain the three-dimensional boundary of the target area in the vehicle coordinate system based on the two-dimensional boundary of the target area; and obtain the target point cloud based on the three-dimensional boundary of the target area; The three-dimensional data processing unit is configured to determine, based on the depth image, a point cloud of the object being measured in the vehicle coordinate system; determine a second detection target set, where the detection targets in the second detection target set are obtained by clustering the point cloud, and the detection targets in the second detection target set correspond one-to-one to the classes of the point cloud; and determine positions of the detection targets in the second detection target set in the pixel coordinate system to obtain a third detection target set; The three-dimensional data processing unit is specifically used to determine a second detection target set, where the detection targets in the second detection target set are obtained by clustering the target point cloud, and the detection targets in the second detection target set correspond to the classes of the target point cloud in a one-to-one correspondence; A two-dimensional data processing unit is used to respectively determine the boundaries of the detection targets in the first detection target set and the third detection target set to obtain a first boundary set and a second boundary set; one or more boundaries in the first boundary set are the boundaries of one or more detection targets in the first detection target set; one or more boundaries in the second boundary set are the boundaries of one or more detection targets in the third detection target set; determine the boundaries included in the second boundary set and not included in the first boundary set to obtain a third boundary set; and determine the results of target detection sample omissions based on the third boundary set.

5. The device according to claim 4, characterized in that The three-dimensional data processing unit is specifically used to determine a second detection target set, where the detection targets in the second detection target set are obtained by clustering the point cloud using the K-Means clustering algorithm, and the detection targets in the second detection target set correspond one-to-one to the classes of the point cloud.

6. The device according to claim 4, characterized in that The two-dimensional data processing unit is used to respectively determine the rectangular boundaries of the detection targets in the first detection target set and the third detection target set to obtain a first boundary set and a second boundary set.

7. A device for confirming the omission of target detection samples, characterized in that: The device includes a processor and a memory, wherein the memory stores code, and the processor is configured to call the code stored in the memory to execute the method according to any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method according to any one of claims 1 to 3.

Images