Method and device for acquiring two-dimensional target depth value of image, equipment and medium

Abstract

The invention relates to the technical field of image processing, in particular to an image two-dimensional target depth value obtaining method and device, equipment and a medium. The method comprises the following steps: acquiring image data and point cloud data; performing instance segmentation on the image data by using the instance segmentation model to obtain a two-dimensional bounding box and a corresponding segmentation mask of each two-dimensional target in the image data; and performing compensation processing on the point cloud data, projecting the compensated point cloud data to the two-dimensional bounding box of each two-dimensional target in the image data and the corresponding segmentation mask, and performing calculation to obtain a depth value of at least one two-dimensional target. Therefore, the point cloud is projected to the corresponding two-dimensional target mask area, and the abnormal value of the point cloud depth falling in the area is removed to obtain the accurate depth, so that the problems that the target depths are inconsistent and the depth of a point-cloud-free target at a distance cannot be obtained in the prior art are solved, and the safety of automatic driving is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment and medium for acquiring a two-dimensional target depth value of an image. Background Art

[0002] At present, the key to realizing autonomous driving technology lies in comprehensive and accurate three-dimensional perception of the external environment. The types of sensors currently used include LiDAR, cameras, and millimeter-wave radar. LiDAR can obtain extremely accurate location data, however, it has problems such as uneven point cloud distribution, sparse point cloud at long distances, and lack of semantic information. In addition, due to its high cost, it is not suitable for large-scale mass production, so it is usually only configured on data collection vehicles. In contrast, camera sensors can capture rich semantic information and are relatively low in cost, but their disadvantage is the lack of reliable depth information, which is crucial to ensure the safety of autonomous driving.

[0003] In the related art, a method of directly projecting point cloud data onto an image is usually adopted to obtain a depth value of a corresponding position.

[0004] However, this method easily leads to inconsistent depth information of the same target object, resulting in ambiguity. In addition, for distant targets, if there is a lack of point cloud data support, the corresponding depth information cannot be obtained, which needs to be solved urgently. Summary of the invention

[0005] The present application provides a method, device, equipment and medium for obtaining the two-dimensional target depth value of an image to solve the problems of inconsistent target depth and inability to obtain the depth of distant point cloud-free targets in the prior art, thereby improving the safety of autonomous driving.

[0006] To achieve the above object, a first aspect of the present application provides a method for obtaining a two-dimensional target depth value of an image, comprising the following steps:

[0007] Acquire image data and point cloud data;

[0008] Performing instance segmentation on the image data using an instance segmentation model to obtain a two-dimensional bounding box and a corresponding segmentation mask for each two-dimensional object in the image data;

[0009] The point cloud data is compensated, the compensated point cloud data is projected to a two-dimensional bounding box of each two-dimensional target in the image data and a corresponding segmentation mask, and a depth value of at least one two-dimensional target is calculated.

[0010] According to an embodiment of the present application, projecting the compensated point cloud data to a two-dimensional bounding box and a corresponding segmentation mask of each two-dimensional target in the image data, and calculating a depth value of at least one two-dimensional target, includes:

[0011] Projecting the compensated point cloud data to a two-dimensional bounding box and a corresponding segmentation mask of each two-dimensional target in the image data, and determining whether there is an overlapping area between the compensated point cloud data and the two-dimensional bounding box;

[0012] If the compensated point cloud data and the two-dimensional bounding box have the overlapping area, determining the current two-dimensional bounding box based on the overlapping area;

[0013] Selecting the intersection of the preset range of the center of the current two-dimensional bounding box and the segmentation mask corresponding to the current two-dimensional bounding box as the effective projection area;

[0014] Calculate all point cloud depth values ​​in the effective projection area, and use a preset interquartile range strategy to remove abnormal values ​​in all point cloud depth values ​​to obtain remaining point cloud depth values;

[0015] A minimum depth value among the remaining point cloud depth values ​​is selected as the depth value of the two-dimensional object corresponding to the current two-dimensional bounding box.

[0016] According to an embodiment of the present application, after determining whether the compensated point cloud data and the two-dimensional bounding box have the overlapping area, the method further includes:

[0017] If the compensated point cloud data and the two-dimensional bounding box do not have the overlapping area, using the laser radar to perform at least one observation to obtain the three-dimensional size information of the current target;

[0018] Based on the three-dimensional size information of the current target, a two-dimensional target depth value corresponding to the current target is obtained using a preset nonlinear optimization algorithm.

[0019] According to one embodiment of the present application, after obtaining the two-dimensional bounding box of each target in the image data and the corresponding segmentation mask, the method further includes:

[0020] The two-dimensional bounding box and the corresponding segmentation mask are screened to filter out objects in the image data that meet a preset occlusion condition.

[0021] According to one embodiment of the present application, the compensating processing of the point cloud data includes:

[0022] Based on the external parameters of the laser radar, the preset RTK (Real Time Kinematic) technology is used to adjust the point cloud data to meet the same timestamp as the image data to obtain the compensated point cloud data.

[0023] According to the method for obtaining the depth value of a two-dimensional target in an image proposed in an embodiment of the present application, by obtaining image data and point cloud data, and performing instance segmentation on the image data using an instance segmentation model, a two-dimensional bounding box and a corresponding segmentation mask of each two-dimensional target in the image data can be obtained; then the point cloud data is compensated, and the compensated point cloud data is projected to the two-dimensional bounding box and the corresponding segmentation mask of each two-dimensional target in the image data, thereby calculating the depth value of at least one two-dimensional target. Thus, by projecting the point cloud to the corresponding two-dimensional target mask area and removing outliers from the point cloud depth falling in the area to obtain an accurate depth, the problem of inconsistent target depth and inability to obtain the depth of distant targets without point clouds in the prior art is solved, thereby improving the safety of autonomous driving.

[0024] To achieve the above-mentioned purpose, a second embodiment of the present application provides a device for acquiring a two-dimensional target depth value of an image, comprising:

[0025] An acquisition module, used to acquire image data and point cloud data;

[0026] A segmentation module, used to perform instance segmentation on the image data using an instance segmentation model to obtain a two-dimensional bounding box and a corresponding segmentation mask for each two-dimensional object in the image data;

[0027] A calculation module is used to perform compensation processing on the point cloud data, project the compensated point cloud data to the two-dimensional bounding box and the corresponding segmentation mask of each two-dimensional target in the image data, and calculate the depth value of at least one two-dimensional target.

[0028] According to one embodiment of the present application, the computing module includes:

[0029] a judging unit, configured to project the compensated point cloud data to a two-dimensional bounding box and a corresponding segmentation mask of each two-dimensional target in the image data, and to judge whether there is an overlapping area between the compensated point cloud data and the two-dimensional bounding box;

[0030] a determining unit, configured to determine a current two-dimensional bounding box based on the overlapping area when the compensated point cloud data and the two-dimensional bounding box have the overlapping area;

[0031] A first selection unit, configured to select an intersection of a preset range of the center of the current two-dimensional bounding box and a segmentation mask corresponding to the current two-dimensional bounding box as a valid projection area;

[0032] A screening unit, configured to calculate all point cloud depth values ​​within the effective projection area, and remove abnormal values ​​from all point cloud depth values ​​using a preset interquartile range strategy to obtain remaining point cloud depth values;

[0033] The second selection unit is used to select a minimum depth value among the remaining point cloud depth values ​​as the depth value of the two-dimensional target corresponding to the current two-dimensional bounding box.

[0034] According to an embodiment of the present application, after determining whether the compensated point cloud data and the two-dimensional bounding box have the overlapping area, the determining unit is further configured to:

[0035] When the compensated point cloud data and the two-dimensional bounding box do not have the overlapping area, using a laser radar to perform at least one observation to obtain three-dimensional size information of the current target;

[0036] Based on the three-dimensional size information of the current target, a two-dimensional target depth value corresponding to the current target is obtained using a preset nonlinear optimization algorithm.

[0037] According to one embodiment of the present application, after obtaining the two-dimensional bounding box of each target in the image data and the corresponding segmentation mask, the segmentation module is further used to:

[0038] The two-dimensional bounding box and the corresponding segmentation mask are screened to filter out objects in the image data that meet a preset occlusion condition.

[0039] According to one embodiment of the present application, the computing module is specifically used for:

[0040] Based on the external parameters of the laser radar, the preset RTK technology is used to adjust the point cloud data to meet the same timestamp as the image data to obtain the compensated point cloud data.

[0041] According to the device for acquiring the depth value of a two-dimensional target of an image proposed in an embodiment of the present application, by acquiring image data and point cloud data, and performing instance segmentation on the image data using an instance segmentation model, a two-dimensional bounding box and a corresponding segmentation mask of each two-dimensional target in the image data can be obtained; then the point cloud data is compensated, and the compensated point cloud data is projected to the two-dimensional bounding box and the corresponding segmentation mask of each two-dimensional target in the image data, thereby calculating the depth value of at least one two-dimensional target. Thus, by projecting the point cloud to the corresponding two-dimensional target mask area and removing outliers from the depth of the point cloud falling in the area to obtain an accurate depth, the problem of inconsistent target depth and inability to obtain the depth of distant targets without point clouds in the prior art is solved, thereby improving the safety of autonomous driving.

[0042] To achieve the above-mentioned objectives, the third aspect embodiment of the present application proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for obtaining the two-dimensional target depth value of the image as described in the above-mentioned embodiment.

[0043] To achieve the above objectives, the fourth aspect of the present application proposes a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method for obtaining a two-dimensional target depth value of an image as described in the above embodiments.

[0044] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0046] Figure 1 A flowchart of a method for obtaining a two-dimensional target depth value of an image provided according to an embodiment of the present application;

[0047] Figure 2 A flowchart of another method for obtaining a two-dimensional target depth value of an image provided according to an embodiment of the present application;

[0048] Figure 3 A schematic diagram of removing outliers using the interquartile range method according to an embodiment of the present application;

[0049] Figure 4 A block diagram of a device for acquiring a two-dimensional target depth value of an image provided by an embodiment of the present application;

[0050] Figure 5 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0052] The following describes a method, device, equipment and medium for obtaining a two-dimensional target depth value of an image according to an embodiment of the present application with reference to the accompanying drawings.

[0053] Figure 1 It is a flowchart of a method for obtaining a two-dimensional target depth value of an image according to an embodiment of the present application.

[0054] For example, Figure 1 As shown, the method for obtaining the two-dimensional target depth value of the image includes the following steps:

[0055] In step S101 , image data and point cloud data are acquired.

[0056] It can be understood that image data refers to the two-dimensional image obtained by taking a camera; point cloud data refers to the point set information in three-dimensional space obtained by sensors such as lidar.

[0057] In step S102, the instance segmentation model is used to perform instance segmentation on the image data to obtain a two-dimensional bounding box and a corresponding segmentation mask for each two-dimensional object in the image data.

[0058] Among them, the instance segmentation model is a deep learning model that can identify and distinguish different object instances in an image, and generate accurate bounding boxes and masks for each instance; a two-dimensional bounding box refers to a rectangular box defined on a two-dimensional image plane, which is used to represent the position and size of the identified object; a segmentation mask is an image processing technique that can mark each pixel in an image as an instance of a specific object. The mask is usually a binary image, in which each pixel is either a foreground (usually white) or a background (usually black), or different grayscale values ​​are used to represent different object instances, thereby distinguishing the target from the background.

[0059] That is to say, when processing the collected image data, the advanced instance segmentation model can be used to accurately identify and segment each object in the image data. Through this process, not only the 2D bounding box of each 2D object can be obtained, but also the corresponding segmentation mask can be obtained, through which the outline and shape of each 2D object can be depicted in detail.

[0060] Furthermore, in some embodiments, after obtaining the two-dimensional bounding box and the corresponding segmentation mask of each target in the image data, the method further includes: screening the two-dimensional bounding box and the corresponding segmentation mask to filter out targets in the image data that meet preset occlusion conditions.

[0061] It is understandable that in order to ensure the accuracy and reliability of the analysis results, the segmentation results (i.e., the two-dimensional bounding box and the corresponding segmentation mask obtained in step S102) can be further screened and filtered. Special attention should be paid to targets that are severely occluded by other objects. Occlusion may cause errors in the segmentation results and affect subsequent analysis and processing. Therefore, a series of algorithms and technical means can be used to identify and exclude these severely occluded targets (i.e., targets that meet the preset occlusion conditions) to ensure that the final data set contains only clear and complete target instances, ensuring the accuracy and effectiveness of subsequent processing.

[0062] In step S103, the point cloud data is compensated, and the compensated point cloud data is projected to the two-dimensional bounding box of each two-dimensional target in the image data and the corresponding segmentation mask, and the depth value of at least one two-dimensional target is calculated.

[0063] The depth value refers to the distance information of a two-dimensional target in three-dimensional space, and usually indicates the distance of an object or target from an observer or camera.

[0064] Specifically, by compensating the point cloud data, the temporal consistency of the point cloud data and the image data can be ensured, thereby improving the accuracy of data fusion. The compensated point cloud data will be projected onto the image data, that is, the point cloud information in the three-dimensional space is converted to the two-dimensional image plane. Through this projection process, each pixel in the image can find its corresponding point in the three-dimensional space and calculate the depth value of at least one two-dimensional target (i.e., the object or area in the image).

[0065] The following describes in detail how to compensate the point cloud data.

[0066] As a possible implementation method, in some embodiments, the point cloud data is compensated, including: based on the external parameters of the lidar, using the preset RTK technology, adjusting the point cloud data to meet the same timestamp as the image data to obtain compensated point cloud data.

[0067] It can be understood that compensating the point cloud data is to adjust the point cloud data to the same timing as the image data. LiDAR and camera are two different sensors, used to collect point cloud data and image data from the environment, respectively. The acquisition frequency of LiDAR is about 10Hz, while the acquisition frequency of camera is about 25Hz. Due to their different acquisition frequencies and the fact that the vehicle (self-vehicle) is still moving when collecting data, RTK technology can be used to ensure the synchronization of point cloud data and image data in time for time alignment.

[0068] Specifically, based on the external parameters of the laser radar, RTK technology is used to accurately adjust the point cloud data and synchronize the timestamps of the point cloud data and image data to ensure that the two are aligned at the same time point, thereby obtaining the point cloud data after compensation processing, making it completely consistent with the image data in time. This compensation processing not only improves the accuracy of the data, but also provides a solid foundation for subsequent data analysis and processing.

[0069] Next, it is described in detail how to project the compensated point cloud data to the two-dimensional bounding box and the corresponding segmentation mask in the image data, and calculate the depth value of at least one two-dimensional target.

[0070] As a possible implementation method, in some embodiments, the compensated point cloud data is projected onto the two-dimensional bounding box and the corresponding segmentation mask of each two-dimensional target in the image data, and the depth value of at least one two-dimensional target is calculated, including: projecting the compensated point cloud data onto the two-dimensional bounding box and the corresponding segmentation mask of each two-dimensional target in the image data, and determining whether there is an overlapping area between the compensated point cloud data and the two-dimensional bounding box; if there is an overlapping area between the compensated point cloud data and the two-dimensional bounding box, determining the current two-dimensional bounding box based on the overlapping area; selecting the intersection of a preset range of the center of the current two-dimensional bounding box and the segmentation mask corresponding to the current two-dimensional bounding box as the valid projection area; calculating all point cloud depth values ​​within the valid projection area, and using a preset interquartile range strategy to remove outliers in all point cloud depth values ​​to obtain the remaining point cloud depth values; selecting the minimum depth value among the remaining point cloud depth values ​​as the depth value of the two-dimensional target corresponding to the current two-dimensional bounding box.

[0071] Specifically, combined Figure 2 As shown, the compensated point cloud data is projected to the two-dimensional bounding box and the corresponding segmentation mask in the image data. If there is point cloud data falling within the two-dimensional bounding box (that is, there is an overlapping area between the compensated point cloud data and the two-dimensional bounding box), the current two-dimensional bounding box can be determined based on the overlapping area. In order to ensure the consistency of the target depth and reduce the impact caused by the camera calibration error, a valid projection area can be further determined in the current two-dimensional bounding box. The area is determined by taking the intersection of a preset range (such as 1 / 3 area) of the center of the current two-dimensional bounding box and the corresponding segmentation mask. Only the point cloud depth that falls within the valid projection area will be used for statistical calculations. In order to further improve the accuracy of depth calculations, the preset interquartile range strategy IQR (Interquartile Range) method can also be used to remove outliers in all calculated point cloud depth values, such as Figure 3 As shown, finally, the minimum depth value among the remaining point cloud depth values ​​is taken as the depth value of the two-dimensional target corresponding to the current two-dimensional bounding box.

[0072] Among them, for the impact caused by the camera calibration error, taking a telephoto camera with a FOV (Field Of View) of 30 degrees as an example, the corresponding depth error when the pixel deviation is 1 pixel under different depth conditions at a long distance can be shown in Table 1:

[0073] Table 1

[0074]

[0075] Furthermore, in some embodiments, after determining whether there is an overlapping area between the compensated point cloud data and the two-dimensional bounding box, it also includes: if there is no overlapping area between the compensated point cloud data and the two-dimensional bounding box, using a laser radar to perform at least one observation to obtain three-dimensional size information of the current target; based on the three-dimensional size information of the current target, using a preset nonlinear optimization algorithm to obtain a two-dimensional target depth value corresponding to the current target.

[0076] That is to say, the compensated point cloud data is projected into the image data. If there is no point cloud data falling within the two-dimensional bounding box (that is, there is no overlapping area between the compensated point cloud data and the two-dimensional bounding box), then a nonlinear optimization method can be used to estimate its depth. This process usually involves complex mathematical models and algorithms. Through iterative optimization, a reasonable depth estimate can be inferred for these targets, even if they do not have direct corresponding points in the original point cloud data. The embodiment of the present application can also provide a complete depth information for all targets in the entire scene, further enhancing the depth and accuracy of scene understanding.

[0077] Specifically, when there is no point cloud data within the two-dimensional bounding box, the laser radar can be used to perform at least one observation operation, so as to effectively obtain the three-dimensional size information of the current target object, which includes key parameters such as the height, width and depth of the target. Next, based on the three-dimensional size information, a pre-set nonlinear optimization algorithm can be used for further processing to obtain the two-dimensional target depth value corresponding to the current target, which can more accurately reflect the actual depth of the target on the two-dimensional plane, and provide more reliable data support for subsequent image processing and target recognition.

[0078] For example, the actual width of the current target detected by the LiDAR in a certain frame of data is w, and the optimization target is as follows:

[0079] Minimize||K*[x,y,z]-z*[u,v,1]||2

[0080] subject to(x1-x2) 2 +(z1-z2) 2 =w2

[0081] y1-y2=0

[0082] Among them, x, y, z are the coordinates of the two three-dimensional points at the bottom of the target vehicle in the camera coordinate system, u, v are the corresponding two-dimensional coordinates in the image coordinate system (known), and z is the target depth in the camera coordinate system.

[0083] According to the method for obtaining the depth value of a two-dimensional target in an image proposed in an embodiment of the present application, by obtaining image data and point cloud data, and performing instance segmentation on the image data using an instance segmentation model, a two-dimensional bounding box and a corresponding segmentation mask of each two-dimensional target in the image data can be obtained; then the point cloud data is compensated, and the compensated point cloud data is projected to the two-dimensional bounding box and the corresponding segmentation mask of each two-dimensional target in the image data, thereby calculating the depth value of at least one two-dimensional target. Thus, by projecting the point cloud to the corresponding two-dimensional target mask area and removing outliers from the point cloud depth falling in the area to obtain an accurate depth, the problem of inconsistent target depth and inability to obtain the depth of distant targets without point clouds in the prior art is solved, thereby improving the safety of autonomous driving.

[0084] Next, a device for acquiring a two-dimensional target depth value of an image proposed in an embodiment of the present application is described with reference to the accompanying drawings.

[0085] Figure 4 It is a block diagram of a device for acquiring a two-dimensional target depth value of an image according to an embodiment of the present application.

[0086] like Figure 4 As shown, the device 10 for acquiring the two-dimensional target depth value of the image includes: an acquisition module 100 , a segmentation module 200 and a calculation module 300 .

[0087] Wherein, the acquisition module 100 is used to acquire image data and point cloud data;

[0088] The segmentation module 200 is used to perform instance segmentation on the image data using the instance segmentation model to obtain a two-dimensional bounding box and a corresponding segmentation mask for each two-dimensional object in the image data;

[0089] The calculation module 300 is used to perform compensation processing on the point cloud data, project the compensated point cloud data to the two-dimensional bounding box of each two-dimensional target in the image data and the corresponding segmentation mask, and calculate the depth value of at least one two-dimensional target.

[0090] Furthermore, in some embodiments, the computing module 300 includes:

[0091] A judging unit, configured to project the compensated point cloud data to a two-dimensional bounding box of each two-dimensional target in the image data and a corresponding segmentation mask, and to judge whether there is an overlapping area between the compensated point cloud data and the two-dimensional bounding box;

[0092] A determination unit, configured to determine a current two-dimensional bounding box based on the overlapping area when there is an overlapping area between the compensated point cloud data and the two-dimensional bounding box;

[0093] A first selection unit is used to select an intersection of a preset range of the center of the current two-dimensional bounding box and a segmentation mask corresponding to the current two-dimensional bounding box as a valid projection area;

[0094] A screening unit is used to calculate all point cloud depth values ​​in the effective projection area, and use a preset interquartile range strategy to remove abnormal values ​​from all point cloud depth values ​​to obtain remaining point cloud depth values;

[0095] The second selection unit is used to select the minimum depth value among the remaining point cloud depth values ​​as the depth value of the two-dimensional target corresponding to the current two-dimensional bounding box.

[0096] Furthermore, in some embodiments, after determining whether there is an overlapping area between the compensated point cloud data and the two-dimensional bounding box, the determination unit is further configured to:

[0097] When there is no overlapping area between the compensated point cloud data and the two-dimensional bounding box, at least one observation is performed using the laser radar to obtain three-dimensional size information of the current target;

[0098] Based on the three-dimensional size information of the current target, a preset nonlinear optimization algorithm is used to obtain the two-dimensional target depth value corresponding to the current target.

[0099] Furthermore, in some embodiments, after obtaining the two-dimensional bounding box and the corresponding segmentation mask of each object in the image data, the segmentation module 200 is further configured to:

[0100] The two-dimensional bounding box and the corresponding segmentation mask are screened to filter out the objects in the image data that meet the preset occlusion conditions.

[0101] Furthermore, in some embodiments, the calculation module 300 is specifically configured to:

[0102] Based on the external parameters of the lidar and using the preset RTK technology, the point cloud data is adjusted to have the same timestamp as the image data to obtain the compensated point cloud data.

[0103] It should be noted that the above explanation of the embodiment of the method for obtaining the two-dimensional object depth value of an image is also applicable to the device for obtaining the two-dimensional object depth value of an image of this embodiment, and will not be repeated here.

[0104] According to the device for acquiring the depth value of a two-dimensional target of an image proposed in an embodiment of the present application, by acquiring image data and point cloud data, and performing instance segmentation on the image data using an instance segmentation model, a two-dimensional bounding box and a corresponding segmentation mask of each two-dimensional target in the image data can be obtained; then the point cloud data is compensated, and the compensated point cloud data is projected to the two-dimensional bounding box and the corresponding segmentation mask of each two-dimensional target in the image data, thereby calculating the depth value of at least one two-dimensional target. Thus, by projecting the point cloud to the corresponding two-dimensional target mask area and removing outliers from the depth of the point cloud falling in the area to obtain an accurate depth, the problem of inconsistent target depth and inability to obtain the depth of distant targets without point clouds in the prior art is solved, thereby improving the safety of autonomous driving.

[0105] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0106] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0107] When the processor 502 executes the program, the method for acquiring the two-dimensional target depth value of the image provided in the above embodiment is implemented.

[0108] Furthermore, the electronic device further comprises:

[0109] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0110] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0111] The memory 501 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0112] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0113] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0114] The processor 502 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0115] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for obtaining a two-dimensional target depth value of an image as described above.

[0116] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0117] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0118] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for obtaining a two-dimensional target depth value of an image, characterized in that: The following steps are involved: Acquire image data and point cloud data; Performing instance segmentation on the image data using an instance segmentation model to obtain a two-dimensional bounding box and a corresponding segmentation mask for each two-dimensional object in the image data; The point cloud data is compensated, the compensated point cloud data is projected to a two-dimensional bounding box of each two-dimensional target in the image data and a corresponding segmentation mask, and a depth value of at least one two-dimensional target is calculated.

2. The method according to claim 1, characterized in that The step of projecting the compensated point cloud data onto a two-dimensional bounding box of each two-dimensional target in the image data and a corresponding segmentation mask, and calculating a depth value of at least one two-dimensional target, comprises: Projecting the compensated point cloud data to a two-dimensional bounding box and a corresponding segmentation mask of each two-dimensional target in the image data, and determining whether there is an overlapping area between the compensated point cloud data and the two-dimensional bounding box; If the compensated point cloud data and the two-dimensional bounding box have the overlapping area, determining the current two-dimensional bounding box based on the overlapping area; Selecting the intersection of the preset range of the center of the current two-dimensional bounding box and the segmentation mask corresponding to the current two-dimensional bounding box as the effective projection area; Calculate all point cloud depth values ​​in the effective projection area, and use a preset interquartile range strategy to remove abnormal values ​​in all point cloud depth values ​​to obtain remaining point cloud depth values; A minimum depth value among the remaining point cloud depth values ​​is selected as the depth value of the two-dimensional object corresponding to the current two-dimensional bounding box.

3. The method according to claim 2, characterized in that After determining whether the compensated point cloud data and the two-dimensional bounding box have the overlapping area, the method further includes: If the compensated point cloud data and the two-dimensional bounding box do not have the overlapping area, using the laser radar to perform at least one observation to obtain the three-dimensional size information of the current target; Based on the three-dimensional size information of the current target, a two-dimensional target depth value corresponding to the current target is obtained using a preset nonlinear optimization algorithm.

4. The method according to claim 1, characterized in that: After obtaining the two-dimensional bounding box of each object in the image data and the corresponding segmentation mask, the method further includes: The two-dimensional bounding box and the corresponding segmentation mask are screened to filter out objects in the image data that meet a preset occlusion condition.

5. The method according to claim 1, characterized in that: The compensating process for the point cloud data includes: Based on the external parameters of the laser radar, the preset RTK technology is used to adjust the point cloud data to meet the same timestamp as the image data to obtain the compensated point cloud data.

6. A device for acquiring a two-dimensional target depth value of an image, characterized in that: include: An acquisition module, used to acquire image data and point cloud data; A segmentation module, used to perform instance segmentation on the image data using an instance segmentation model to obtain a two-dimensional bounding box and a corresponding segmentation mask for each two-dimensional object in the image data; A calculation module is used to perform compensation processing on the point cloud data, project the compensated point cloud data to the two-dimensional bounding box and the corresponding segmentation mask of each two-dimensional target in the image data, and calculate the depth value of at least one two-dimensional target.

7. The device according to claim 6, characterized in that The computing module comprises: a judging unit, configured to project the compensated point cloud data to a two-dimensional bounding box and a corresponding segmentation mask of each two-dimensional target in the image data, and to judge whether there is an overlapping area between the compensated point cloud data and the two-dimensional bounding box; a determining unit, configured to determine a current two-dimensional bounding box based on the overlapping area when the compensated point cloud data and the two-dimensional bounding box have the overlapping area; A first selection unit, configured to select an intersection of a preset range of the center of the current two-dimensional bounding box and a segmentation mask corresponding to the current two-dimensional bounding box as a valid projection area; A screening unit, configured to calculate all point cloud depth values ​​within the effective projection area, and remove abnormal values ​​from all point cloud depth values ​​using a preset interquartile range strategy to obtain remaining point cloud depth values; The second selection unit is used to select a minimum depth value among the remaining point cloud depth values ​​as the depth value of the two-dimensional target corresponding to the current two-dimensional bounding box.

8. The device according to claim 7, characterized in that After determining whether the compensated point cloud data and the two-dimensional bounding box have the overlapping area, the determining unit is further configured to: When the compensated point cloud data and the two-dimensional bounding box do not have the overlapping area, using a laser radar to perform at least one observation to obtain three-dimensional size information of the current target; Based on the three-dimensional size information of the current target, a two-dimensional target depth value corresponding to the current target is obtained using a preset nonlinear optimization algorithm.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for acquiring a two-dimensional target depth value of an image as described in any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method for acquiring a two-dimensional target depth value of an image as described in any one of claims 1 to 5.

Images