A three-dimensional detection method, apparatus, electronic device and storage medium

Abstract

This disclosure provides a 3D detection method, apparatus, electronic device, and storage medium, relating to the field of artificial intelligence, particularly to the fields of intelligent transportation, autonomous driving, and smart parking. The specific implementation scheme includes: performing target detection on an original image to obtain a first parameter set of the target object, the first parameter set including at least one of the following: the target object's category, the target object's coordinates on a first coordinate axis in a 3D coordinate system, the target object's coordinates on a second coordinate axis in a 3D coordinate system, the target object's size, and the target object's angle; performing depth detection on the original image to obtain the depth of each pixel in the original image; determining the depth of the target object using the depth of each pixel in the original image; and combining the first parameter set and the depth of the target object to obtain the 3D detection parameters of the target object. This disclosure can perform 3D object detection.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, particularly to the fields of intelligent transportation, autonomous driving, and smart parking. Background Technology

[0002] 3D inspection, also known as three-dimensional inspection, can reflect the three-dimensional shape of objects in a scene. 3D inspection is an essential technology in fields such as autonomous driving, intelligent transportation, and smart parking. Summary of the Invention

[0003] This disclosure provides a three-dimensional detection method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, a three-dimensional detection method is provided, comprising:

[0005] Target detection is performed on the original image to obtain a first set of parameters of the target object. The first set of parameters includes at least one of the following: the category of the target object, the coordinate value of the target object on the first coordinate axis in the three-dimensional coordinate system, the coordinate value of the target object on the second coordinate axis in the three-dimensional coordinate system, the size of the target object, and the angle of the target object.

[0006] Depth detection is performed on the original image to obtain the depth of each pixel in the original image; the depth of the target object is determined using the depth of each pixel in the original image; and,

[0007] The first set of parameters is combined with the depth of the target object to obtain the three-dimensional detection parameters of the target object.

[0008] According to another aspect of this disclosure, a three-dimensional inspection apparatus is provided, comprising:

[0009] The target detection module is used to perform target detection on the original image and obtain a first set of parameters of the target object. The first set of parameters includes at least one of the following: the category of the target object, the coordinate value of the target object on the first coordinate axis in the three-dimensional coordinate system, the coordinate value of the target object on the second coordinate axis in the three-dimensional coordinate system, the size of the target object, and the angle of the target object.

[0010] A depth detection module is used to perform depth detection on the original image to obtain the depth of each pixel in the original image; using the depth of each pixel in the original image, the depth of the target object is determined; and,

[0011] The combination module is used to combine the first set of parameters and the depth of the target object to obtain the three-dimensional detection parameters of the target object.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0013] At least one processor; and

[0014] The memory is communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods of any embodiment of the present disclosure.

[0016] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method according to any embodiment of this disclosure.

[0017] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method according to any embodiment of this disclosure.

[0018] This disclosure obtains the three-dimensional detection parameters of the target object in the original image by combining the results of target detection and depth detection.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0021] Figure 1 This is a schematic diagram of a system 100 architecture for which the three-dimensional detection method of the present disclosure embodiments can be applied;

[0022] Figure 2 This is a flowchart illustrating the implementation of a three-dimensional detection method 200 according to an embodiment of the present disclosure;

[0023] Figure 3 This is a schematic diagram of a three-dimensional coordinate system according to an embodiment of the present disclosure;

[0024] Figure 4 This is a schematic diagram of a target detection model 400 according to an embodiment of the present disclosure;

[0025] Figure 5 This is a schematic diagram of a depth detection model 500 according to an embodiment of the present disclosure;

[0026] Figure 6This is an overall flowchart of a three-dimensional detection method according to an embodiment of the present disclosure;

[0027] Figure 7A This is a schematic diagram of the first region in one embodiment of the present disclosure. Figure 1 ;

[0028] Figure 7B This is a schematic diagram of the first region in one embodiment of the present disclosure. Figure 2 ;

[0029] Figure 7C This is a schematic diagram of the first region in one embodiment of the present disclosure. Figure 3 ;

[0030] Figure 7D This is a schematic diagram of the first region in one embodiment of the present disclosure. Figure 4 ;

[0031] Figure 8 This is a schematic diagram of the structure of a three-dimensional detection device 800 according to an embodiment of the present disclosure;

[0032] Figure 9 This is a schematic diagram of the structure of a three-dimensional detection device 900 according to an embodiment of the present disclosure;

[0033] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0034] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0035] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0036] In related technologies, 3D inspection has a wide range of applications. 3D inspection is used to detect the three-dimensional shape of objects in a scene. For example, in scenarios such as autonomous driving, intelligent transportation, and smart parking, it is necessary to determine the three-dimensional shape and position of target objects (such as vehicles and pedestrians).

[0037] Currently available 3D detection methods mainly include:

[0038] 1. Point clouds are obtained using lidar, and then 3D target detection is performed using the point cloud data. Specifically, a laser pulse is emitted by a laser, and the emission time is recorded by a timer. The returning laser pulse is received by a receiver, and the return time is recorded by a timer. Subtracting the two times gives the "time of flight" of light. Since the speed of light is constant, the distance can be calculated once the speed and time are known.

[0039] 2. 3D detection using a binocular camera. This approach requires camera calibration to obtain the camera's intrinsic and extrinsic parameters. By calculating the parallax between two images, the distance to objects in front (the area captured by the images) can be directly measured without needing to determine the type of obstacle. The principle of a binocular camera is similar to that of the human eye. The human eye can perceive the distance of objects because the images of the same object presented by the two eyes differ, also known as "parallax." The farther the object is, the smaller the parallax; conversely, the closer the object, the greater the parallax.

[0040] 3. 3D Detection Using a Monocular Camera and Deep Learning Algorithms. A monocular camera is a common everyday camera with only one camera lens. In principle, a monocular camera captures two-dimensional (2D) images and cannot provide a 3D viewpoint. However, by using supervised deep learning algorithms, images captured by the monocular camera are used as training data, and the 3D detection parameters of the target object (such as 3D bounding box information) are used as annotation information. By training the deep learning model using the training data and annotation information, monocular 3D detection can be achieved in specific scenarios. When the model training process is successful, theoretically, the monocular 3D detection results can be very close to the results of LiDAR detection.

[0041] Each of the above methods has its drawbacks: In the first method, LiDAR is expensive, consumes a lot of power, and has a high failure rate. In the second method, a large amount of parameter calibration work is required for the binocular cameras, thus requiring a large number of maintenance personnel. In the third method, the deep learning model used to achieve 3D detection needs to be trained, the training process is complex, requires manually standardized training samples, and the operation of the 3D detection model also requires high time and power consumption.

[0042] This disclosure presents a three-dimensional detection method. Figure 1 This is a schematic diagram of a system 100 architecture for which the three-dimensional detection method of the present disclosure embodiments can be applied. For example... Figure 1As shown, the system architecture includes an image acquisition device 110, a network 120, and a 3D detection device 130. The image acquisition device 110 and the 3D detection device 130 can establish a communication connection via the network 120. The image acquisition device 110 sends a raw image to the 3D detection device 130 via the network 120. In response to the received raw image, the 3D detection device 130 performs 3D detection on the raw image. Finally, the 3D detection device 130 returns the 3D detection result to the image acquisition device or sends it to other servers or terminal devices. The 3D detection device 130 may include a vision processing device or a remote server. The network 120 can use a wired or wireless connection. Specifically, when the 3D detection device 130 is a vision processing device, the image acquisition device 110 can communicate with the vision processing device via a wired connection, such as through a bus for data communication; when the 3D detection device 130 is a remote server, the image acquisition device 110 can interact with the remote server via a wireless network. Additionally, the image acquisition device 110 can be a vehicle-mounted camera, an intelligent traffic camera, etc.

[0043] Figure 2 This is a flowchart illustrating the implementation of a three-dimensional detection method 200 according to an embodiment of this disclosure. In some embodiments of this disclosure, the three-dimensional detection method can be executed by a terminal device, server, or other processing device. In some embodiments of this disclosure, the three-dimensional detection method can be implemented by a processor calling computer-readable instructions stored in memory. Figure 2 As shown, the three-dimensional detection method includes the following steps:

[0044] S210: Perform target detection on the original image to obtain a first set of parameters of the target object. The first set of parameters includes at least one of the following: the category of the target object, the coordinate value of the target object on the first coordinate axis in the three-dimensional coordinate system, the coordinate value of the target object on the second coordinate axis in the three-dimensional coordinate system, the size of the target object, and the angle of the target object.

[0045] S220: Perform depth detection on the original image to obtain the depth of each pixel in the original image; use the depth of each pixel in the original image to determine the depth of the target object; and,

[0046] S230: Combine the first set of parameters with the depth of the target object to obtain the three-dimensional detection parameters of the target object.

[0047] Steps S210 and S220 are executed separately, and there is no restriction on their order. For example, steps S210 and S220 can be executed simultaneously, or step S210 can be executed first and then step S220, or step S220 can be executed first and then step S210; or step S210 or step S220 can be executed at any time, and so on. It is only necessary to ensure that steps S210 and S220 are executed before step S230.

[0048] This disclosure adopts a combination of target detection and depth detection to achieve 3D detection of objects; dividing 3D detection into two independent and simple detection processes can reduce the difficulty of 3D detection and reduce the consumption of time and computing resources for 3D detection.

[0049] This disclosure can be applied to various scenarios, such as parking scenarios. In parking scenarios, since vehicles are generally stationary and usually located in fixed parking spaces, compared to autonomous driving or driving scenarios, this scenario does not require precise 3D detection of the target object (such as a vehicle), and the accuracy requirements for 3D detection are not high. Furthermore, in parking scenarios, the position and angle of the image acquisition device (such as a camera) in the parking lot are fixed. Therefore, only one parameter adjustment of the camera is needed. Subsequent 3D detection can use the camera parameters as a constant for calculation, without needing to re-input calibration parameters before each 3D detection. Therefore, the detection process is simple and stable. Additionally, image acquisition devices in parking scenarios are generally positioned at a higher level, making it easier to determine the depth of the target object from the acquired raw images (the reasons will be explained in detail in subsequent embodiments).

[0050] It should be noted that the above-mentioned parking scenarios are broad terms, including parking lots, temporary parking areas along roadsides, vehicles parked inside exhibition halls, and vehicles that are nearly stationary near traffic lights. Furthermore, the application scenarios of this disclosure are not limited to parking scenarios. As long as the accuracy of the 3D detection using the method proposed in this disclosure meets the requirements of the scenario, it can be applied to that scenario. For example, the embodiments of this disclosure can also be applied to scenarios such as warehouses and docks for 3D detection of goods, and so on.

[0051] In some implementations, a pre-trained object detection model can be used to detect objects in the original image. For example, the original image is input into the pre-trained object detection model to obtain a first set of parameters of the target object output by the model. The first set of parameters includes at least one of the following:

[0052] (1) The category of the target object (denoted as cls). For example, the object detection model can output the confidence scores of multiple categories, and the category with the highest confidence score is the category of the target object predicted by the object detection model.

[0053] (2) The coordinates of the target object on the first coordinate axis in the three-dimensional coordinate system and the coordinates of the target object on the second coordinate axis in the three-dimensional coordinate system.

[0054] Figure 3 This is a schematic diagram of a three-dimensional coordinate system according to an embodiment of the present disclosure. Figure 3 As shown, in some examples, the position of the image acquisition device lens can be taken as the origin, with the direction perpendicular to the lens as the first coordinate axis (denoted as the X-axis), the direction perpendicular to the ground plane as the second coordinate axis (denoted as the Y-axis), and the direction parallel to the lens as the third coordinate axis (denoted as the Z-axis). The target detection model can detect the two coordinate values ​​of the target object, denoted as (x, y).

[0055] (3) Dimensions of the target object. For example, the length (denoted as L), width (denoted as W), and height (denoted as H) of the target object. The dimensions of the target object are denoted as (W, H, L).

[0056] (4) The angle of the target object. For example, the yaw angle of the target object, denoted as θ.

[0057] Figure 4 This is a schematic diagram of a target detection model 400 according to an embodiment of the present disclosure. Figure 4 As shown, the target detection model 400 includes a backbone network 410 and multiple branch networks 420. The branch networks 420 may include a branch network 421 corresponding to the target object's category, a branch network 422 corresponding to partial coordinates of the target object (i.e., the coordinates of the target object on the first coordinate axis and the second coordinate axis in the three-dimensional coordinate system), a branch network 423 corresponding to the target object's size, and a branch network 424 corresponding to the target object's angle. The branch networks 420 can use the image features extracted from the original image by the backbone network 410 to predict the first parameter set corresponding to the target object. The original image input to the backbone network 410 can be an image captured by a monocular camera, referred to as a monocular image.

[0058] Analyzing the first parameter set reveals that it contains most of the information required for 3D detection. Combined with the depth of the target object, the 3D detection data can be determined. The target detection model proposed in this embodiment makes minor modifications to a standard 2D detection model; it retains the backbone network used to acquire image feature data and adds several branches after the backbone network to determine the parameters in the first parameter set. Compared to 3D detection models in related technologies, this reduces model complexity, decreases time and computational cost, and simplifies the model training process.

[0059] In some examples, embodiments of this disclosure employ a pre-trained depth detection model to determine the depth of each pixel in the original image. For instance, by inputting the original image into the pre-trained depth detection model, the depth of each pixel in the original image output by the depth detection model can be obtained. The depth of the target object is then determined using the depth of each pixel in the original image. The depth of the target object can be considered as the coordinate value (z) of the target object on the third coordinate axis (Z-axis) in the three-dimensional coordinate system. This value is then combined with the coordinate values ​​(x, y) in the aforementioned first parameter set to obtain the position (x, y, z) of the target object in the three-dimensional coordinate system.

[0060] That is, the coordinates of the target object on the first coordinate axis in the three-dimensional coordinate system, the coordinates of the target object on the second coordinate axis in the three-dimensional coordinate system, and the depth of the target object are combined to obtain the position of the target object in three-dimensional space.

[0061] The three-dimensional detection parameters of the target object are obtained by combining at least one of the following: the target object's position in three-dimensional space, the target object's category, the target object's size, and the target object's angle.

[0062] This embodiment divides 3D detection into two independent and simplified detection processes. A target detection process is used to obtain partial 3D data of the target object (such as a first parameter set), and depth detection is used to obtain another portion of the target object's 3D data (such as the target object's depth). The target object's depth is then combined with the first parameter set to obtain the target object's 3D detection parameters. This approach reduces the difficulty of 3D detection, decreases the consumption of time and computing power, and simplifies the training process of related models.

[0063] Figure 5 This is a schematic diagram of a depth detection model 500 according to an embodiment of the present disclosure. Figure 5 As shown, the depth detection model 500 includes a backbone network 510 and a depth detection network 520; the backbone network 510 extracts image features from the original image (such as a monocular image), and the depth detection network 520 uses these image features to predict the depth of each pixel in the original image.

[0064] Figure 6 This is an overall flowchart of a three-dimensional detection method according to an embodiment of the present disclosure. Figure 6 As shown, the original image is input into both the object detection model 400 and the depth detection model 500. The object detection model 400 outputs a first set of parameters for the target object, and the depth detection model 500 outputs the depth of each pixel in the original image. The depth of the target object is determined using the depth of each pixel in the original image. The depth of the target object is combined with the first set of parameters to obtain the three-dimensional detection parameters of the target object. The three-dimensional detection parameters of the target object include at least one of the following:

[0065] (1) The category of the target object (denoted as cls).

[0066] (2) The position (x, y, z) of the target object in the three-dimensional coordinate system.

[0067] (3) Dimensions of the target object (W, H, L).

[0068] (4) θ of the target object.

[0069] Among them, (1), (3), and (4) are determined by the target detection model 400, and (2) is composed of the depth of the target object and the coordinates (x, y) determined by the target detection model 400.

[0070] This disclosure allows the use of the YOLO (You Only Look Once) model as the object detection model, and the use of a monocular depth estimation model for depth detection. This disclosure does not impose any limitations on the structure of the model.

[0071] The following describes how to determine the depth of a target object based on the depth of each pixel in the original image.

[0072] The depth of a target object can be determined in at least three ways according to the embodiments of this disclosure:

[0073] The first method is to determine the depth of the target object by using the depth of each element in the target detection bounding box.

[0074] For example, determining the target detection box of the target object in the original image;

[0075] The depth of each pixel in the target detection box is determined using the depth of each pixel in the original image and the target detection box.

[0076] Calculate the average depth of all pixels in the target detection bounding box, and use this average value as the depth of the target object.

[0077] An object detection bounding box is a rectangular box that defines the target object in the original image. Most pixels in the bounding box are pixels of the target object, so the average depth of all pixels in the bounding box can roughly represent the depth of the target object. Of course, since the bounding box contains pixels of other objects besides the target object, the depth of the target object determined in this way is not very precise; however, because it is relatively easy to determine the bounding box, this method has the advantage of fast computation speed.

[0078] The second method involves determining the depth of the target object by using the depth of each pixel within the area defined by the object's boundary.

[0079] For example, determining the boundaries of target objects in the original image;

[0080] Using the depth of each pixel in the original image and the boundary of the target object, determine the depth of each pixel within the boundary limit;

[0081] Calculate the average depth of all pixels within the boundary range, and use this average value as the depth of the target object.

[0082] Since the region defined by the boundary of the target object contains all pixels of the target object and does not contain pixels of any other objects, determining the depth value of the target object using the depth values ​​of each pixel within the region defined by the target object's boundary is obviously accurate. Embodiments of this disclosure can use a mask image method to determine the target object. Besides the mask image method, other methods can determine the outline of the target object, such as instance segmentation.

[0083] The third method involves determining the first region within the target detection box and using the depth of each pixel in the first region to determine the depth of the target object.

[0084] Since the overall depth of a target object is determined using the depth of multiple pixels, the accuracy requirement for the depth of a single pixel is not high, allowing for the confirmation of the target object's depth at a lower cost and with lower power consumption. For example, when the accuracy of a single pixel's depth is not high, in the depth detection results, some pixels may have a depth higher than the actual depth, and some pixels may have a depth lower than the actual depth, with the probability and / or degree of deviation exhibiting a random distribution. However, if the average value of multiple pixels is used as the depth of the target object, according to mathematical principles, many of the higher or lower pixel deviations will be canceled out when calculating the average, ensuring the accuracy of the final calculated average depth (i.e., the depth of the target object). Therefore, in this embodiment, the depth of the target object can be determined using the depth of each pixel in the first region within the target detection box. It should be noted that this embodiment can also employ other methods to determine the depth of the target object using the depth of each pixel in the first region. For example, the median of all pixels in the first region can be used as the depth of the target object.

[0085] Based on the above analysis, the first region for determining the depth of a target object proposed in this embodiment of the present disclosure has the following characteristics:

[0086] 1. Define the first region within the target detection box; that is, the first region is inside the target detection box. This is because all pixels of the target object are within the target detection box.

[0087] 2. The center point of the first region coincides with the center point of the target detection box. This is because the target object is located in the middle of the target detection box. By aligning the center point of the first region with the center point of the target detection box, it can be ensured that the target object is also located in the middle of the first region, thus ensuring that most of the pixels in the first region are pixels of the target object.

[0088] 3. The ratio of the area of ​​the first region to the area of ​​the target detection box is greater than or equal to a preset threshold. This is to ensure that the first region can contain most of the pixels of the target object. This preset threshold can be set according to the actual situation, for example, to 50%.

[0089] With the above characteristics, it can be ensured that the depth of the target object can be accurately determined by using the depth of each pixel in the first region. Furthermore, since the first region has a fixed shape and is located at a fixed position within the target detection box, the pixels contained within the first region can be easily determined. Therefore, the method proposed in this embodiment can both ensure accurate estimation of the target object's depth and reduce time and computational costs, thereby increasing speed.

[0090] In this embodiment of the disclosure, the method for determining the depth of the target object using the depth of each pixel in the original image may include:

[0091] Determine the target detection box for the target object in the original image;

[0092] A first region is defined, wherein the center point of the first region coincides with the center point of the target detection box, and the ratio of the area of ​​the first region to the area of ​​the target detection box of the target object is greater than or equal to a preset threshold.

[0093] Using the depth of each pixel in the original image, the target detection box, and the first region, the depth of each pixel in the first region is determined;

[0094] The depth of the target object is determined by using the depth of all pixels in the first region.

[0095] For example, the average depth of all pixels in the first region can be calculated and used as the depth of the target object. Alternatively, the median depth of all pixels in the first region can be used as the depth of the target object; and so on.

[0096] After determining the aforementioned characteristics of the first region, we will then analyze which regions are suitable as the first region.

[0097] Figures 7A-7D This is a schematic diagram of the first region in one embodiment of this disclosure. It should be noted that... Figures 7A-7D The image shown is the object detection box within the original image, not the original image itself.

[0098] like Figure 7A As shown, the target detection box is rectangular in shape;

[0099] The first region can be rhomboid or square in shape, and its four vertices are located at the midpoints of the four sides of the target detection box. The ratio of the area of ​​the first region to the area of ​​the target detection box is 50%.

[0100] by Figure 7A For example, when the target detection bounding box is rectangular, the shape of the first region is rhomboid. When the target detection bounding box is square, the shape of the first region is square.

[0101] Depend on Figure 7A As can be seen, most pixels within the first region belong to the target object (e.g., ...). Figure 7A (The vehicle in the image), and most of the pixels in the target object are in the first region. According to experimental statistics, in the first region shown in 7A, the pixels of the target object account for 83% of all pixels. Therefore, the pixels in the first region can largely reflect the depth of the target object.

[0102] like Figure 7B As shown, the target detection box is rectangular in shape;

[0103] The shape of the first region can be circular or elliptical.

[0104] by Figure 7B For example, when the target detection box is rectangular, the first region is elliptical, and its four vertices are located at the midpoints of the four sides of the target detection box. When the target detection box is square, all four sides of the target detection box are tangents to the first region. Figure 7B In the example, the ratio of the area of ​​the first region to the area of ​​the target detection box is approximately 80%.

[0105] Depend on Figure 7B As can be seen, most pixels in the first region belong to the target object (e.g., Figure 7B (The vehicle), and most pixels of the target object are within the first region. Experimental statistics show that... Figure 7B In the first region shown, the pixels of the target object account for 79% of all pixels. Therefore, the pixels in the first region can largely reflect the depth of the target object.

[0106] like Figure 7C As shown, the target detection box is rectangular in shape;

[0107] The first region can be polygonal in shape, and each vertex of the first region is located on the edge of the target detection box.

[0108] by Figure 7C For example, when the target detection box is rectangular, the shape of the first region is a regular hexagon. Figure 7C In the example, the ratio of the area of ​​the first region to the area of ​​the target detection box is approximately 75%.

[0109] Depend on Figure 7C As can be seen, most pixels in the first region belong to the target object (e.g., Figure 7C (The vehicle), and most pixels of the target object are within the first region. Experimental statistics show that... Figure 7C In the first region shown, the pixels of the target object account for 76% of all pixels. Therefore, the pixels in the first region can largely reflect the depth of the target object.

[0110] like Figure 7D As shown, the target detection box is rectangular in shape;

[0111] The first region can be an irregular shape, and it contains the midpoints of each side of the target detection box.

[0112] by Figure 7D For example, when the target detection bounding box is rectangular, the shape of the first region is cross-shaped. Figure 7DAs shown, the four sides of the target detection box coincide with four of the twelve sides of the cross shape, and the width of each of these four sides is one-third the length of the corresponding side of the target detection box. Figure 7D In the example, the ratio of the area of ​​the first region to the area of ​​the target detection box is 5 / 9.

[0113] Depend on Figure 7D As can be seen, most pixels in the first region belong to the target object (e.g., Figure 7D (The vehicle), and most pixels of the target object are within the first region. Experimental statistics show that... Figure 7D In the first region shown, the pixels of the target object account for 77% of all pixels. Therefore, the pixels in the first region can largely reflect the depth of the target object.

[0114] This disclosure is particularly applicable to high-position camera scenarios. In such scenarios, the image acquisition device is higher than the target object (e.g., a vehicle). Taking a high-position parking scenario as an example... Figures 7A-7D The images show that most vehicles are oriented at a certain angle to the left or right, and include various areas such as the roof, front, and body. Using the pixel depth of these locations allows for a more accurate determination of the overall vehicle depth.

[0115] In this embodiment, the first region determined based on the center point of the target detection box can cover all or most of the target object. Determining the depth of the target object using the depth of each pixel within the first region can reflect the true depth of the target object to the greatest extent possible with relatively low computational power and high speed. The process of determining the depth of the target object using the depth of each pixel within the first region proposed in this embodiment can also eliminate the influence of a large amount of background.

[0116] This disclosure also proposes a three-dimensional detection device. Figure 8 This is a schematic diagram of the structure of a three-dimensional detection device 800 according to an embodiment of the present disclosure, including:

[0117] The target detection module 810 is used to perform target detection on the original image and obtain a first parameter set of the target object. The first parameter set includes at least one of the following: the category of the target object, the coordinate value of the target object on the first coordinate axis in the three-dimensional coordinate system, the coordinate value of the target object on the second coordinate axis in the three-dimensional coordinate system, the size of the target object, and the angle of the target object.

[0118] The depth detection module 820 is used to perform depth detection on the original image to obtain the depth of each pixel in the original image; and to determine the depth of the target object using the depth of each pixel in the original image; and,

[0119] The combination module 830 is used to combine the first parameter set and the depth of the target object to obtain the three-dimensional detection parameters of the target object.

[0120] Figure 9 This is a schematic diagram of the structure of a three-dimensional detection device 900 according to an embodiment of the present disclosure, as shown below. Figure 9 As shown, the three-dimensional detection device 900 includes a target detection module 910, a depth detection module 920, and a combination module 930. In some embodiments, the combination module 930 includes:

[0121] The first combination submodule 931 is used to combine the coordinate values ​​of the target object on the first coordinate axis in the three-dimensional coordinate system, the coordinate values ​​of the target object on the second coordinate axis in the three-dimensional coordinate system, and the depth of the target object to obtain the position of the target object in three-dimensional space.

[0122] The second combination submodule 932 is used to combine at least one of the target object's position in three-dimensional space, the target object's category, the target object's size, and the target object's angle to obtain the target object's three-dimensional detection parameters.

[0123] In some implementations, the depth of the target object corresponds to the coordinate value of the target object on the third coordinate axis in a three-dimensional coordinate system.

[0124] In some implementations, the depth detection module 920 is used for:

[0125] Determine the target detection box of the target object in the original image;

[0126] Using the depth of each pixel in the original image and the target detection box, the depth of each pixel in the target detection box is determined;

[0127] Calculate the average depth of all pixels in the target detection box, and use this average value as the depth of the target object.

[0128] In some implementations, the depth detection module 920 is used for:

[0129] Determine the boundary of the target object in the original image;

[0130] Using the depth of each pixel in the original image and the boundary of the target object, the depth of each pixel within the boundary is determined;

[0131] Calculate the average depth of all pixels within the boundary range, and use this average value as the depth of the target object.

[0132] In some implementations, the depth detection module 920 includes:

[0133] The first region determination submodule 921 is used to determine the target detection box of the target object in the original image and determine the first region, wherein the center point of the first region coincides with the center point of the target detection box, and the ratio of the area of ​​the first region to the area of ​​the target detection box of the target object is greater than or equal to a preset threshold.

[0134] The first region depth determination submodule 922 uses the depth of each pixel in the original image, the target detection box, and the first region to determine the depth of each pixel in the first region.

[0135] The target object depth determination submodule 923 determines the depth of the target object using the depth of all pixels in the first region.

[0136] In some implementations, the target object depth determination submodule 923 is used to calculate the average depth of all pixels in the first region and use the average value as the depth of the target object.

[0137] In some implementations, the target detection box is rectangular in shape;

[0138] The first region is rhomboid or square in shape, and the four vertices of the first region are located at the midpoints of the four sides of the target detection box.

[0139] In some implementations, the target detection box is rectangular in shape;

[0140] The first region is elliptical in shape, and its four vertices are located at the midpoints of the four sides of the target detection box.

[0141] In some implementations, the target detection box is square in shape;

[0142] The first region is circular in shape, and all four sides of the target detection box are tangents to the first region.

[0143] In some implementations, target detection is performed on the original image to obtain a first set of parameters for the target object, including:

[0144] The original image is input into a pre-trained object detection model to obtain the first set of parameters of the target object output by the object detection model.

[0145] In some implementations, depth detection is performed on the original image to obtain the depth of each pixel in the original image, including:

[0146] The original image is input into a pre-trained depth detection model to obtain the depth of each pixel in the original image as output by the depth detection model.

[0147] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0148] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0149] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0150] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0151] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0152] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0153] The computing unit 1001 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the three-dimensional detection method. For example, in some embodiments, the three-dimensional detection method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the three-dimensional detection method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the three-dimensional detection method by any other suitable means (e.g., by means of firmware).

[0154] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0156] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0157] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0158] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0159] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0160] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0161] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A three-dimensional detection method, comprising: inputting an original image into a pre-trained target detection model to obtain a first parameter set of a target object, the first parameter set comprising a category of the target object, a coordinate value of the target object on a first coordinate axis in a three-dimensional coordinate system, a coordinate value of the target object on a second coordinate axis in the three-dimensional coordinate system, a size of the target object, and an angle of the target object; the target detection model comprising a backbone network, a branch network corresponding to the category of the target object, a branch network corresponding to the coordinate value of the target object on the first coordinate axis and the coordinate value of the target object on the second coordinate axis in the three-dimensional coordinate system, a branch network corresponding to the size of the target object, and a branch network corresponding to the angle of the target object; inputting the original image into a pre-trained depth detection model to obtain a depth of each pixel in the original image output by the depth detection model; wherein the depth detection model comprises a backbone network and a depth detection network, the backbone network is used to extract image features of the original image, and the depth detection network is used to predict the depth of each pixel in the original image by using the image features; the depth of the target object is determined by using the depth of each pixel in the original image; the original image comprises an image captured by a monocular camera; and combining the first parameter set and the depth of the target object to obtain a three-dimensional detection parameter of the target object.

2. The method of claim 1, wherein, The combining the first parameter set and the depth of the target object to obtain the three-dimensional detection parameter of the target object comprises: combining the coordinate value of the target object on the first coordinate axis in the three-dimensional coordinate system, the coordinate value of the target object on the second coordinate axis in the three-dimensional coordinate system, and the depth of the target object to obtain a position of the target object in a three-dimensional space; combining at least one of the position of the target object in the three-dimensional space, the category of the target object, the size of the target object, and the angle of the target object to obtain the three-dimensional detection parameter of the target object.

3. The method of claim 1, wherein, The depth of the target object corresponds to a coordinate value of the target object on a third coordinate axis in the three-dimensional coordinate system.

4. The method of any one of claims 1-3, wherein, The determining the depth of the target object by using the depth of each pixel in the original image comprises: determining a target detection frame of the target object in the original image; determining the depth of each pixel in the target detection frame by using the depth of each pixel in the original image and the target detection frame; calculating an average value of the depths of all pixels in the target detection frame, and taking the average value as the depth of the target object.

5. The method of any one of claims 1-3, wherein, The determining the depth of the target object by using the depth of each pixel in the original image comprises: determining a boundary of the target object in the original image; determining the depth of each pixel within the boundary by using the depth of each pixel in the original image and the boundary of the target object; calculating an average value of the depths of all pixels within the boundary, and taking the average value as the depth of the target object.

6. The method of any one of claims 1-3, wherein, The determining the depth of the target object by using the depth of each pixel in the original image comprises: determining a target detection frame of the target object in the original image, and determining a first region, wherein a center point of the first region coincides with a center point of the target detection frame, and a ratio of an area of the first region to an area of the target detection frame of the target object is greater than or equal to a preset threshold value; determining the depth of each pixel in the first region by using the depth of each pixel in the original image, the target detection frame and the first region; determining the depth of the target object by using the depth of all pixels in the first region.

7. The method of claim 6, wherein, The determining the depth of the target object by using the depth of all pixels in the first region comprises: calculating an average value of the depth of all pixels in the first region, and taking the average value as the depth of the target object.

8. The method of claim 6, wherein, The shape of the target detection frame is a rectangle; the shape of the first region is a rhombus or a square, and four vertices of the first region are respectively located at midpoints of four sides of the target detection frame.

9. The method of claim 6, wherein, The shape of the target detection frame is a rectangle; the shape of the first region is an ellipse, and four vertices of the first region are respectively located at midpoints of four sides of the target detection frame.

10. The method of claim 6, wherein, The shape of the target detection frame is a square; the shape of the first region is a circle, and four sides of the target detection frame are all tangent lines of the first region.

11. A three-dimensional detection device, comprising: a target detection module, which inputs an original image into a pre-trained target detection model to obtain a first parameter set of a target object, wherein the first parameter set comprises a category of the target object, a coordinate value of the target object on a first coordinate axis in a three-dimensional coordinate system, a coordinate value of the target object on a second coordinate axis in the three-dimensional coordinate system, a size of the target object and an angle of the target object; the target detection model comprises a backbone network, a branch network corresponding to the category of the target object, a branch network corresponding to the coordinate value of the target object on the first coordinate axis and the coordinate value of the target object on the second coordinate axis in the three-dimensional coordinate system, a branch network corresponding to the size of the target object and a branch network corresponding to the angle of the target object; a depth detection module, which is configured to input the original image into a pre-trained depth detection model to obtain a depth of each pixel in the original image output by the depth detection model; wherein the depth detection model comprises a backbone network and a depth detection network, the backbone network is configured to extract image features of the original image, and the depth detection network is configured to predict the depth of each pixel in the original image by using the image features; the depth of the target object is determined by using the depth of each pixel in the original image; the original image comprises an image captured by a monocular camera; and a combination module, which is configured to combine the first parameter set and the depth of the target object to obtain a three-dimensional detection parameter of the target object.

12. The apparatus of claim 11, wherein, The combination module comprises: The first combination sub-module is configured to combine a coordinate value of the target object on a first coordinate axis in a three-dimensional coordinate system, a coordinate value of the target object on a second coordinate axis in the three-dimensional coordinate system, and a depth of the target object, to obtain a position of the target object in a three-dimensional space. The second combination sub-module is configured to combine at least one of the position of the target object in the three-dimensional space, a category of the target object, a size of the target object, and an angle of the target object, to obtain a three-dimensional detection parameter of the target object.

13. The apparatus of claim 11, wherein, The depth of the target object corresponds to a coordinate value of the target object on a third coordinate axis in the three-dimensional coordinate system.

14. The apparatus of any one of claims 11-13, wherein, The depth detection module is configured to: determine a target detection frame of the target object in the original image; determine, by using the depth of each pixel in the original image and the target detection frame, the depth of each pixel in the target detection frame; calculate an average value of the depths of all the pixels in the target detection frame, and take the average value as the depth of the target object.

15. The apparatus of any one of claims 11-13, wherein, The depth detection module is configured to: determine a boundary of the target object in the original image; determine, by using the depth of each pixel in the original image and the boundary of the target object, the depth of each pixel within the boundary defined range; calculate an average value of the depths of all the pixels within the boundary defined range, and take the average value as the depth of the target object.

16. The apparatus of any one of claims 11-13, wherein, The depth detection module comprises: a first region determination sub-module configured to determine a target detection frame of the target object in the original image, and determine a first region, wherein a center point of the first region coincides with a center point of the target detection frame, and a ratio of an area of the first region to an area of the target detection frame is greater than or equal to a preset threshold value; a first region depth determination sub-module configured to determine, by using the depth of each pixel in the original image, the target detection frame, and the first region, the depth of each pixel in the first region; a target object depth determination sub-module configured to determine, by using the depths of all the pixels in the first region, the depth of the target object.

17. The apparatus of claim 16, wherein, The target object depth determination sub-module is configured to calculate an average value of the depths of all the pixels in the first region, and take the average value as the depth of the target object. 18.An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-10.

19. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are configured to enable the computer to perform the method of any one of claims 1-10. 20.A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-10.

Images