Single-camera-based size estimation method, device, equipment and storage medium

Abstract

The application relates to a single-camera-based size estimation method, device, equipment and storage medium. The size estimation method comprises the following steps: detecting an image comprising a pseudo-three-dimensional bounding box of an obstacle and a two-dimensional bounding box to obtain two end points of a bottom side of the pseudo-three-dimensional bounding box, and determining a base point according to the two end points; constructing a first ray and a second ray in a world coordinate system based on the base point; projecting the first ray and the second ray onto the image, so that the projections of the first ray and the second ray intersect with a top side and a side side of the two-dimensional bounding box at a first intersection point and a second intersection point respectively; and determining size information of the obstacle according to the two end points, the first intersection point and the second intersection point. The size estimation method provided by the application can improve the low size estimation efficiency in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a size estimation method and device based on a single camera, equipment and storage medium. BACKGROUND

[0002] Obstacle perception is an important branch of automatic driving technology. An automatic driving vehicle needs to identify and detect obstacles such as vehicles in the surrounding environment to achieve a more intelligent and safe automatic driving process. When detecting obstacles, if the size information of the obstacles can be detected, it can help the automatic driving system to make more reasonable and effective path planning.

[0003] At present, in order to estimate the size of the obstacle, the radar perception technology is used in the prior art to label the bounding box of the obstacle, and the size information of the obstacle is determined based on the bounding box. Specifically, the point cloud data of the obstacle is first obtained through the laser radar, and then a point cloud cuboid is constructed using the point cloud data. Since the point cloud cuboid is the real three-dimensional bounding box of the obstacle, the length, width and height of the obstacle can be obtained by measuring the point cloud cuboid.

[0004] However, the radar perception method not only has high cost, but also has the problem of low size estimation efficiency due to the large amount of data and complex calculation of the point cloud data. SUMMARY

[0005] Therefore, the present application provides a size estimation method and device based on a single camera, equipment and storage medium to improve the problem of low size estimation efficiency in the prior art.

[0006] In a first aspect, the present application provides a size estimation method based on a single camera, which comprises: detecting an image including a pseudo-three-dimensional bounding box and a two-dimensional bounding box of an obstacle to obtain two end points of the bottom edge of the pseudo-three-dimensional bounding box, and determining a base point according to the two end points; constructing a first ray and a second ray in a world coordinate system based on the base point, wherein the first ray and the second ray are respectively used to point to the height and width directions of the obstacle; projecting the first ray and the second ray onto the image, so that the projections of the first ray and the second ray intersect with the top edge and the side edge of the two-dimensional bounding box at a first intersection point and a second intersection point; and determining the size information of the obstacle according to the two end points, the first intersection point and the second intersection point, wherein the size information includes at least one of the length, width, height and center point.

[0007] In conjunction with the first aspect, in a first possible implementation of the first aspect, the step of detecting an image including a pseudo-3D bounding box and a 2D bounding box containing obstacles to obtain the two endpoints of the bottom edge of the pseudo-3D bounding box and determining a base point based on the two endpoints includes: detecting the side bottom edge, front bottom edge, or back bottom edge of the 3D bounding box on the image; if the side bottom edge of the pseudo-3D bounding box is detected, obtaining the two endpoints of the side bottom edge of the pseudo-3D bounding box and using one of the two endpoints as a base point; if the side bottom edge of the pseudo-3D bounding box is not detected, obtaining the two endpoints of the front bottom edge or back bottom edge of the pseudo-3D bounding box and using the midpoint between the two endpoints as a base point.

[0008] In conjunction with the first possible implementation of the first aspect, in the second possible implementation of the first aspect, the steps of constructing the first ray and the second ray in the world coordinate system based on the base point include: when the side bottom edge of the pseudo-3D bounding box is detected, connecting the two endpoints to determine the heading of the obstacle; when the side bottom edge of the pseudo-3D bounding box is not detected, taking the direction of the camera's line of sight or the direction of the lane line of the lane where the obstacle is located as the heading of the obstacle; obtaining the world coordinates of the base point based on the pixel coordinates of the base point on the image; and constructing the first ray and the second ray starting from the world coordinates of the base point, wherein the first ray is perpendicular to the ground where the obstacle is located and points to the top of the obstacle, and the second ray is perpendicular to the heading and the first ray and points to the side of the obstacle.

[0009] In conjunction with the first possible implementation of the first aspect, in the third possible implementation of the first aspect, the step of determining the size information of the obstacle based on the two endpoints, the first intersection point, and the second intersection point includes: when the side bottom edge of the pseudo-3D bounding box is detected, converting the pixel coordinates of the two endpoints, the first intersection point, and the second intersection point to obtain the world coordinates of the two endpoints, the first intersection point, and the second intersection point respectively, and determining the length, width, height, and / or center point of the obstacle based on the world coordinates of the two endpoints, the first intersection point, and the second intersection point; when the side bottom edge of the pseudo-3D bounding box is not detected, converting the pixel coordinates of the midpoint of the two endpoints, the first intersection point, and the second intersection point to obtain the world coordinates of the midpoint of the two endpoints, the first intersection point, and the second intersection point respectively, and determining the width and / or height of the obstacle based on the world coordinates of the midpoint of the two endpoints, the first intersection point, and the second intersection point.

[0010] In conjunction with the first aspect, in the fourth possible implementation of the first aspect, the step of determining the size information of the obstacle based on the two endpoints, the first intersection point, and the second intersection point includes: determining the height of the obstacle based on the pixel coordinates of the base point, the world coordinates of the base point, the pixel coordinates of the first intersection point, and the unit vector of the first ray, wherein the mathematical expression of the step of determining the height of the obstacle based on the pixel coordinates of the base point, the world coordinates of the base point, the pixel coordinates of the first intersection point, and the unit vector of the first ray is as follows:

[0011] or,

[0012] Where L is the height of the obstacle, (r1,c1) are the pixel coordinates of the base point, (x1,y1,z1) are the world coordinates of the base point, (r3,c3) are the pixel coordinates of the first intersection point, and (u x ,u y ,u z Let f be the unit vector of the first ray. x and f y This refers to the camera's intrinsic focal length parameter.

[0013] In conjunction with the first possible implementation of the first aspect, in the fifth possible implementation of the first aspect, after determining the size information of the obstacle based on the two endpoints, the first intersection point, and the second intersection point when the side bottom edge of the pseudo three-dimensional bounding box is detected, the method further includes: connecting the base point to the other endpoint to obtain a first line segment representing the length of the obstacle; connecting the base point to the first intersection point to obtain a second line segment representing the height of the obstacle; connecting the base point to the second intersection point to obtain a third line segment representing the width of the obstacle; and recovering the three-dimensional bounding box of the obstacle based on the first line segment, the second line segment, and the third line segment.

[0014] In conjunction with the second possible implementation of the first aspect, in the sixth possible implementation of the first aspect, the step of using the lane line direction or the camera's line of sight as the heading of the obstacle includes: detecting the distance between the obstacle and the camera; if the distance is less than or equal to a threshold, using the direction of the lane line of the lane where the obstacle is located as the heading of the obstacle; if the distance is greater than the threshold, using the camera's line of sight as the heading of the obstacle.

[0015] Secondly, this application provides a size estimation device, which includes: a detection unit for detecting an image including a pseudo-3D bounding box and a 2D bounding box of an obstacle to obtain two endpoints of the bottom edge of the pseudo-3D bounding box and determine a base point based on the two endpoints; a construction unit for constructing a first ray and a second ray in a world coordinate system based on the base point, wherein the first ray and the second ray are respectively used to point in the directions of the height and width of the obstacle; a projection unit for projecting the first ray and the second ray onto the image, such that the projections of the first ray and the second ray intersect the top edge and the side edge of the 2D bounding box at a first intersection point and a second intersection point, respectively; and a determination unit for determining the size information of the obstacle based on the two endpoints, the first intersection point, and the second intersection point, wherein the size information includes at least one of length, width, height, and center point.

[0016] In conjunction with the second aspect, in the first possible implementation of the second aspect, the detection unit is specifically used to: detect the side bottom edge, front bottom edge, or back bottom edge of the three-dimensional bounding box on the image; if the side bottom edge of the pseudo-three-dimensional bounding box is detected, obtain the two endpoints of the side bottom edge of the pseudo-three-dimensional bounding box and use one of the two endpoints as a base point; if the side bottom edge of the pseudo-three-dimensional bounding box is not detected, obtain the two endpoints of the front bottom edge or back bottom edge of the pseudo-three-dimensional bounding box and use the midpoint between the two endpoints as a base point.

[0017] In conjunction with the first possible implementation of the second aspect, in the second possible implementation of the second aspect, the aforementioned construction unit is specifically used for: when the side bottom edge of the pseudo-3D bounding box is detected, connecting two endpoints to determine the heading of the obstacle; when the side bottom edge of the pseudo-3D bounding box is not detected, taking the camera's line of sight or the direction of the lane line of the lane where the obstacle is located as the heading of the obstacle; obtaining the world coordinates of the base point based on the pixel coordinates of the base point on the image; and constructing a first ray and a second ray starting from the world coordinates of the base point, wherein the first ray is perpendicular to the ground where the obstacle is located and points to the top of the obstacle, and the second ray is perpendicular to the heading and the first ray and points to the side of the obstacle.

[0018] In conjunction with the first possible implementation of the second aspect, in the third possible implementation of the second aspect, the aforementioned determining unit is specifically used for: when the side bottom edge of the pseudo-3D bounding box is detected, converting the pixel coordinates of the two endpoints, the first intersection point, and the second intersection point to obtain the world coordinates of the two endpoints, the first intersection point, and the second intersection point respectively, and determining the length, width, height, and / or center point of the obstacle based on the world coordinates of the two endpoints, the first intersection point, and the second intersection point; when the side bottom edge of the pseudo-3D bounding box is not detected, converting the pixel coordinates of the midpoint of the two endpoints, the first intersection point, and the second intersection point to obtain the world coordinates of the midpoint of the two endpoints, the first intersection point, and the second intersection point respectively, and determining the width and / or height of the obstacle based on the world coordinates of the midpoint of the two endpoints, the first intersection point, and the second intersection point.

[0019] In conjunction with the second aspect, in the fourth possible implementation of the second aspect, the aforementioned determining unit is specifically used to: determine the height of the obstacle based on the pixel coordinates of the base point, the world coordinates of the base point, the pixel coordinates of the first intersection point, and the unit vector of the first ray, wherein the mathematical expression of the step of determining the height of the obstacle based on the pixel coordinates of the base point, the world coordinates of the base point, the pixel coordinates of the first intersection point, and the unit vector of the first ray is as follows:

[0020] or,

[0021] Where L is the height of the obstacle, (r1,c1) are the pixel coordinates of the base point, (x1,y1,z1) are the world coordinates of the base point, (r3,c3) are the pixel coordinates of the first intersection point, and (u x ,u y ,u z Let f be the unit vector of the first ray. x and f y This refers to the camera's intrinsic focal length parameter.

[0022] In conjunction with the first possible implementation of the second aspect, in the fifth possible implementation of the second aspect, the above-mentioned size estimation device further includes a recovery unit, which is used to: connect a base point to another endpoint to obtain a first line segment representing the length of the obstacle; connect the base point to a first intersection point to obtain a second line segment representing the height of the obstacle; connect the base point to a second intersection point to obtain a third line segment representing the width of the obstacle; and recover the three-dimensional bounding box of the obstacle based on the first line segment, the second line segment, and the third line segment.

[0023] In conjunction with the second possible implementation of the second aspect, in the sixth possible implementation of the second aspect, the above-mentioned building unit is further configured to: detect the distance between the obstacle and the camera; when the distance is less than or equal to a threshold, take the direction of the lane line of the lane where the obstacle is located as the heading of the obstacle; when the distance is greater than the threshold, take the direction of the camera's line of sight as the heading of the obstacle.

[0024] Thirdly, this application also provides a size estimation device, which includes a processor and a memory connected via a bus; the processor is used to execute multiple instructions; the memory is used to store multiple instructions, which are adapted to be loaded by the processor and executed as a size estimation method as described in the first aspect or any embodiment of the first aspect.

[0025] Fourthly, this application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor and executing a size estimation method as described in the first aspect or any embodiment of the first aspect.

[0026] In summary, this application provides a size estimation method, apparatus, device, and storage medium based on a single camera. The size estimation method includes: detecting an image containing a pseudo-3D bounding box and a 2D bounding box of an obstacle to obtain the two endpoints of the bottom edge of the pseudo-3D bounding box, and determining a base point based on the two endpoints; constructing a first ray and a second ray in a world coordinate system based on the base point; projecting the first ray and the second ray onto the image, such that the projections of the first ray and the second ray intersect the top edge and side edge of the 2D bounding box at a first intersection point and a second intersection point, respectively; and determining the size information of the obstacle based on the two endpoints, the first intersection point, and the second intersection point. It is evident that the size estimation method of this application can recover the size information of the obstacle in 3D space using only the pseudo-3D bounding box and the 2D bounding box in an image captured by a single camera. Therefore, by using visual perception technology to achieve obstacle size estimation, it can not only accurately calculate the size information of the obstacle but also greatly reduce computational complexity, solving the problem of low size estimation efficiency in existing technologies. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of an image including a pseudo-3D bounding box and a 2D bounding box of obstacles in one embodiment;

[0028] Figure 2 This is a schematic diagram of an image including a pseudo-3D bounding box and a 2D bounding box of obstacles in another embodiment;

[0029] Figure 3 This is a schematic block diagram of a pseudo-3D bounding box in one embodiment;

[0030] Figure 4 A schematic block diagram of a pseudo-3D bounding box in another embodiment;

[0031] Figure 5 This is a schematic diagram illustrating the steps for constructing the first and second rays in one embodiment;

[0032] Figure 6 This is a schematic diagram illustrating the steps for determining the first intersection point and the second intersection point in one embodiment;

[0033] Figure 7 This is a schematic diagram illustrating the steps for determining the first intersection point and the second intersection point in another embodiment;

[0034] Figure 8 This is a schematic diagram illustrating the steps for determining the height of an obstacle in one embodiment;

[0035] Figure 9 This is a schematic diagram illustrating the steps of constructing a 3D bounding box for an obstacle in one embodiment;

[0036] Figure 10 This is a flowchart illustrating the size estimation method in one embodiment;

[0037] Figure 11 A schematic block diagram of a size estimation device provided in this application;

[0038] Figure 12 A structural block diagram of a size estimation device provided in this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] Since the embodiments of this application involve a relatively large number of technical terms, for ease of understanding, the relevant terms and concepts that may be involved in the embodiments of this application will be introduced below.

[0041] 1. Two-dimensional (2D) bounding box

[0042] Two-dimensional bounding boxes, also known as rectangular box annotations, include regular rectangles and oblique rectangles. In this application, the pseudo-3D bounding box refers to a regular rectangle, for example... Figure 1 As shown, the two-dimensional bounding box is GCEH.

[0043] 2. Pseudo-3D (3D) bounding box

[0044] A pseudo-3D bounding box, also known as a 2.5D bounding box, is a bounding box that treats obstacles in an image as "pseudo-cubes." Generally, pseudo-3D bounding boxes include frontal / side 2D view cubes and frontal / side-top 3D view cubes, etc. In this application, the pseudo-3D bounding box refers to a frontal / side 2D view cube. A frontal / side 2D view cube includes a 2D bounding box representing the side of the obstacle and / or a 2D bounding box representing the front (back) of the obstacle.

[0045] It should be noted that when an obstacle is located at a side position close to the camera, the camera can capture not only the side of the obstacle, but also its back or front. In this case, for example... Figure 1 As shown, the pseudo-3D bounding box of the obstacle includes a 2D bounding box ABCD representing the side of the obstacle, and a 2D bounding box ADEF representing the back of the obstacle. This pseudo-3D bounding box includes the bottom edge of the side (line segment AB) and the bottom edge of the back (line segment AF). When the obstacle is located directly in front of or behind the camera, or at a distance from the camera, the camera can only capture the front or back of the obstacle. In this case, for example... Figure 2 As shown, the pseudo-3D bounding box of the obstacle only includes the 2D bounding box ADEF used to represent the back of the obstacle, which includes the bottom edge of the back (line segment AF).

[0046] 3. Three-dimensional (3D) bounding box

[0047] A 3D bounding box is a six-sided cube used to identify the true three-dimensional boundaries of an obstacle. When displayed on an image, this 3D bounding box looks like... Figure 9 The six-sided cube used to frame the space where the car is located.

[0048] It should be noted that the size estimation device and apparatus involved in this application may include, but are not limited to, dedicated size estimation devices, terminal devices, computers, processors, etc., and may be a device integrated into a vehicle or a detachable, independent device within a vehicle. The size estimation device and apparatus can interact with cameras on the vehicle, for example, acquiring images of obstacles captured by the camera. The processor may include, but is not limited to, a central processing unit (CPU), a general-purpose processor, a coprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The processor can implement the methods described in this application, such as detecting images including pseudo-3D bounding boxes and 2D bounding boxes of obstacles, which will not be elaborated further in this application.

[0049] It should also be noted that the illustrations provided in this embodiment are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show components related to the present invention and are not drawn according to the actual number, shape, and size of the components in implementation. In actual implementation, the form, quantity, and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex. The structures, proportions, sizes, etc., shown in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modification to the structure, change in the proportional relationship, or adjustment of the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention. Furthermore, the orientations or positional relationships indicated by terms such as "upper," "lower," "left," "right," "middle," "longitudinal," "lateral," "horizontal," "inner," "outer," "radial," and "circumferential" used in this specification are based on the orientations or positional relationships shown in the accompanying drawings and are only for the purpose of simplifying the description. They do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the present invention. Therefore, they should not be construed as limitations on the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0050] Currently, existing technologies often employ radar sensing to estimate the size of obstacles. This is because radar detection can construct a realistic 3D bounding box for the obstacle, which directly reflects the obstacle's length, height, and width. Although radar sensing technology has relatively high detection accuracy, it involves a large amount of data and complex calculations, thus its size estimation efficiency is not high in practical applications.

[0051] In response, this application employs visual perception technology to estimate the size of obstacles, which not only accurately estimates the size of obstacles but also significantly reduces hardware costs and simplifies calculations, thereby improving the problem of low size estimation efficiency in existing technologies.

[0052] Specifically, to balance the accuracy and computational ease of size estimation and address the issue of low efficiency in existing size estimation technologies, this application proposes a single-camera-based size estimation method that incorporates visual perception technology. Next, this application will combine... Figure 1 and Figure 2 The images shown include pseudo-3D bounding boxes and 2D bounding boxes containing obstacles, and the size estimation method proposed in this application is explained using a size estimation device as the execution entity. Specifically:

[0053] First, the size estimation device detects the pseudo-3D bounding box and the 2D bounding box in the image, including obstacles, to obtain the two endpoints of the bottom edge of the pseudo-3D bounding box, and determines the base point based on the two endpoints, such as... Figure 1 The image shown includes obstacles, as well as pseudo-3D bounding boxes ABCDEF and 2D bounding boxes GCEH used to label the locations of the obstacles. Figure 2 The image shown includes obstacles, as well as pseudo-3D bounding boxes ADEF and 2D bounding boxes GCEH used to mark the location of the obstacles. It should be noted that when the size estimation device detects the bottom edge of the pseudo-3D bounding box of the obstacle, it prioritizes detecting the side bottom edges. If no side bottom edge is detected, it then detects the back or front bottom edge. This is because the side bottom edge is closer to the true boundary of the obstacle than the back or front bottom edge, and thus more accurately reflects the true position of the obstacle. Therefore, depending on whether the pseudo-3D bounding box in the image includes side bottom edges, the implementation process of the size estimation method in this application can be divided into two cases: the first is when the side bottom edge is detected, and the second is when the side bottom edge is not detected, but the back or front bottom edge is detected. In the first case, the size estimation device uses one of the two endpoints of the pseudo side bottom edge as a base point, for example... Figure 3In the pseudo-3D bounding box, the side bottom edge is line segment AB, and the two endpoints of the side bottom edge are points A and B, with point A used as the base point; in the second case, the size estimation device uses the midpoint between the two endpoints of the back bottom edge or the front bottom edge as the base point, for example... Figure 4 The back bottom edge of the pseudo 3D bounding box is line segment AF, and the two endpoints of the back bottom edge are point A and point F. The midpoint P between point A and point F is taken as the base point.

[0054] Secondly, after determining the base point, the size estimation device obtains the world coordinates of the base point based on its pixel coordinates on the image. Using these world coordinates as the starting point, it constructs a first ray and a second ray. The first ray is perpendicular to the ground where the obstacle is located and points towards the top of the obstacle. The second ray is perpendicular to both the obstacle's heading and the first ray's direction and points towards the side of the obstacle. The unit vector of the first ray is (0,0,1). For example, such as... Figure 5 As shown, the size estimation device first transforms the pixel coordinates (r1, c1) of point A (base point) on the image to obtain the world coordinates (x1, y1, z1) of point A in the world coordinate system, which is point a. Then, the size estimation device establishes a first ray β perpendicular to the ground and pointing to the top of the obstacle, starting from point a, and establishes a second ray γ perpendicular to the first ray β and the obstacle's heading α, pointing to the side of the obstacle. It should be noted that in the first case above, the obstacle's heading is the direction indicated by the line connecting the two endpoints of the bottom edge of the side in the world coordinate system, for example... Figure 5 As shown, the size estimation device transforms the pixel coordinates (r1, c1) of point A on the bottom edge of the side image into the world coordinates (x1, y1, z1), and transforms the pixel coordinates (r2, c2) of point B on the bottom edge of the side image into the world coordinates (x2, y2, z2). Then, starting from point a, a target vector pointing to point b is established, and this target vector is used as the heading α of the obstacle. In the second case, since the side of the obstacle is not visible, the direction of the camera's line of sight or the direction of the lane line of the lane where the obstacle is located is used as the heading of the obstacle. It should also be noted that in the first case, the side to which the second ray points is another invisible side of the obstacle, that is, the side opposite to the side where the base point is located; in the second case, the side to which the second ray points is either of the two sides of the obstacle.

[0055] Next, after constructing the first and second rays in the world coordinate system, project the first and second rays onto the image, such that the projections of the first and second rays intersect the top and side edges of the two-dimensional bounding box at the first and second intersection points, respectively. In the first case described above, as... Figure 6As shown, the projection of the first ray β onto the image is β', and the projection of the second ray γ onto the image is γ'. The projections β' of the first ray and γ' of the second ray intersect the top edge CE and the side edge EH of the two-dimensional bounding box GCEH at the first intersection point M and the second intersection point N, respectively. In the second case described above, as... Figure 7 As shown, the projection of the first ray β onto the image is β', and the projection of the second ray γ onto the image is γ'. The projections of the first ray β' and the second ray γ' intersect the top edge CE and the side edge EH of the two-dimensional bounding box GCEH at the first intersection point M and the second intersection point N, respectively.

[0056] Finally, after determining four points on the image (two endpoints, a first intersection, and a second intersection), the size estimation device determines the obstacle's size information based on these four points. The size information includes at least one of length, width, height, and center point. In the first case described above, the size estimation device can determine the obstacle's length, width, height, and center point based on these four points, for example... Figure 6 As shown, the size estimation device first converts the pixel coordinates of points A, B, M, and N in the image to obtain their world coordinates in the world coordinate system. Then, it calculates the distance between the world coordinates of points A and B using the Pythagorean theorem to obtain the length of the obstacle; it calculates the distance between the world coordinates of points A and M to obtain the height of the obstacle; and it calculates the distance between the world coordinates of points A and N to obtain the width of the obstacle. Finally, it takes the midpoint of the world coordinates of points A, B, M, and N to obtain the center point of the obstacle. In the second case described above, the size estimation device can determine the length, width, height, and center point of the obstacle based on these four points. For example... Figure 7 As shown, the size estimation device first converts the pixel coordinates of the center points P, M, and N of the two endpoints (points A and F) in the image to obtain the world coordinates of points P, M, and N in the world coordinate system. Then, it obtains a preset length as the length of the obstacle and calculates the distance between the world coordinates of points P and M using the Pythagorean theorem to obtain the height of the obstacle. Finally, it calculates twice the distance between the world coordinates of points P and N to obtain the width of the obstacle. It should be noted that the size estimation device can convert the pixel coordinates of any point in the image to world coordinates based on camera intrinsic and extrinsic parameters, which will not be elaborated further in this application.

[0057] In one embodiment, such as Figure 10 As shown, this application provides a size estimation method based on a single camera. Next, this application will describe the method provided in this embodiment using a size estimation device as the execution subject. Specifically:

[0058] 1001: Detect the images of pseudo-3D bounding boxes and 2D bounding boxes including obstacles to obtain the two endpoints of the bottom edge of the pseudo-3D bounding box, and determine the base point based on the two endpoints.

[0059] The size estimation device acquires an image captured by a single camera, which includes obstacles and pseudo-3D bounding boxes and 2D bounding boxes used to label the positions of the obstacles. After acquiring the image, the size estimation device examines the side bottom edge, back bottom edge, and front bottom edge of the pseudo-3D bounding box in the image, obtains the two endpoints of one of the bottom edges, and determines a base point based on these two endpoints. The determination of the base point is done in two ways, depending on whether the side bottom edge of the 3D bounding box is detected. Specifically:

[0060] The first method involves detecting the bottom side edge of the pseudo-3D bounding box, obtaining the two endpoints of the bottom side edge, and using one of these endpoints as a base point. For example... Figure 3 The pseudo-3D bounding box in the figure has a side bottom edge of line segment AB, and two endpoints of the side bottom edge are point A and point B. Point A is taken as the base point.

[0061] The second method involves obtaining the two endpoints of the front or back bottom edge of the pseudo-3D bounding box when no side bottom edge is detected, and using the midpoint between these two endpoints as the base point. For example... Figure 4 The pseudo-3D bounding box in the image has a front bottom edge as line segment AF, and two endpoints of the front bottom edge as points A and F. The midpoint P between points A and F is taken as the base point.

[0062] 1002: Construct the first and second rays in the world coordinate system based on the base point.

[0063] The size estimation device transforms the pixel coordinates of the base point in the image to obtain its world coordinates in the world coordinate system. Using these world coordinates as the starting point, it constructs a first ray and a second ray, which point in the directions of the obstacle's height and width, respectively. Specifically, the first ray is perpendicular to the ground where the obstacle is located and points towards the top of the obstacle, while the second ray is perpendicular to both the heading and the first ray direction and points towards the side of the obstacle. For example... Figure 5 As shown, the size estimation device first transforms the pixel coordinates (r1, c1) of point A (base point) on the image to obtain the world coordinates (x1, y1, z1) of point A in the world coordinate system, which is point a. Then, the size estimation device establishes a first ray β perpendicular to the ground and pointing to the top of the obstacle, starting from point a, and establishes a second ray γ perpendicular to the first ray β and the obstacle's heading α, pointing to the side of the obstacle. It should be noted that the obstacle's heading can be determined in two ways depending on whether the bottom edge of the 3D bounding box is detected. Specifically:

[0064] The first method involves connecting the two endpoints to determine the obstacle's heading when the bottom side edge of a pseudo-3D bounding box is detected. For example... Figure 5 As shown, the size estimation device transforms the pixel coordinates (r1, c1) of point A on the bottom edge of the side to obtain the world coordinates (x1, y1, z1) of point A in the world coordinate system, and transforms the pixel coordinates (r2, c2) of point B on the bottom edge of the side to obtain the world coordinates (x2, y2, z2) of point B in the world coordinate system. Then, starting from point a, a target vector pointing to point b is established, and this target vector is used as the heading α of the obstacle.

[0065] The second approach, when no side bottom edge of the pseudo-3D bounding box is detected, uses the camera's line of sight or the direction of the lane line of the lane containing the obstacle as the obstacle's heading. Further, in this second approach, to determine the obstacle's heading, the size estimation device first detects the distance between the obstacle and the camera. Then, if the distance is less than or equal to a threshold, the direction of the lane line of the lane containing the obstacle is used as the obstacle's heading; if the distance is greater than the threshold, the camera's line of sight is used as the obstacle's heading.

[0066] 1003: Project the first ray and the second ray onto the image such that the projections of the first ray and the second ray intersect the top edge and the side edge of the two-dimensional bounding box at the first intersection point and the second intersection point, respectively.

[0067] The size estimation device projects a first ray and a second ray in the world coordinate system onto the image, such that the first ray intersects the bottom edge of the 2D bounding box in the image at a first intersection point, and the second ray intersects the 2D bounding box in the image at a second intersection point. It should be noted that the first and second intersection points can be determined in two ways, depending on whether the side bottom edge of the 3D bounding box is detected. Specifically:

[0068] The first method involves detecting the bottom side edge of a pseudo-3D bounding box, such as... Figure 6 As shown, the projection of the first ray β onto the image is β', and the projection of the second ray γ onto the image is γ'. The projections of the first ray β' and the second ray γ' intersect the top edge CE and the side edge EH of the two-dimensional bounding box GCEH at the first intersection point M and the second intersection point N, respectively.

[0069] The second method, in the case where the side bottom edge of the pseudo-3D bounding box is not detected, such as Figure 7 As shown, the projection of the first ray β onto the image is β', and the projection of the second ray γ onto the image is γ'. The projections of the first ray β' and the second ray γ' intersect the top edge CE and the side edge EH of the two-dimensional bounding box GCEH at the first intersection point M and the second intersection point N, respectively.

[0070] 1004: Determine the size information of the obstacle based on the two endpoints, the first intersection point, and the second intersection point.

[0071] After determining four points on the image (two endpoints, a first intersection, and a second intersection), the size estimation device determines the obstacle's size information based on these four points. The size information includes at least one of length, width, height, and center point. It should be noted that the size information can be determined in two ways, depending on whether the side bottom edge of the 3D bounding box is detected. Specifically:

[0072] The first approach involves detecting the side bottom edge of a pseudo-3D bounding box. The size estimation device can convert the pixel coordinates of the two endpoints, the first intersection point, and the second intersection point into world coordinates, and then determine the length, width, height, and / or center point of the obstacle based on these world coordinates. For example... Figure 6 As shown, the size estimation device first converts the pixel coordinates of points A, B, M, and N on the image to obtain their world coordinates in the world coordinate system. Then, it calculates the distance between the world coordinates of points A and B using the Pythagorean theorem to obtain the length of the obstacle. It calculates the distance between the world coordinates of points A and M to obtain the height of the obstacle. It calculates the distance between the world coordinates of points A and N to obtain the width of the obstacle. Finally, it takes the midpoint of the world coordinates of points A, B, M, and N to obtain the center point of the obstacle.

[0073] The second approach, where no side bottom edge of the pseudo-3D bounding box is detected, involves the size estimation device converting the pixel coordinates of the midpoint, first intersection, and second intersection of the two endpoints into world coordinates, and then determining the width and / or height of the obstacle based on these world coordinates. For example... Figure 7 As shown, the size estimation device first converts the pixel coordinates of the center points P, M, and N of the two endpoints (point A and point F) in the image to obtain the world coordinates of points P, M, and N in the world coordinate system. Then, it obtains the preset length as the length of the obstacle, and calculates the distance between the world coordinates of points P and M according to the Pythagorean theorem to obtain the height of the obstacle. Finally, it calculates twice the distance between the world coordinates of points P and N to obtain the width of the obstacle.

[0074] In one feasible embodiment, the step of determining the obstacle's size information based on the two endpoints, the first intersection point, and the second intersection point further includes: determining the obstacle's height based on the pixel coordinates of the base point, the world coordinates of the base point, the pixel coordinates of the first intersection point, and the unit vector of the first ray. The mathematical expression for determining the obstacle's height based on the pixel coordinates of the base point, the world coordinates of the base point, the pixel coordinates of the first intersection point, and the unit vector of the first ray is as follows:

[0075] or,

[0076] Where L is the height of the obstacle, (r1,c1) are the pixel coordinates of the base point, (x1,y1,z1) are the world coordinates of the base point, (r3,c3) are the pixel coordinates of the first intersection point, and (u x ,u y ,u z Let f be the unit vector of the first ray. x and f y This refers to the camera's intrinsic focal length parameter. It should be noted that this embodiment provides a simplified calculation method. After mapping the first ray onto the image and obtaining the first intersection point, the height of the obstacle is directly calculated using the pixel coordinates of the first intersection point. This eliminates the need to first convert the pixel coordinates of the first intersection point to its world coordinates before using those coordinates to calculate the obstacle's height, thereby further reducing computational complexity and improving the efficiency of size estimation.

[0077] Next, this application will use the case of detecting the side bottom edge of the pseudo 3D bounding box as an example to illustrate this step, such as... Figure 8 As shown, O is the location of the camera, the pixel coordinates of the base point A are (r1, c1), the world coordinates of the base point A are (x1, y1, z1), the pixel coordinates of the first intersection point M are (r3, c3), and the unit vector of the first ray is (u). x ,u y ,u z Camera intrinsic focal length parameter f x and f y Since all these parameters are known, the size estimation device can directly substitute these known parameters into the above mathematical expression to obtain the height of the obstacle, i.e., the length of line segment AM. Similarly, the width of the obstacle can also be calculated in the same way, based on the pixel coordinates of the base point, the world coordinates of the base point, the pixel coordinates of the second intersection point, and the unit vector of the second ray. The specific mathematical expression can be found in the mathematical expression used to calculate the length of the obstacle, and will not be repeated here.

[0078] In one feasible approach, the size estimation device can also construct a three-dimensional bounding box based on the obstacle size information obtained in the aforementioned steps to reflect the true three-dimensional boundary of the obstacle. Specifically, in the case of detecting the side bottom edge of the pseudo-three-dimensional bounding box, after the step of determining the obstacle size information based on the two endpoints, the first intersection point, and the second intersection point, the device further includes: connecting the base point to the other endpoint to obtain a first line segment representing the length of the obstacle; connecting the base point to the first intersection point to obtain a second line segment representing the height of the obstacle; connecting the base point to the second intersection point to obtain a third line segment representing the width of the obstacle; and recovering the three-dimensional bounding box of the obstacle based on the first, second, and third line segments.

[0079] The size estimation device reconstructs a six-sided cube representing the true three-dimensional boundary of the obstacle based on the first, second, and third line segments. For example... Figure 9 As shown, the size estimation device first connects the base point A to the other endpoint B to obtain the first line segment AB, connects the base point A to the first intersection point M to obtain the second line segment AM, and connects the base point A to the second intersection point N to obtain the third line segment AN. Then, based on the line segments AB, AM and AN, the other line segments of the three-dimensional bounding box are recovered, thereby recovering the three-dimensional bounding box of the obstacle.

[0080] In summary, the size estimation method of this application can recover the size information of obstacles in three-dimensional space based solely on the pseudo-3D bounding boxes and 2D bounding boxes in images captured by a single camera. It is evident that achieving obstacle size estimation based on visual perception technology not only accurately calculates obstacle size information but also significantly reduces computational complexity, solving the problem of low efficiency in size estimation in existing technologies.

[0081] In one embodiment, the present invention also provides a size estimation device, see [link to previous embodiment]. Figure 11 The embodiments of the present invention can divide the device into functional units according to the above method examples. For example, each function can be divided into separate functional units, or two or more functions can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in the embodiments of the present invention is illustrative and only represents one logical functional division; other division methods may be used in actual implementation. Figure 5As shown, the obstacle detection device includes a detection unit 1110, a construction unit 1120, a projection unit 1130, and a determination unit 1140. Specifically: the detection unit 1110 is used to detect images including a pseudo-3D bounding box and a 2D bounding box containing obstacles to obtain the two endpoints of the bottom edge of the pseudo-3D bounding box and determine a base point based on the two endpoints; the construction unit 1120 is used to construct a first ray and a second ray in the world coordinate system based on the base point, wherein the first ray and the second ray are respectively used to point to the height and width of the obstacle; the projection unit 1130 is used to project the first ray and the second ray onto the image, such that the projections of the first ray and the second ray intersect the top edge and the side edge of the 2D bounding box at a first intersection point and a second intersection point, respectively; the determination unit 1140 is used to determine the size information of the obstacle based on the two endpoints, the first intersection point, and the second intersection point, wherein the size information includes at least one of length, width, height, and center point.

[0082] In one feasible embodiment, the detection unit 1110 is specifically used to: detect the side bottom edge, front bottom edge, or back bottom edge of a three-dimensional bounding box on the image; if the side bottom edge of a pseudo-three-dimensional bounding box is detected, obtain the two endpoints of the side bottom edge of the pseudo-three-dimensional bounding box and use one of the two endpoints as a base point; if the side bottom edge of the pseudo-three-dimensional bounding box is not detected, obtain the two endpoints of the front bottom edge or back bottom edge of the pseudo-three-dimensional bounding box and use the midpoint between the two endpoints as a base point.

[0083] In one feasible embodiment, the aforementioned building unit 1120 is specifically used to: connect two endpoints to determine the heading of the obstacle when the side bottom edge of the pseudo-3D bounding box is detected; when the side bottom edge of the pseudo-3D bounding box is not detected, take the camera's line of sight or the direction of the lane line of the lane where the obstacle is located as the heading of the obstacle; obtain the world coordinates of the base point based on the pixel coordinates of the base point on the image; and construct a first ray and a second ray starting from the world coordinates of the base point, wherein the first ray is perpendicular to the ground where the obstacle is located and points to the top of the obstacle, and the second ray is perpendicular to the heading and the first ray and points to the side of the obstacle.

[0084] In one implementable manner, the determining unit 1140 is specifically configured to: when the side bottom edge of the pseudo-3D bounding box is detected, convert the pixel coordinates of the two endpoints, the first intersection point, and the second intersection point to obtain the world coordinates of the two endpoints, the first intersection point, and the second intersection point respectively, and determine the length, width, height, and / or center point of the obstacle based on the world coordinates of the two endpoints, the first intersection point, and the second intersection point; when the side bottom edge of the pseudo-3D bounding box is not detected, convert the pixel coordinates of the midpoint of the two endpoints, the first intersection point, and the second intersection point to obtain the world coordinates of the midpoint of the two endpoints, the first intersection point, and the second intersection point respectively, and determine the width and / or height of the obstacle based on the world coordinates of the midpoint of the two endpoints, the first intersection point, and the second intersection point.

[0085] In one implementable manner, the determining unit 1140 is specifically used to: determine the height of the obstacle based on the pixel coordinates of the base point, the world coordinates of the base point, the pixel coordinates of the first intersection point, and the unit vector of the first ray, wherein the mathematical expression of the step of determining the height of the obstacle based on the pixel coordinates of the base point, the world coordinates of the base point, the pixel coordinates of the first intersection point, and the unit vector of the first ray is as follows:

[0086] or,

[0087] Where L is the height of the obstacle, (r1,c1) are the pixel coordinates of the base point, (x1,y1,z1) are the world coordinates of the base point, (r3,c3) are the pixel coordinates of the first intersection point, and (u x ,u y ,u z Let f be the unit vector of the first ray. x and f y This refers to the camera's intrinsic focal length parameter.

[0088] In one implementable embodiment, the aforementioned size estimation device further includes a recovery unit 1150, which is configured to: connect a base point to another endpoint to obtain a first line segment representing the length of the obstacle; connect the base point to a first intersection point to obtain a second line segment representing the height of the obstacle; connect the base point to a second intersection point to obtain a third line segment representing the width of the obstacle; and recover the three-dimensional bounding box of the obstacle based on the first line segment, the second line segment, and the third line segment.

[0089] In one implementable manner, the aforementioned building unit 1120 is further configured to: detect the distance between the obstacle and the camera; if the distance is less than or equal to a threshold, use the direction of the lane line of the lane where the obstacle is located as the heading of the obstacle; if the distance is greater than the threshold, use the direction of the camera's line of sight as the heading of the obstacle.

[0090] In one embodiment, this application also provides a size estimation device, see [link to relevant documentation]. Figure 12 The size estimation device can be, but is not limited to, various personal computers, laptops, smartphones, tablets, portable wearable devices, and servers. The server can be a standalone server or a server cluster consisting of multiple servers. As shown in the figure, the size estimation device in this embodiment may include a processor 1210 and a memory 1220. The processor 1210 and the memory 1220 are connected via a link 1230. The processor 1210 is used to execute multiple instructions; the memory 1220 is used to store multiple instructions adapted to be loaded by the processor 1210 and executed as in the size estimation method described in the above embodiment.

[0091] The processor 1210 can be an Electronic Control Unit (ECU), a Central Processing Unit (CPU), a general-purpose processor, a coprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The processor 1210 can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. In this embodiment, the processor 1210 can be a microcontroller. By programming the microcontroller, various control functions can be implemented. For example, in this embodiment, it can implement the detection function of images including pseudo-3D bounding boxes and 2D bounding boxes containing obstacles. The processor has the advantages of powerful computing capabilities and fast processing speed. Specifically: the processor 1210 is used to execute the function of the detection unit 1110, to detect the image including the pseudo-3D bounding box and the 2D bounding box of the obstacle, to obtain the two endpoints of the bottom edge of the pseudo-3D bounding box, and to determine the base point based on the two endpoints; it is also used to execute the function of the construction unit 1120, to construct a first ray and a second ray in the world coordinate system based on the base point, wherein the first ray and the second ray are respectively used to point to the height and width of the obstacle; it is also used to execute the function of the projection unit 1130, to project the first ray and the second ray onto the image, such that the projections of the first ray and the second ray intersect the top edge and the side edge of the 2D bounding box at a first intersection point and a second intersection point, respectively; and it is also used to execute the function of the determination unit 1140, to determine the size information of the obstacle based on the two endpoints, the first intersection point and the second intersection point, wherein the size information includes at least one of length, width, height and center point.

[0092] In one implementable manner, the processor 1210 is specifically configured to: detect the side bottom edge, front bottom edge, or back bottom edge of a 3D bounding box on an image; if the side bottom edge of a pseudo 3D bounding box is detected, obtain the two endpoints of the side bottom edge of the pseudo 3D bounding box and use one of the two endpoints as a base point; if the side bottom edge of the pseudo 3D bounding box is not detected, obtain the two endpoints of the front bottom edge or back bottom edge of the pseudo 3D bounding box and use the midpoint between the two endpoints as a base point.

[0093] In one implementable manner, the processor 1210 is specifically configured to: when the side bottom edge of the pseudo-3D bounding box is detected, connect two endpoints to determine the heading of the obstacle; when the side bottom edge of the pseudo-3D bounding box is not detected, take the camera's line of sight or the direction of the lane line of the lane where the obstacle is located as the heading of the obstacle; obtain the world coordinates of the base point based on the pixel coordinates of the base point on the image; and construct a first ray and a second ray starting from the world coordinates of the base point, wherein the first ray is perpendicular to the ground where the obstacle is located and points to the top of the obstacle, and the second ray is perpendicular to the heading and the first ray and points to the side of the obstacle.

[0094] In one implementable manner, the processor 1210 is specifically configured to: when the side bottom edge of the pseudo-3D bounding box is detected, convert the pixel coordinates of the two endpoints, the first intersection point, and the second intersection point to obtain the world coordinates of the two endpoints, the first intersection point, and the second intersection point respectively, and determine the length, width, height, and / or center point of the obstacle based on the world coordinates of the two endpoints, the first intersection point, and the second intersection point; when the side bottom edge of the pseudo-3D bounding box is not detected, convert the pixel coordinates of the midpoint of the two endpoints, the first intersection point, and the second intersection point to obtain the world coordinates of the midpoint of the two endpoints, the first intersection point, and the second intersection point respectively, and determine the width and / or height of the obstacle based on the world coordinates of the midpoint of the two endpoints, the first intersection point, and the second intersection point.

[0095] In one implementable manner, the processor 1210 is specifically used to: determine the height of the obstacle based on the pixel coordinates of the base point, the world coordinates of the base point, the pixel coordinates of the first intersection point, and the unit vector of the first ray, wherein the mathematical expression of the step of determining the height of the obstacle based on the pixel coordinates of the base point, the world coordinates of the base point, the pixel coordinates of the first intersection point, and the unit vector of the first ray is as follows:

[0096] or,

[0097] Where L is the height of the obstacle, (r1,c1) are the pixel coordinates of the base point, (x1,y1,z1) are the world coordinates of the base point, (r3,c3) are the pixel coordinates of the first intersection point, and (u x ,u y ,u z Let f be the unit vector of the first ray. x and f y This refers to the camera's intrinsic focal length parameter.

[0098] In one possible implementation, the processor 1210 is further configured to perform the function of the recovery unit 1150, for: connecting the base point to another endpoint to obtain a first line segment representing the length of the obstacle; connecting the base point to a first intersection point to obtain a second line segment representing the height of the obstacle; connecting the base point to a second intersection point to obtain a third line segment representing the width of the obstacle; and recovering the three-dimensional bounding box of the obstacle based on the first line segment, the second line segment, and the third line segment.

[0099] In one implementable manner, the processor 1210 is further configured to: detect the distance between the obstacle and the camera; if the distance is less than or equal to a threshold, use the direction of the lane line of the lane where the obstacle is located as the heading of the obstacle; if the distance is greater than the threshold, use the direction of the camera's line of sight as the heading of the obstacle.

[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A size estimation method based on a single camera, characterized in that, include: The image of the pseudo 3D bounding box and the 2D bounding box including the obstacle is detected to obtain the two endpoints of the bottom edge of the pseudo 3D bounding box, and a base point is determined based on the two endpoints. The base point is either one of the two endpoints of the side bottom edge of the 3D bounding box, or the midpoint between the two endpoints of the front bottom edge or the back bottom edge of the 3D bounding box. Based on the base point, a first ray and a second ray are constructed in the world coordinate system, wherein the first ray and the second ray are respectively used to point to the height and width of the obstacle; The first ray and the second ray are projected onto the image such that the projections of the first ray and the second ray intersect the top edge and the side edge of the two-dimensional bounding box at a first intersection point and a second intersection point, respectively. Based on the two endpoints, the first intersection point, and the second intersection point, the size information of the obstacle is determined. The size information includes at least one of length, width, height, and center point. The step of determining the size information of the obstacle based on the two endpoints, the first intersection point, and the second intersection point includes: determining the height of the obstacle based on the pixel coordinates of the base point, the world coordinates of the base point, the pixel coordinates of the first intersection point, and the unit vector of the first ray, mathematically expressed as... or, L is the height of the obstacle, (r1,c1) are the pixel coordinates of the base point, (x1,y1,z1) are the world coordinates of the base point, (r3,c3) are the pixel coordinates of the first intersection point, and (u x ,u y ,u z f is the unit vector of the first ray. x and f y This refers to the camera's intrinsic focal length parameter.

2. The method according to claim 1, characterized in that, The step of detecting the image including the pseudo-3D bounding box and the 2D bounding box containing the obstacle to obtain the two endpoints of the bottom edge of the pseudo-3D bounding box, and determining the base point based on the two endpoints, includes: Detect the side bottom edge, front bottom edge, or back bottom edge of the 3D bounding box in the image; If the side bottom edge of the pseudo 3D bounding box is detected, the two endpoints of the side bottom edge of the pseudo 3D bounding box are obtained, and one of the two endpoints is used as the base point; If the side bottom edge of the pseudo 3D bounding box is not detected, obtain the two endpoints of the front bottom edge or the back bottom edge of the pseudo 3D bounding box, and take the midpoint between the two endpoints as the base point.

3. The method according to claim 2, characterized in that, The steps of constructing the first ray and the second ray in the world coordinate system based on the base point include: If the side bottom edge of the pseudo-3D bounding box is detected, the two endpoints are connected to determine the heading of the obstacle; if the side bottom edge of the pseudo-3D bounding box is not detected, the direction of the camera's line of sight or the direction of the lane line of the lane where the obstacle is located is taken as the heading of the obstacle. The world coordinates of the base point are obtained based on the pixel coordinates of the base point on the image. Starting from the world coordinates of the base point, construct a first ray and a second ray, wherein the first ray is perpendicular to the ground where the obstacle is located and points to the top of the obstacle, and the second ray is perpendicular to the heading and the first ray and points to the side of the obstacle.

4. The method according to claim 2, characterized in that, The step of determining the size information of the obstacle based on the two endpoints, the first intersection point, and the second intersection point includes: When the side bottom edge of the pseudo 3D bounding box is detected, the world coordinates of the two endpoints, the first intersection point and the second intersection point are converted according to the pixel coordinates of the two endpoints, the first intersection point and the second intersection point respectively, and the length, width, height and / or center point of the obstacle are determined according to the world coordinates of the two endpoints, the first intersection point and the second intersection point. If the side bottom edge of the pseudo 3D bounding box is not detected, the world coordinates of the midpoint, first intersection, and second intersection of the two endpoints are converted according to the pixel coordinates of the midpoint, first intersection, and second intersection of the two endpoints, respectively, and the width and / or height of the obstacle are determined according to the world coordinates of the midpoint, first intersection, and second intersection of the two endpoints.

5. The method according to claim 2, characterized in that, After determining the size information of the obstacle based on the two endpoints, the first intersection point, and the second intersection point, in the case of detecting the side bottom edge of the pseudo-3D bounding box, the method further includes: Connect the base point to the other endpoint to obtain a first line segment representing the length of the obstacle; Connecting the base point with the first intersection point yields a second line segment representing the height of the obstacle; Connecting the base point and the second intersection point yields a third line segment representing the width of the obstacle; The three-dimensional bounding box of the obstacle is reconstructed based on the first line segment, the second line segment, and the third line segment.

6. The method according to claim 3, characterized in that, The step of using the lane line direction or the camera's line of sight as the heading of the obstacle includes: Detect the distance between the obstacle and the camera; If the distance is less than or equal to the threshold, the direction of the lane line of the lane where the obstacle is located shall be taken as the heading of the obstacle; If the distance is greater than a threshold, the camera's line of sight is used as the heading of the obstacle.

7. A size estimation device, characterized in that, include: The detection unit is used to detect images including pseudo-3D bounding boxes and 2D bounding boxes containing obstacles, so as to obtain the two endpoints of the bottom edge of the pseudo-3D bounding box and determine a base point based on the two endpoints, wherein the base point is one of the two endpoints of the side bottom edge of the 3D bounding box, or the midpoint between the two endpoints of the front bottom edge or the back bottom edge of the 3D bounding box. A construction unit is used to construct a first ray and a second ray in the world coordinate system based on the base point, wherein the first ray and the second ray are respectively used to point in the directions of the height and width of the obstacle; A projection unit is used to project the first ray and the second ray onto the image, such that the projections of the first ray and the second ray intersect the top edge and the side edge of the two-dimensional bounding box at a first intersection point and a second intersection point, respectively. A determining unit is configured to determine the size information of the obstacle based on the two endpoints, the first intersection point, and the second intersection point. The size information includes at least one of length, width, height, and center point. The step of determining the size information of the obstacle based on the two endpoints, the first intersection point, and the second intersection point includes: determining the height of the obstacle based on the pixel coordinates of the base point, the world coordinates of the base point, the pixel coordinates of the first intersection point, and the unit vector of the first ray, mathematically expressed as... or, L is the height of the obstacle, (r1,c1) are the pixel coordinates of the base point, (x1,y1,z1) are the world coordinates of the base point, (r3,c3) are the pixel coordinates of the first intersection point, and (u x ,u y ,u z f is the unit vector of the first ray. x and f y This refers to the camera's intrinsic focal length parameter.

8. A size estimation device, characterized in that, The device includes a processor and a memory connected via a bus; the processor is configured to execute multiple instructions; the memory is configured to store the multiple instructions, which are adapted to be loaded by the processor and executed using the size estimation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor and executing the size estimation method as described in any one of claims 1 to 6.

Images