Ranging and velocity measurement methods based on monocular computer vision

Abstract

This application relates to a distance measurement method and a velocity measurement method based on monocular computer vision. The distance measurement method based on monocular computer vision includes: acquiring a distance-to-be-measured image and using a neural network model based on object detection to determine the bounding box of the object to be measured in the distance-to-be-measured image; determining the coordinates of the distance-measuring pixels of the object to be measured based on the bounding box of the object to be measured in the distance-to-be-measured image; performing a perspective transformation on the coordinates of the distance-measuring pixels to obtain transformed coordinates of the coordinates of the distance-measuring pixels on a bird's-eye view corresponding to the distance-to-be-measured image; determining the scale in the bird's-eye view; and determining the actual distance of the object to be measured based on the transformed coordinates and the scale. Thus, by integrating object detection and perspective transformation and / or coordinate transformation of the source image based on a neural network model, accurate distance and velocity measurement results can be obtained.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a ranging method and a speed measurement method based on monocular computer vision. Background Technology

[0002] For monitoring scenarios with a fixed perspective, it is necessary to identify various targets (such as people, vehicles, etc.), measure the distance between targets (between people, people and vehicles, etc.), and calculate the speed of the targets (people, vehicles, etc.).

[0003] Current monitoring solutions mainly include: monocular, cost ≥ 100 yuan, low cost, stable, and a direct upgrade to existing monitoring systems; binocular, cost ≥ 300 yuan, range of tens of meters, complex algorithms, low real-time performance, and requires replacement of existing systems; speed radar, cost ≥ 4000 yuan, expensive; and TOF (Time of Flight): emits light waves and measures distance and speed based on the echo, similar to radar.

[0004] Since monocular images lack depth information about the scene, depth information needs to be obtained through geometric models of objects or multi-view constraints. Monocular vision-based 3D object detection methods in indoor environments typically require depth estimation, and their detection accuracy is often lower than that of binocular / depth vision algorithms. However, monocular vision algorithms have the advantages of lower sensor cost, wider application range, and the ability to perform 3D object detection with a single sensor, resulting in greater system stability.

[0005] In addition, by using neural network models based on object detection (networks that can outline objects, such as Faster-RCNN, SSD, and YOLO), after training, it is possible to identify and mark the bounding boxes of various objects in a surveillance scene. Figure 1 As shown. Figure 1 The diagram illustrates how a neural network model performs the detection of a predetermined target.

[0006] Based on this, it is desirable to further measure the distance between targets and calculate the speed of the targets. Therefore, this application aims to provide a ranging method and a velocity measurement method based on monocular computer vision. Summary of the Invention

[0007] This application provides a ranging and velocity measurement method based on monocular computer vision. By integrating target detection, perspective transformation and / or coordinate transformation of the source image based on a neural network model, it can obtain accurate distance and velocity measurement results.

[0008] According to one aspect of this application, a ranging method based on monocular computer vision is provided, comprising: acquiring a ranging image to be measured and using a neural network model based on object detection to determine the bounding box of an object to be measured in the ranging image; determining the coordinates of a ranging pixel of the object to be measured based on the bounding box of the object to be measured in the ranging image; performing a perspective transformation on the coordinates of the ranging pixel to obtain the transformed coordinates of the coordinates of the ranging pixel on a bird's-eye view corresponding to the ranging image to be measured; determining the scale in the bird's-eye view; and determining the actual distance of the object to be measured based on the transformed coordinates and the scale.

[0009] In the above-mentioned ranging method based on monocular computer vision, performing perspective transformation on the coordinates of the ranging pixel to obtain the transformed coordinates of the ranging pixel on the bird's-eye view corresponding to the image to be measured includes: determining the transformation matrix between the image to be measured and the bird's-eye view, wherein the transformation matrix is ​​the transformation matrix between the homogeneous coordinates of the image to be measured and the homogeneous coordinates of the bird's-eye view, expressed as:

[0010]

[0011] Where (u,v,w) are the homogeneous coordinates of the image to be measured, and (x... ′ ,y ′ (w′) are the homogeneous coordinates of the bird's-eye view; and the transformed coordinates are obtained from the coordinates of the ranging pixel based on the transformation matrix, expressed as:

[0012]

[0013]

[0014] In the above-mentioned ranging method based on monocular computer vision, the transformation matrix is ​​determined by prior information of multiple corresponding coordinates in the image to be measured and the bird's-eye view.

[0015] In the above-mentioned ranging method based on monocular computer vision, determining the scale ratio in the bird's-eye view includes: determining the reference distance in the reference object in the bird's-eye view and the pixel distance corresponding to the reference distance; and determining the scale ratio based on the ratio between the reference distance and the pixel distance.

[0016] According to another aspect of this application, a ranging method based on monocular computer vision is provided, comprising: acquiring a ranging image and using a neural network model based on object detection to determine the bounding box of an object to be measured in the ranging image; determining the coordinates of a ranging pixel of the object to be measured based on the bounding box of the object to be measured in the ranging image; performing a coordinate transformation on the coordinates of the ranging pixel to transform the coordinates of the ranging pixel from an image pixel coordinate system to a world coordinate system; and determining the distance between the objects to be measured based on the world coordinates of the object points corresponding to the ranging pixels in the world coordinate system.

[0017] In the above-described ranging method based on monocular computer vision, transforming the coordinates of the ranging pixel from the image pixel coordinate system to the world coordinate system includes: determining the relationship between the imaging plane coordinate system and the image pixel coordinate system based on the camera DPI; determining the imaging plane coordinate system based on the camera coordinate system and the camera's focal length; determining the altitude, pitch angle, yaw angle, and roll angle corresponding to the camera coordinate system; determining the coordinates of the imaging plane center in the world coordinate system based on the camera normal vector in the world coordinate system; determining the image point world coordinates in the world coordinate system from the image point pixel coordinates in the image pixel coordinate system based on the altitude, pitch angle, yaw angle, roll angle, and the image plane center coordinates in the world coordinate system; and determining the object point world coordinates corresponding to the coordinates of the ranging pixel in the world coordinate system based on the image point world coordinates in the world coordinate system and the camera optical center coordinates.

[0018] In the above-mentioned ranging method based on monocular computer vision, the coordinate transformation of the ranging pixel coordinates to convert the coordinates of the ranging pixel from the image pixel coordinate system to the world coordinate system includes: transforming the coordinates of the ranging pixel from the image pixel coordinate system to the world coordinate system based on the camera's height, horizontal field of view, vertical field of view, and pitch angle.

[0019] According to another aspect of this application, a speed measurement method based on monocular computer vision is provided, comprising: acquiring a first image at a first moment and a second image at a second moment; obtaining the distance change of a speed-to-be-measured object in the first image and the second image based on the distance measurement method based on monocular computer vision as described above; and determining the speed of the speed-to-be-measured object based on the distance change of the speed-to-be-measured object and the time difference between the first moment and the second moment.

[0020] In the above-mentioned speed measurement method based on monocular computer vision, the speed-to-be-measured object includes a first speed-to-be-measured object and a second speed-to-be-measured object, and obtaining the distance change of the speed-to-be-measured object in the first image and the second image based on the above-mentioned distance measurement method based on monocular computer vision includes: matching the same speed-to-be-measured object in the first image and the second image.

[0021] The distance and velocity measurement methods based on monocular computer vision provided in this application can obtain accurate distance and velocity measurement results by integrating target detection, perspective transformation and / or coordinate transformation of the source image based on a neural network model. Attached Figure Description

[0022] Various other advantages and benefits of this application will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiments below. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0023] Figure 1 The diagram illustrates how a neural network model performs the detection of a predetermined target.

[0024] Figure 2 The diagram illustrates how a neural network model is used to detect and determine pixel distances for a predetermined target.

[0025] Figure 3 The diagram illustrates how a neural network model is used to detect and determine the actual distance to a predetermined target.

[0026] Figure 4 The illustration shows a schematic flowchart of a first example of a ranging method based on monocular computer vision according to an embodiment of this application.

[0027] Figure 5 The illustration shows a top-view image of the distance to be measured according to an embodiment of this application.

[0028] Figure 6 The illustration shows a bird's-eye view corresponding to the distance measurement image from a top-down perspective according to an embodiment of this application.

[0029] Figure 7 The illustration shows a schematic diagram of a floor tile object in a surveillance scene from a top-down perspective.

[0030] Figure 8 The illustration shows a schematic diagram of the floor tile object within the corresponding monitoring scene in the bird's-eye view.

[0031] Figure 9 The illustration shows a schematic flowchart of a second example of a ranging method based on monocular computer vision according to an embodiment of this application.

[0032] Figure 10 The diagram illustrates a coordinate transformation based on the field of view.

[0033] Figure 11 The diagram illustrates the vertical field of view during camera shooting.

[0034] Figure 12 The illustration shows a schematic flowchart of a speed measurement method based on monocular computer vision according to an embodiment of this application. Detailed Implementation

[0035] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0036] Schematic distance measurement method

[0037] As mentioned above, current neural network models based on object detection can identify predetermined targets in an input image and return the coordinate information of the bounding box of the predetermined targets (i.e., the pixel coordinates of the center point and the four vertices, or the center point coordinates, width, height, etc. in pixels). Therefore, the pixel distance between predetermined targets can be calculated using the distance between their center points (any point within the bounding box, including the edges). Figure 2 As shown, the distance between the person and the vehicle is 739 pixels. Here, Figure 2 The diagram illustrates how a neural network model is used to detect and determine pixel distances for a predetermined target.

[0038] In this scenario, to complete the actual distance measurement, it is necessary to locate the predetermined target in the image using bounding boxes, calculate the pixel distance between the targets using the coordinate information of the bounding boxes, and then convert the pixel distance into the actual distance. This allows for the construction of a ranging solution that integrates target detection and ranging, such as... Figure 3 As shown, the actual distance between the person and the vehicle is 46.54m. Here, Figure 3 The diagram illustrates how a neural network model is used to detect and determine the actual distance to a predetermined target.

[0039] For calculating actual distance based on pixel distance in an image, there are two classic methods: one is perspective transformation, which is used when measuring based on a scale (meters / pixels). In large areas of a surveillance scene, the scales at different locations in the image are different. For example, the scale of nearby pixels is smaller than that of distant pixels, causing a visual effect of near objects appearing larger than distant ones. Perspective transformation is needed to transform the image into a visual effect with uniform near and far distances. The second method is coordinate transformation, which requires establishing the relationship between the pixel coordinate system of the image and the real-world coordinate system, and then using coordinate transformation to solve the problem.

[0040] Based on this, embodiments of this application provide a ranging method based on monocular computer vision. Figure 4 The illustration shows a schematic flowchart of a first example of a ranging method based on monocular computer vision according to an embodiment of this application.

[0041] like Figure 4 As shown, a first example of a ranging method based on monocular computer vision according to an embodiment of this application includes the following steps.

[0042] Step S110: Acquire the image to be measured and use a neural network model based on object detection to determine the bounding box of the object to be measured in the image. That is, as described above, obtain the image to be measured, for example, in a surveillance scene, and use a neural network model based on object detection to determine the bounding box of the object to be measured, such as a person or a vehicle.

[0043] Step S120: Determine the coordinates of the ranging pixels of the object to be measured based on the bounding box of the object in the image to be measured. For example, the pixel of the center point of the bottom edge of the bounding box of the object to be measured can be selected as a pair of ranging pixels of the object.

[0044] Step S130: Perform a perspective transformation on the coordinates of the ranging pixel to obtain the transformed coordinates of the ranging pixel on the bird's-eye view corresponding to the image to be measured. Here, perspective transformation is also called perspective projection, which essentially projects an image onto a new viewing plane, also known as projection mapping. Figure 4 As shown. Here, Figure 4 The illustration shows a perspective transformation diagram according to an embodiment of this application.

[0045] In this embodiment of the application, perspective transformation is used to transform the image of the distance to be measured from the original viewpoint (e.g., as shown in the image). Figure 5 The top view shown is processed and converted into a bird's-eye view (e.g., as shown in the image). Figure 6 As shown in the figure, this corrects the unevenness of the scale that occurs during distance calculation. Here, Figure 5 The illustration shows a top-view image of the distance to be measured according to an embodiment of this application, and Figure 6 The illustration shows a bird's-eye view corresponding to the distance measurement image from a top-down perspective according to an embodiment of this application.

[0046] Specifically, to obtain a new bird's-eye view from the image to be measured (e.g., a top view), the following processing is required using a transformation matrix:

[0047]

[0048] Where (u,v,w) are the homogeneous coordinates (w=1) of the image to be measured, and (x... ′ ,y ′ Let w′ be the homogeneous coordinates of the new bird's-eye view, and the transformation matrix is:

[0049]

[0050] The transformation matrix A can be obtained, for example, through prior information, i.e., in the image to be measured (such as...). Figure 5 Select the coordinates of four points in the top view shown, and set the coordinates of these four points after perspective transformation (they need to be consistent with the scale of the real object, and the rest can be set freely) to obtain the transformation matrix A.

[0051] Based on this transformation matrix A, the coordinates of any point in the image to be measured can be transformed to form a new bird's-eye view.

[0052]

[0053]

[0054] Here, in the new bird's-eye view, the pixel scale is uniform everywhere, for example, as shown in the image. Figure 6 As shown, this way, when calculating distances, there will be no effect from the scale being larger when it is closer and smaller when it is farther away, and the change in distance will be uniform.

[0055] Therefore, in the monocular computer vision-based ranging method according to the embodiments of this application, performing perspective transformation on the coordinates of the ranging pixel to obtain the transformed coordinates of the ranging pixel on the bird's-eye view corresponding to the image to be measured includes: determining the transformation matrix between the image to be measured and the bird's-eye view, wherein the transformation matrix is ​​the transformation matrix between the homogeneous coordinates of the image to be measured and the homogeneous coordinates of the bird's-eye view, expressed as:

[0056]

[0057] Where (u,v,w) are the homogeneous coordinates of the image to be measured, and (x... ′ ,y ′ (w′) are the homogeneous coordinates of the bird's-eye view; and

[0058] The transformed coordinates are obtained from the coordinates of the ranging pixel based on the transformation matrix, and are expressed as follows:

[0059]

[0060]

[0061] Furthermore, in the above-mentioned ranging method based on monocular computer vision, the transformation matrix is ​​determined by prior information of multiple corresponding coordinates in the image to be measured and the bird's-eye view.

[0062] Step S140: Determine the scale ratio in the bird's-eye view. That is, determine how much actual distance a single pixel of the scale in the bird's-eye view represents, which can be obtained, for example, by having a target of actual size in the image to be measured. Figure 7 As shown, there is a white square floor tile with a side length of 80cm. Here, Figure 7 The illustration shows a schematic diagram of a floor tile object in a surveillance scene from a top-down perspective.

[0063] However, due to the perspective angle, the floor tiles in the original image are not square; that is, the actual size of each pixel varies in different locations, and the scale is not uniform. In the new bird's-eye view obtained after perspective transformation, the floor tiles should be standard squares, as shown below. Figure 8 As shown. Here, Figure 8 The illustration shows a schematic diagram of the floor tile object within the corresponding monitoring scene in the bird's-eye view.

[0064] In this way, by obtaining the coordinates of the four vertices of a reference object, such as a white floor tile, on the image to be measured, and setting the coordinates of the four vertices of the floor tile on the new bird's-eye view (for example, it only needs to be a square scale, and the rest can be set freely), the image to be measured is transformed into a bird's-eye view based on the transformation matrix A mentioned above. Since the scale in the bird's-eye view is uniform in all places, the scale ratio can be calculated by using the reference distance of the reference object, such as the side length of the white floor tile (80cm) and the pixel distance corresponding to the reference distance, such as the number of pixels occupied by the side length (100 pixels). For example, it is 0.8cm / pixel.

[0065] Of course, those skilled in the art will understand that a reference object can be directly selected in the bird's-eye view, and the scale ratio can be determined based on the ratio between the pixel distance and the reference distance.

[0066] That is, in the ranging method based on monocular computer vision according to the embodiments of this application, determining the scale ratio in the bird's-eye view includes: determining the reference distance in the reference object in the bird's-eye view and the pixel distance corresponding to the reference distance; and determining the scale ratio based on the ratio between the reference distance and the pixel distance.

[0067] Step S150: Determine the actual distance of the object to be measured based on the transformed coordinates and the scale ratio. That is, using the aforementioned scale, the actual distance between any two points in the bird's-eye view can be calculated by multiplying the pixel distance between the two points by the scale.

[0068] Figure 9 The illustration shows a schematic flowchart of a second example of a ranging method based on monocular computer vision according to an embodiment of this application.

[0069] like Figure 9 As shown, a second example of a ranging method based on monocular computer vision according to an embodiment of this application includes the following steps.

[0070] Step S210: Acquire the image to be measured and use a neural network model based on object detection to determine the bounding box of the object to be measured in the image. That is, as described above, obtain the image to be measured, for example, in a surveillance scene, and use a neural network model based on object detection to determine the bounding box of the object to be measured, such as a person or a vehicle.

[0071] Step S220: Determine the coordinates of the ranging pixels of the object to be measured based on the bounding box of the object in the image to be measured. For example, the pixel of the center point of the bottom edge of the bounding box of the object to be measured can be selected as a pair of ranging pixels of the object.

[0072] Step S230: Perform coordinate transformation on the coordinates of the ranging pixel to convert the coordinates of the ranging pixel from the image pixel coordinate system to the world coordinate system.

[0073] Here, the coordinate transformation utilizes the pinhole imaging principle of a camera to convert the image to be measured from the image pixel coordinate system to the world coordinate system. Generally, the world coordinate system is the camera coordinate system (C-XYZ), with the camera's optical center C as its origin (also the projection center O). The Z-axis coincides with the optical axis and points in front of the camera. The coordinates of a point in the real world in the camera coordinate system are, for example, (X, Y, Z). Furthermore, the imaging plane coordinate system (i.e., the physical image coordinate system) can be obtained from the camera coordinate system. Specifically, the Z-axis of the camera coordinate system is perpendicular to the image plane. The origin of the imaging plane is the intersection of the camera's optical axis and the physical image coordinate system. This coordinate system is denoted as o-xyz, and its Z-axis coincides with the Z-axis of the camera coordinate system, but differs by a focal length f. Therefore, the projection of a point with coordinates (X, Y, Z) in the camera coordinate system onto the image plane lies on the image plane, which is located between the camera's optical center C and this point. The distance from C is the focal length f. Therefore, the coordinates of this point in the imaging plane coordinate system are (x, y, z) = (X, Y, Zf), which is the relationship between the two coordinate systems.

[0074] Image pixel coordinate system: The imaging plane coordinate system is the physical coordinate system of the image, that is, (x, y, z) are physical coordinates, expressed in meters. The image is measured in pixels, based on the image pixel coordinate system, with coordinates (u, v, w) also expressed in pixels. The relationship between the two is as follows:

[0075]

[0076] DPM (Dots Per Meter) is a camera intrinsic parameter that represents the number of pixels per meter of the image on the camera target surface. It can be obtained from the camera intrinsic parameter DPI (Dots Per Inch), for example, DPM = DPI / 0.0254.

[0077] For example, in a surveillance scenario, the camera is at a height h looking down at the ground, meaning the optical center is located at (h, 0, 0).

[0078] At this point, establishing the relationship between the imaging plane coordinate system (x, y, z) and the world coordinate system (O-XYZ) requires considering the factors caused by rotation and performing coordinate transformations, i.e., increasing the pitch angle (rotation around the y-axis). ), yaw angle (rotation θ about the x-axis) and roll angle (rotation γ about the z-axis).

[0079] In the world coordinate system, the camera was originally facing the z-axis, i.e., the normal vector was (0, 0, 1). After a pitch angle (rotation around the y-axis)... By rotating the yaw angle (θ about the x-axis) and the roll angle (γ about the z-axis), we can obtain:

[0080] Pitch angle (rotation about the y-axis) ):

[0081] Yaw angle (rotation θ about the x-axis):

[0082] Roll angle (rotation γ about the z-axis):

[0083] The camera normal vector (0,0,1) is transformed into the following vector:

[0084]

[0085] We can obtain the coordinates (T) of the center of the imaging plane in the world coordinate system. x ,T y ,T z )for:

[0086]

[0087] Notation:

[0088] T = (T x ,T y ,T z ) T

[0089] Therefore, combining the three-axis rotation and the change of the coordinate center, the coordinate transformation relationship from the world coordinate system to the image plane coordinate system of the monitored scene can be obtained as follows:

[0090]

[0091] The actual coordinates in the world coordinate system can then be derived from the image plane coordinates:

[0092]

[0093] Among them, the planar coordinates (x, y, z) are physical coordinates, which are derived from the image pixel coordinate system (u, v, w), and their simplified relationship is as follows (in actual situations, adjustments need to be made to consider the center and the image pixel width and height values):

[0094]

[0095] We can obtain:

[0096]

[0097] DPM (Dots Per Meter) is the number of pixels per meter of the camera target surface, which is obtained from the camera's intrinsic parameter DPI (Dots Per Inch), i.e., DPM = DPI / 0.0254.

[0098] The right side of the equation represents the coordinates of a point on the image plane in the image pixel coordinate system, while the left side represents the coordinates of the point on the image plane in the world coordinate system. This equation shows that, given the pixel coordinates (u, v, w) of a point on the image plane, its coordinates (X, Y, Z) in the world coordinate system can be determined.

[0099] The camera's optical center coordinates are (h, 0, 0). The intersection of the ray from the optical center to a point (X, Y, Z) on the image plane and the OYZ plane is the object's physical coordinate point (object point) in the world coordinate system.

[0100]

[0101] In summary, we first obtain the physical coordinates of any point on the image plane (i.e., the image point) in the image pixel coordinate system from its pixel coordinates. Then, we obtain the coordinates of the image point in the world coordinate system using coordinate transformation. Finally, we obtain the coordinates of the object point in the world coordinate system from the equation of the line connecting the image point and the optical center. This process only requires knowing the camera's intrinsic focal length f and extrinsic parameters height h and pitch angle. The yaw angle θ and roll angle γ are sufficient.

[0102] Therefore, in the ranging method based on monocular computer vision according to the embodiments of this application, transforming the coordinates of the ranging pixel from the image pixel coordinate system to the world coordinate system includes: determining the relationship between the imaging plane coordinate system and the image pixel coordinate system based on the camera DPI (Dots Per Inch); determining the imaging plane coordinate system based on the camera coordinate system and the camera's focal length; determining the altitude, pitch angle, yaw angle, and roll angle corresponding to the camera coordinate system; determining the coordinates of the center of the imaging plane in the world coordinate system based on the vector of the camera normal vector in the world coordinate system; determining the world coordinates of the image point in the world coordinate system from the image point coordinates in the image pixel coordinate system based on the altitude, pitch angle, yaw angle, roll angle, and the coordinates of the center of the imaging plane in the world coordinate system; and determining the world coordinates of the object point corresponding to the coordinates of the ranging pixel in the world coordinate system based on the world coordinates of the image point and the camera optical center coordinates.

[0103] Step S240: Determine the distance between the objects to be measured based on the world coordinates of the ranging pixels in the world coordinate system. That is, after determining the world coordinates of the ranging pixels in the world coordinate system, the distance of the objects to be measured in the world coordinate system can be determined, i.e., the distance of the objects to be measured in the real physical world.

[0104] In an alternative example, given that the camera's field of view is known, the pixel coordinates can be transformed from the camera coordinate system to the world coordinate system based on the camera's field of view. Specifically:

[0105] like Figure 10 As shown in (a), plane ABU represents the ground, ABCD is the trapezoidal region on the road plane captured by the camera, point O is the center point of the camera lens, OG is the camera optical axis, point G is the intersection of the camera optical axis and the ground (which is also the intersection of the diagonals of the field of view trapezoid), point I is the vertical projection of point O onto the ground, and the height of point O above the ground plane is h. Point G is defined as the origin of the coordinate system, and the direction extending towards the distance relative to the camera is defined as the Y-axis, thus constructing the GXY real coordinate system. The imaging plane is as follows... Figure 10 As shown in (b), a, b, c, and d are the four endpoints of the image plane rectangle, and H and W are the height and width of the image plane (either true values ​​or pixel values ​​are acceptable). g is the image point of G, and gxy constitutes the image plane coordinate system. Here, Figure 10 The diagram illustrates a coordinate transformation based on the field of view.

[0106] Let the camera's horizontal field of view be 2β0, its vertical field of view be 2α0, and its pitch angle be γ0. The vertical field of view is as follows: Figure 11 As shown. Here, Figure 11 The diagram illustrates the vertical field of view during camera shooting.

[0107] Take a point P on the ground, and its coordinates in the world coordinate system are (X... p ,Y p Point P's corresponding point in the image plane is p, and its coordinates in the image plane coordinate system are (x, y). p ,x p This coordinate value can be a physical coordinate value or a pixel coordinate value. The relationship between these two coordinate systems is as follows:

[0108]

[0109] in,

[0110]

[0111] Based on this transformation formula, the pixel coordinates (x, y) of the image pixel coordinate system can be transformed. p ,x p ) transform into (X p ,Y p All that is needed are the camera height h, horizontal field of view 2β0, vertical field of view 2α0, and pitch angle γ0.

[0112] Therefore, in the ranging method based on monocular computer vision according to the embodiments of this application, transforming the coordinates of the ranging pixel from the image pixel coordinate system to the world coordinate system includes: transforming the coordinates of the ranging pixel from the camera coordinate system to the world coordinate system based on the camera's height, horizontal field of view, vertical field of view, and pitch angle.

[0113] Indicative speed measurement method

[0114] By using the ranging method based on monocular computer vision according to the embodiments of this application as described above, the speed measurement method based on monocular computer vision according to the embodiments of this application can be obtained.

[0115] Specifically, during speed measurement, a frame of image at time t is first acquired, and a neural network model (e.g., YOLOv4) is used to detect people and vehicles, obtaining the coordinates of the ranging pixels used for distance measurement (e.g., the coordinates of the center of the bottom edge of the rectangle). Then, a frame of image at time t+1 is acquired, and people and vehicles are detected in the same way, obtaining the coordinates of the ranging pixels used for distance measurement. The distance traveled by people and vehicles from time t to time t+1 is calculated using the previous distance measurement method. This distance is divided by the time interval between the two frames to obtain the speed of people and vehicles.

[0116] Furthermore, in practice, the algorithm also needs to match the same object in two frames. This is done by obtaining the pixel coordinates of objects with the same label in the two frames to calculate the position difference. If this difference is below a certain value, the two positions are considered to be the same object in the two frames, and the object's velocity is then calculated.

[0117] Based on this, the speed measurement method based on monocular computer vision according to the embodiments of this application includes: acquiring a first image at a first moment and a second image at a second moment; obtaining the distance change of the object to be measured in the first image and the second image based on the distance measurement method based on monocular computer vision as described above; and determining the speed of the object to be measured based on the distance change of the object to be measured and the time difference between the first moment and the second moment.

[0118] Figure 12 A schematic flowchart illustrating a speed measurement method based on monocular computer vision according to an embodiment of this application is shown. Figure 12As shown, the speed measurement method based on monocular computer vision according to an embodiment of this application includes: S310, acquiring a first image at a first moment and a second image at a second moment; S320, obtaining the distance change of the object to be measured in the first image and the second image through the distance measurement method based on monocular computer vision as described above; and S330, determining the speed of the object to be measured based on the distance change of the object to be measured and the time difference between the first moment and the second moment.

[0119] Furthermore, in the above-mentioned speed measurement method based on monocular computer vision, the speed-to-be-measured object includes a first speed-to-be-measured object and a second speed-to-be-measured object, and obtaining the distance change of the speed-to-be-measured object in the first image and the second image based on the above-mentioned distance measurement method based on monocular computer vision includes: matching the same speed-to-be-measured object in the first image and the second image.

[0120] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0121] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0122] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0123] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0124] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A ranging method based on monocular computer vision, characterized in that, include: Acquire an image to be measured and use a neural network model based on object detection to determine the bounding box of the object to be measured in the image; The coordinates of the ranging pixels of the object to be measured are determined based on the bounding box of the object to be measured in the image to be measured. A perspective transformation is performed on the coordinates of the ranging pixel to obtain the transformed coordinates of the ranging pixel on the bird's-eye view corresponding to the image to be measured; Determine the scale of the bird's-eye view; as well as The actual distance to the object to be measured is determined based on the transformed coordinates and the scale ratio. Performing a perspective transformation on the coordinates of the ranging pixel to obtain the transformed coordinates of the ranging pixel on the bird's-eye view corresponding to the image to be ranging includes: Determine the transformation matrix between the image to be measured and the bird's-eye view. The transformation matrix is ​​the transformation matrix between the homogeneous coordinates of the image to be measured and the homogeneous coordinates of the bird's-eye view, expressed as: ； in These are the homogeneous coordinates of the image to be measured, and These are the homogeneous coordinates of the aforementioned bird's-eye view; as well as The transformed coordinates are obtained from the coordinates of the ranging pixel based on the transformation matrix, and are expressed as follows: ； 2. The ranging method based on monocular computer vision according to claim 1, wherein, The transformation matrix is ​​determined using prior information from multiple corresponding coordinates in the image to be measured and the bird's-eye view.

3. The ranging method based on monocular computer vision according to claim 1, wherein, Determining the scale in the bird's-eye view includes: Determine the reference distance in the reference object in the bird's-eye view and the pixel distance corresponding to the reference distance; and, The scale ratio is determined based on the ratio between the reference distance and the pixel distance.

4. A ranging method based on monocular computer vision, characterized in that, include: Acquire an image to be measured and use a neural network model based on object detection to determine the bounding box of the object to be measured in the image; The coordinates of the ranging pixels of the object to be measured are determined based on the bounding box of the object to be measured in the image to be measured. The coordinates of the ranging pixels are transformed from the image pixel coordinate system to the world coordinate system; and The distance between the objects to be measured is determined based on the world coordinates of the object points corresponding to the ranging pixels in the world coordinate system. Transforming the coordinates of the ranging pixels from the image pixel coordinate system to the world coordinate system includes: The relationship between the imaging plane coordinate system and the image pixel coordinate system is determined based on the camera DPI (Dots Per Inch). The imaging plane coordinate system is determined based on the camera coordinate system and the focal length of the camera; Determine the altitude, pitch angle, yaw angle, and roll angle corresponding to the camera coordinate system; The coordinates of the center of the imaging plane in the world coordinate system are determined based on the vector of the camera normal vector in the world coordinate system. Based on the altitude, pitch angle, yaw angle, roll angle, and the coordinates of the center of the imaging plane in the world coordinate system, the world coordinates in the world coordinate system are determined from the imaging plane coordinates in the imaging plane coordinate system; and... The world coordinates of the object point corresponding to the coordinates of the ranging pixel in the world coordinate system are determined based on the world coordinates in the world coordinate system and the camera optical center coordinates.

5. The ranging method based on monocular computer vision according to claim 4, wherein, Transforming the coordinates of the ranging pixels from the image pixel coordinate system to the world coordinate system includes: The coordinates of the ranging pixels are transformed from the image pixel coordinate system to the world coordinate system based on the camera's height, horizontal field of view, vertical field of view, and pitch angle.

6. A speed measurement method based on monocular computer vision, characterized in that, include: Acquire the first image at the first time step and the second image at the second time step; The distance change of the speed-to-measure object in the first image and the second image is obtained based on the ranging method based on monocular computer vision as described in any one of claims 1 to 5. as well as, The speed of the object to be measured is determined based on the change in distance of the object and the time difference between the first moment and the second moment.

7. The speed measurement method based on monocular computer vision according to claim 6, wherein, The speed-to-be-measured objects include a first speed-to-be-measured object and a second speed-to-be-measured object, and the distance change of the speed-to-be-measured objects in the first image and the second image is obtained based on the distance measurement method based on monocular computer vision as described above, including: Match the same speed-to-be-measured object in the first image and the second image.

Images