A method for estimating the height of an object in motion based on a monocular camera and optical flow velocity measurement

Abstract

The present invention relates to a method for estimating the height of an object based on a monocular camera and optical flow velocity measurement in a moving state, aiming to improve the real-time and convenience of the height measurement method. The method first calibrates the internal parameters and distortion coefficients of the monocular camera, performs distortion correction on the captured image to eliminate the distortion effect during imaging, and then uses the recognition and tracking algorithm to locate the target, and then selects the pixel position of the target feature point, calculates the optical flow of the feature point according to the optical flow method, and obtains the optical flow velocity in combination with the time interval between two frames, and then based on the acquired camera motion information and the camera's internal parameters, a reference plane is set to determine the reference optical flow velocity of the reference plane, and finally the height of the object relative to the reference plane is calculated through the perspective relationship. The present invention provides a solution to extract the height information of the target in the planar motion video of the monocular lens, which is suitable for a variety of application scenarios such as robot navigation, laser weeding, and industrial automation.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and intelligent measurement, and specifically is a method for estimating the height of an object in motion based on a monocular camera and optical flow velocity measurement. Background Art

[0002] Extracting depth information (also called height information in this article) is an important task in the field of computer vision. It involves obtaining distance information of different objects in a scene from an image or video, and has important applications in autonomous driving, robot navigation, 3D reconstruction, laser weeding, etc. Although the image and video data obtained by the camera contain rich information, extracting depth information from these data is a challenging task.

[0003] Traditional methods for measuring depth mainly include binocular vision, laser radar, etc. Although these methods can provide accurate measurement results, the equipment cost is high and the operation is complicated. Binocular stereo vision is a depth measurement method that estimates depth and distance by comparing two images from different perspectives. This method can provide accurate depth information, but it requires two precisely calibrated cameras, the equipment cost is high, and the method has high requirements for the installation consistency of the two cameras, otherwise it will affect the accuracy of depth detection. Especially in dynamic environments, bumps may cause the position and angle between the two cameras to change, reducing consistency. Binocular vision requires complex matching algorithm support, real-time performance will be affected to a certain extent, and the requirements for algorithms and equipment are relatively high. In practical applications, factors such as the space occupation of precision equipment may also need to be considered. These factors will affect the performance and application scope of the binocular vision system. The laser radar ranging method uses electromagnetic waves to measure distance. By transmitting and receiving electromagnetic waves and measuring the round-trip time of electromagnetic waves, the distance can be calculated. This method can provide very accurate depth and distance information and is suitable for a variety of complex scenes, but it requires specialized equipment and has a certain equipment cost. The laser radar equipment is large in size and is not suitable for portable devices. There is a TOF (Time of Flight) depth camera, the basic principle of which is to continuously emit light pulses to the observed object, then receive the light pulses reflected from the object, and calculate the distance of the measured object from the camera by detecting the flight time of the light pulses. TOF depth cameras have high requirements for the accuracy of time measurement. Due to the limitations of current electronic components, the accuracy of TOF depth cameras is not high enough in the field of close-range measurement.

[0004] Monocular cameras have only one perspective and cannot directly measure depth information, but the development of deep learning technology in recent years has provided new possibilities for monocular vision. By training a large number of samples to learn depth information, deep learning methods can use powerful feature extraction capabilities to automatically learn feature expressions in images, thereby effectively improving the accuracy of depth estimation. However, the accuracy of this method is affected by many factors, including the quality and quantity of training data, the choice of network structure, and the optimization strategy during training. In addition, deep learning models are prone to overfitting the training data, resulting in decreased performance on the test set. This means that if the scenes in the training data set are very different from the test scenes, the performance of the model may decline. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] This method uses image data captured by a monocular camera to locate the target based on recognition and tracking algorithms, measures the optical flow velocity of target feature points through the optical flow method, obtains the optical flow of the reference plane based on the camera's motion state and the camera's internal parameters, and finally estimates the height of the moving object through the perspective relationship of the optical flow.

[0007] A method for estimating the height of an object in motion based on a monocular camera and optical flow velocity measurement:

[0008] The following steps are involved:

[0009] S1: Camera calibration and distortion correction: Calibrate the camera's internal parameters and distortion coefficients, and perform distortion correction on the image;

[0010] S2: Image frame acquisition: The camera acquires two adjacent image frames frame_t and frame_t_pre, records the corresponding time t and t_pre of the two frames, and calculates the time difference dt=t-t_pre between the two frames;

[0011] S3: Acquisition of target optical flow: For the two frames of images frame_t and frame_t_pre acquired by the camera, the pixel position of the target on the image frame_t_pre is confirmed by the target recognition and tracking algorithm, and the feature point O(x, y) is selected. Then, the optical flow method of OpenCV is used to track the feature point O_next (x_next, y_next) of the target in the next frame frame_t, and the optical flow flow_x and flow_y of the feature point pixels in the horizontal x and vertical y directions are obtained, and the optical flow speed is calculated:

[0012] S4: Obtaining the camera motion speed: obtaining the camera motion speed v_cam through the motion sensor;

[0013] S5: Obtaining the average optical flow speed and average camera speed: Repeat the above S2, S3 and S4 processes, take N groups of data, calculate the average optical flow speed and camera motion speed, and obtain the average optical flow speed V_flow_x, V_flow_y of the feature points and the average speed V_cam of the camera;

[0014] S6: Set the reference plane, the distance between the reference plane and the optical center of the camera is D, and use the camera's internal parameters to calculate the optical flow velocity V_flow_base of the reference plane. The process is as follows: the direction of the camera is horizontal u and vertical v, which are consistent with the x and y directions of the image respectively. The average speed of the camera V_cam is decomposed into the average optical flow velocity in the horizontal u and vertical v directions as V_cam_u and V_cam_v. Combined with the camera's internal parameters: focal length (f_u, f_v), the base optical flow velocity in the horizontal u and vertical v directions is obtained: Finally, the optical flow velocity of the synthetic reference plane is:

[0015] S7: Calculate the target height: Combine the optical flow velocities of the target in the horizontal and vertical directions to obtain the optical flow velocity of the target: Then, according to the perspective relationship, the target optical flow and the reference optical flow are combined to calculate the height of the target center point relative to the reference plane.

[0016] Preferably, in step S1, the image is distorted and corrected using an OpenCV algorithm.

[0017] A non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by a robot, the robot executes any of the methods described above.

[0018] A robot performs any of the methods described above.

[0019] Compared with the prior art, the beneficial effects achieved by the present invention are: the method can obtain height information in real time, and does not require complicated equipment and operations, and does not require precisely calibrated cameras or special equipment, so the equipment cost is low and it is very convenient to use. In addition, the monocular camera is small in size and is suitable for portable devices and resource-constrained application scenarios. The present method can handle motion cameras or objects that move relative to the camera, can meet the needs of specific scenarios, and has broad application prospects. In general, the method of the present invention provides a real-time, convenient, low-cost, small-volume, and highly adaptable height measurement solution, which has great application potential in fields such as robot navigation, laser weeding, and industrial automation.

[0020] The depth measurement method based on monocular computer vision and optical flow method in a moving state involved in the present invention provides a solution for extracting depth information of moving objects in ordinary monocular plane videos. This method can only process moving objects, can meet the needs of specific scenarios, and has broad application prospects. The main advantages of this method are simple structure, low cost, small size, and the ability to obtain height information in real time, which is suitable for application scenarios with high real-time requirements. The present invention effectively overcomes the shortcomings of existing binocular vision and laser radar in depth measurement, and provides an efficient, low-cost, portable height measurement method with broad application prospects.

[0021] In this article, "depth" and "height" will be used interchangeably to refer to the vertical distance information of an object. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0023] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0024] Figure 2 The schematic diagram for calculating the target height based on optical flow and perspective relationship;

[0025] Figure 3 The schematic diagram shows the principle of calculating the reference optical flow speed based on the camera motion speed and internal parameters. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The objective of the present invention is to solve how to obtain the pixel position of the target on the image through recognition and tracking algorithms in the image acquired by a monocular camera in a moving state, and then use the optical flow method to calculate the corresponding optical flow velocity of the target center pixel, and then combine the camera's movement speed and the camera's internal parameters with the reference plane to calculate the reference plane optical flow velocity, and then use the perspective relationship to calculate the height of the target center point relative to the reference plane.

[0028] A method for estimating the height of an object in motion based on a monocular camera and optical flow velocity measurement. The method first performs distortion correction on the image taken by the monocular camera to eliminate the distortion effect during imaging, then uses the recognition and tracking algorithm to locate the target, then selects the pixel position of the target feature point, calculates the optical flow of the feature point according to the optical flow method, and obtains the optical flow velocity by combining the time interval between two frames, then based on the obtained camera motion information and the camera's internal parameters, a reference plane is set to determine the reference optical flow velocity of the reference plane, and finally the height of the object is calculated through the perspective relationship.

[0029] The present invention comprises the following steps:

[0030] 1. Camera calibration and distortion correction: Use OpenCV algorithm to calibrate the camera internal parameters and correct the image distortion;

[0031] 2. Image acquisition: The camera acquires two adjacent frames of images and records the time points t and t_pre, and calculates the time difference dt between the two frames;

[0032] 3. Image processing and target tracking: Use recognition and tracking algorithms to locate the target position and select the target's feature points. Then use OpenCV's optical flow method to track the target feature points, obtain the optical flow flow_x and flow_y in the horizontal (x) and vertical (y) directions, and calculate the optical flow speed:

[0033] 4. Get the camera motion speed: Get the camera motion speed v_cam through the motion sensor;

[0034] 5. Calculate the average optical flow speed and average camera speed: Repeat steps 2, 3, and 4 above, take N groups of data, calculate the average optical flow speed and camera motion speed, and obtain the average optical flow speed V_flow_x, V_flow_y of the feature points and the average speed V_cam of the motion camera;

[0035] 6. Set the reference plane: The distance between the reference plane and the optical center of the camera is D, and the optical flow velocity V_flow_base of the reference plane is calculated using the camera's internal parameters;

[0036] 7. Calculate the target height: Combine the optical flow velocities in two directions of the target to get the target's optical flow velocity: Then, according to the perspective relationship, the target optical flow and the reference optical flow are combined to calculate the height of the target center point relative to the reference plane.

[0037] The following is a further detailed description through Example 1.

[0038] Embodiment 1:

[0039] The present invention proposes a method for estimating the height of an object in motion based on a monocular camera and optical flow velocity measurement. The method uses the image data captured by the monocular camera to correct the image distortion, and then uses the recognition and tracking algorithm to locate the target and select the target feature points. The optical flow velocity of the target feature point is obtained by the optical flow method, and then a reference plane is set, and the optical flow of the reference plane is obtained by combining the motion state of the camera and the camera's internal parameters. Finally, the height estimation of the moving object is achieved through the geometric perspective relationship of the optical flow.

[0040] like Figure 1 As shown, the implementation process of the present invention is described, including the following steps:

[0041] 1. First, use the OpenCV algorithm to correct the image distortion. The specific steps are as follows:

[0042] Camera calibration: Use the calibration plate to take multiple images and calculate the camera's internal parameters (such as focal length f_u, f_v and principal point coordinates c_u, c_v) and distortion parameters (radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2);

[0043] Distortion correction: Use the cv2.undistort function provided by OpenCV, combined with the camera's intrinsic parameters and distortion parameters. Whenever the camera acquires an image, it corrects the image to eliminate lens distortion and ensure the accuracy of the image.

[0044] 2. Image frame acquisition: The camera acquires two adjacent image frames frame_t and frame_t_pre, records the corresponding time t and t_pre of the two frames, and calculates the time difference between the two frames dt = t-t_pre;

[0045] 3. Obtaining the target optical flow: For the two frames of images frame_t and frame_t_pre acquired by the camera, the pixel position of the target on the image frame_t_pre is confirmed by the target recognition and tracking algorithm, and the feature point O(x, y) is selected. Then, the optical flow method of OpenCV is used to track the feature point O_next (x_next, y_next) of the target in the next frame frame_t, and the optical flow flow_x and flow_y of the feature point pixels in the horizontal and vertical directions are obtained, and the optical flow speed is calculated: It should also be noted that, due to relative motion, the direction of the target's optical flow velocity is opposite to the direction of the camera's motion velocity;

[0046] 4. Obtaining the camera motion speed: Obtain the camera motion speed v_cam through the motion sensor;

[0047] 5. Obtaining the average optical flow speed and average camera speed: Repeat steps 2, 3, and 4 above, take N groups of data, calculate the average optical flow speed and camera motion speed, and obtain the average optical flow speed V_flow_x, V_flow_y of the feature points and the average speed V_cam of the motion camera:

[0048]

[0049] 6. Calculation of optical flow on the reference plane: Figure 2 As shown in the schematic diagram, the direction of this schematic diagram is horizontal u, the direction of the camera is horizontal u and vertical v, which are consistent with the x and y directions of the image respectively; the component of the average speed V_cam in the u direction is V_cam_u, and the component in the v direction is V_cam_v; the distance of the reference plane relative to the optical center of the camera is set to D, assuming that the time of acquiring the two frames of images is t and t_pre respectively, after dt time, the camera has a displacement in the u direction of: S_u = dt V_cam_u, so the reference plane has a displacement in the -u direction relative to the camera of S_u, f is the physical focal length of the camera, according to the similarity relationship: s_u is the actual displacement of the image of the reference plane on the camera imaging plane; there is also a relationship: s_u = flow_u du, where du is the pixel size in the u direction, that is, the physical size of each pixel in the u direction, the unit (for example: mm) is consistent with f, and flow_u is the optical flow size of the reference plane in the u direction on the image; s_u combines the camera's intrinsic parameters (f_u, f_v) and You can get Then we get the benchmark optical flow speed in two directions: Finally, the optical flow velocity of the synthetic reference plane is:

[0050]

[0051] 7. Calculate the target height: Figure 3 As shown in the schematic diagram, when the target and the reference plane move relative to the camera at the average speed of V_cam, assuming that the movement time is dt, the horizontal displacement is d_u, if the height of the target relative to the reference plane is 0, the optical flow speed of the target's feature points is consistent with the optical flow speed of the reference plane, then the optical flow size in the horizontal direction is flow_u, if the height of the target relative to the reference plane is h, under the same movement time and speed, due to the relationship of camera perspective imaging, the optical flow of the target's feature points on the camera's image becomes flow_u_h, the actual displacement size of the target is d_u, but the equivalent displacement size projected to the reference plane is d_u_h, from the geometric relationship:

[0052] After sorting, you can get: Then get Where V_flow_object is the optical flow velocity of the target obtained by merging the optical flow velocities in two directions of the target:

[0053] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0054] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for estimating the height of an object in motion based on a monocular camera and optical flow velocity measurement, characterized in that: The following steps are involved: S1: Camera calibration and distortion correction: Calibrate the camera's internal parameters and distortion coefficients, and perform distortion correction on the image; S2: Image frame acquisition: The camera acquires two adjacent image frames frame_t and frame_t_pre, records the corresponding time t and t_pre of the two frames, and calculates the time difference dt=t-t_pre between the two frames; S3: Acquisition of target optical flow: For the two frames of images frame_t and frame_t_pre acquired by the camera, the pixel position of the target on the image frame_t_pre is confirmed by the target recognition and tracking algorithm, and the feature point O(x, y) is selected. Then, the optical flow method of OpenCV is used to track the feature point O_next (x_next, y_next) of the target in the next frame frame_t, and the optical flow flow_x and flow_y of the feature point pixels in the horizontal x and vertical y directions are obtained, and the optical flow speed is calculated: S4: Obtaining the camera motion speed: obtaining the camera motion speed v_cam through the motion sensor; S5: Obtaining the average optical flow speed and average camera speed: Repeat the above S2, S3 and S4 processes, take N groups of data, calculate the average optical flow speed and camera motion speed, and obtain the average optical flow speed V_flow_x, V_flow_y of the feature points and the average speed V_cam of the camera; S6: Set the reference plane, the distance between the reference plane and the optical center of the camera is D, and use the camera's internal parameters to calculate the optical flow velocity V_flow_base of the reference plane. The process is as follows: the direction of the camera is horizontal u and vertical v, which are consistent with the x and y directions of the image respectively. The average speed of the camera V_cam is decomposed into the average speed in the horizontal u and vertical v directions as V_cam_u and V_cam_v. Combined with the camera's internal parameters: focal length (f_u, f_v), the base optical flow velocity in the horizontal u and vertical v directions is obtained: Finally, the optical flow velocity of the synthetic reference plane is: S7: Calculate the target height: Combine the optical flow velocities of the target in the horizontal and vertical directions to obtain the optical flow velocity of the target: Then, according to the perspective relationship, the target optical flow and the reference optical flow are combined to calculate the height of the target center point relative to the reference plane.

2. The method for estimating the height of an object in motion based on a monocular camera and optical flow velocity measurement according to claim 1, characterized in that: In the step S1, the image is distorted by using the OpenCV algorithm.

3. A non-volatile computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a robot, the robot executes the method according to any one of claims 1-2.

4. A robot, characterized in that: The robot comprises a monocular camera and executes the method as described in any one of claims 1-2.

Images