Machine vision-based driving target positioning method
By combining neural networks and homography-calibrated cameras, the problem of low positioning accuracy of ordinary 2D cameras in vehicle automation is solved, achieving efficient and low-cost target positioning in complex scenarios, and applicable to target detection placed in any location.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- MATRIXTIME ROBOTICS (SHANGHAI) CO LTD
- Filing Date
- 2022-05-27
- Publication Date
- 2026-07-03
AI Technical Summary
In vehicle automation scenarios, using ordinary 2D cameras for target location suffers from high costs, low accuracy, and difficulty in identification due to the target being obscured by the sling. Existing technologies struggle to achieve accurate positioning in complex scenarios.
By combining neural networks and homography-calibrated cameras, scene images of the target object are acquired through the camera, the distance between the vehicle and the target object is calculated, the center pixel coordinates of the target object are obtained using neural networks, and the distance is calculated using the camera's intrinsic parameter matrix and height. Vertical calibration is then performed using the homography matrix to achieve precise positioning of the target object.
It achieves high-precision target localization based on ordinary 2D cameras in complex scenes, reduces costs, expands application scenarios, improves computing speed and robustness, and can detect targets placed at arbitrary locations.
Smart Images
Figure CN114913231B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of vehicle automation technology, specifically relating to a method for locating vehicle targets based on machine vision. Background Technology
[0002] In automated vehicle scenarios, it's necessary to pinpoint the 3D location of the target object relative to the lifting device. Using 3D lasers for this purpose is costly, while industrial depth cameras with 3D information are prone to damage due to the significant vibrations in older automated vehicle environments. A more economical approach is to use a standard, shock-resistant 2D network camera. However, these cameras typically lack depth information. When only one camera without depth information can be added, 3D positioning using the 2D camera becomes problematic. Since the camera's mounting may not be perfectly vertical, the distance calculated using geometric relationships doesn't accurately reflect the actual distance between the target object and the lifting device, leading to movement errors. Furthermore, using only one camera can result in the target object being obscured by the lifting device during vehicle movement, making it impossible to identify the target object.
[0003] Other related technical solutions commonly used in RTG yards involve determining the 3D target location using technologies such as magnetic nails and QR codes. However, this method is suitable for situations where the target is placed at a fixed location. For targets in the driving area, which may be randomly placed within a certain range, the method of locating by markers cannot solve the problem. Summary of the Invention
[0004] The purpose of this invention is to provide a machine vision-based method for locating targets on a vehicle. This invention is mainly applied to the location of targets during the retrofitting of old vehicles. By combining target identification in a neural network and homography calibration of a camera, the relative position of the target in 3D space can be accurately calculated, enabling precise location of targets in complex scenes.
[0005] To achieve the above objectives, the present invention provides the following technical solution:
[0006] A machine vision-based method for locating a target object while driving, characterized in that: a camera on the vehicle acquires a scene image of the target object; a neural network is used to obtain the center pixel coordinate information of the target object in the scene image; and the distance between the vehicle and the target object is calculated using the height of the camera from the ground and the camera's intrinsic parameter matrix.
[0007] Furthermore, the distance (dx, dy) between the vehicle and the target object:
[0008] dx = (u - cx) * d / fx
[0009] dy=(v-cy)*d / fy
[0010] The known center pixel coordinates of the target object are (u, v), the camera's height above the ground is d, and the camera's intrinsic parameter matrix is:
[0011] Furthermore, the positioning method also includes vertical calibration of the camera: assuming the camera is vertically downward, the hypothetical center pixel coordinate information is obtained, and a mapping relationship between the hypothetical center pixel coordinate information and the real center pixel coordinate information is established; the hypothetical center pixel coordinate information corresponding to the real center pixel coordinate information is calculated through the mapping relationship, which is the center pixel coordinate information of the target object as (u, v).
[0012] Furthermore, the establishment of the mapping relationship includes:
[0013] (1) Record the initial position information (x0, y0) of the vehicle.
[0014] (2) Place four targets within the assumed field of view of the camera, move the vehicle directly above the first target, and record the vehicle's position information (x1, y1).
[0015] The distance between the initial position of the vehicle and the first target object is denoted as: dx1 = x1 - x0, dy1 = y1 - y0;
[0016] The following can be calculated using the intrinsic parameter matrix and the camera's height d above the ground:
[0017] ui1 = dx1 * fx / d + cx
[0018] vi1 = dy1 * fy / d + cy
[0019] Point (ui1, vi1) represents the center pixel coordinates of the first target object under the assumed camera view;
[0020] (3) The real camera acquires scene images of the target object at the initial position of the vehicle, and uses a neural network to identify the center pixel coordinates (ur1, vr1) of the first target object;
[0021] (4) Repeating the above process will yield four pairs of points (ui1,vi1) and (ur1,vr1), (ui2,vi2) and (ur2,vr2), (ui3,vi3) and (ur3,vr3), (ui4,vi4) and (ur4,vr4);
[0022] (5) The homography matrix H between the real camera viewpoint and the assumed camera viewpoint is calculated through the above four pairs of points, which is the mapping relationship.
[0023] Furthermore, the four targets are respectively located at the four corners of the assumed field of view of the camera.
[0024] Furthermore, the neural network identifies the center pixel coordinates (ur, vr) of the target object, which are written as homogeneous coordinates (ur, vr, 1). Let each term of the homography matrix H be hij, where i and j range from 1 to 3. Then, the following calculation is performed:
[0025] ui=(h11*ur+h12*vr+h13) / (h31*ur+h32*vr+h33)
[0026] vi=(h21*ur+h22*vr+h23) / (h31*ur+h32*vr+h33)
[0027] Then (ui, vi) are the assumed center pixel coordinates of the target object under the camera. Using the camera's height d above the ground and the intrinsic parameter matrix, the distance from the vehicle to the center of the target object is calculated as follows:
[0028] dx = (ui - cx) * d / fx
[0029] dy=(vi-cy)*d / fy
[0030] Based on (dx, dy), control the vehicle to move to the corresponding position.
[0031] Furthermore, the neural network is a key point detection network or a frame recognition network.
[0032] Furthermore, the neural network comprises a keypoint detection network and a bounding box recognition network, wherein the keypoint detection network has a higher priority than the bounding box recognition network.
[0033] Furthermore, the positioning method also includes real-time correction, which includes: continuously acquiring scene images during the vehicle's movement toward the target object; first, performing a key point detection network on each scene image to obtain the center pixel coordinate information of the target object; and then using a bounding box recognition network to detect the target object when the key point detection network fails to detect it.
[0034] Furthermore, the training of the keypoint detection network includes: acquiring scene images of the target object through a camera, cropping ROIs so that the target object appears in the center, labeling key points, and then using the keypoint detection network for training; the training of the bounding box recognition network includes: acquiring scene images of the target object through a camera, labeling the bounding boxes of the target object in the scene images, and then using the bounding box recognition network for training.
[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0036] First, this invention proposes a method for positioning based on a regular 2D camera in a driving scenario, which saves costs and improves computing speed compared to 3D lasers and 3D depth cameras.
[0037] Secondly, this invention combines the flexibility of neural networks with the geometric characteristics of homogrphy calibration, enabling the detection of targets placed at arbitrary locations and expanding the application scenarios.
[0038] Third, this invention combines two neural networks to achieve long-distance and short-distance positioning respectively. By utilizing the long-distance recognition of the bounding box and the short-distance accuracy of key points, the robustness of the positioning is guaranteed. Attached Figure Description
[0039] Figure 1 This is a flowchart of camera calibration in the embodiment.
[0040] Figure 2 This is a flowchart of the vehicle target localization method based on machine vision in the embodiment. Detailed Implementation
[0041] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to specific examples. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0042] Based on the technical problems in the background art, this embodiment combines the identification of target objects in neural networks and the homography calibration of cameras to design a machine vision-based target object localization method for vehicles. It is mainly applied to the localization of target objects during the retrofitting of old vehicles, accurately calculates the relative position of the target object in 3D space, and achieves accurate localization of target objects in complex scenes.
[0043] like Figure 2 As shown, this method mainly consists of a bounding box recognition neural network, a keypoint recognition neural network, and a homography module during execution. The data flow is as follows: an image of the current scene is acquired through the camera, then fed into the bounding box recognition neural network and the keypoint recognition neural network. The obtained target pixel information is then fed into the homography module to calculate the distance from the vehicle to the center of the target object.
[0044] I. Renovation of the old crane
[0045] When installing a regular 2D network camera (hereinafter referred to as camera) on an old crane (also applicable to new cranes, hereinafter collectively referred to as crane), you can choose one with shockproof function (not required for new cranes). When installing, you need to ensure that the camera is as vertically downward as possible.
[0046] II. Training Neural Networks
[0047] The neural networks used in this design include YOLOv5 (YOLO and other rectangular frame recognition networks, which are neural networks that select targets using rectangular frames, hereinafter referred to as frame recognition networks) and key point detection networks. Both YOLOv5 and key point detection networks can independently identify the center pixel coordinates of the target object.
[0048] Neural networks were chosen to better identify objects under occlusion conditions. Keypoint neural networks can accurately identify and perform calculations when the rigging moves above the object; however, due to their network characteristics, their field of view is relatively small, concentrated in the center of the image, and they cannot identify objects when they are not near or below the rigging (or when the object is occluded by the rigging). Contrast-boundary neural networks can identify objects across the entire field of view, but their accuracy is lower than that of keypoint neural networks.
[0049] 1. Training Yolov5
[0050] Images of the target object are captured by a camera in the scene. The target object is labeled with a positive bounding box and used as a training sample. Yolov5 is used for training to identify situations where the target object is in the field of view but is far away from the hoist (the target object will appear in various corners and edges of the field of view).
[0051] 2. Training of the key point detection network
[0052] By acquiring images of the target object in the scene, setting the ROI of the central part so that the target object appears in the center of the field of view, labeling key points as training samples, and using a key point detection network for training, the target object can be identified when it is close to the rigging in the field of view (without occlusion).
[0053] In this scheme, the simultaneous use of a key point detection network and a bounding box recognition network can complement each other and improve accuracy. The key point detection network has a higher priority than the bounding box recognition network. That is, if the key point detection network can identify the center pixel coordinates of the target object, it is used directly; otherwise, the bounding box recognition network is required.
[0054] Simultaneous use of a keypoint detection network and a bounding box recognition network allows for real-time correction of target object localization. Specifically, this involves continuously acquiring scene images as the vehicle moves toward the target object, updating approximately 20 frames per second. For each scene image, the keypoint detection network is first used to detect and obtain the center pixel coordinates of the target object. If the keypoint detection network fails to detect the target object, the bounding box recognition network is then used. The next scene image is similarly first detected by the keypoint detection network. Only when the target object is located near the bottom of the vehicle and is unobstructed does the more accurate keypoint detection network become available.
[0055] III. Camera Vertical Correction
[0056] Since the camera cannot be mounted perfectly vertically downwards (which would affect accuracy during position calculations), vertical downward calibration of the camera is necessary to improve accuracy. This solution uses the homography matrix method, such as... Figure 1 As shown, the details are as follows:
[0057] Now, suppose that the same camera of the same model is installed in the same position with an absolutely vertical downward orientation. Since the driving application scenario is at a fixed height, homography (homography is used to calculate the transformation of pixel coordinates of a plane under different viewpoints, that is, to transform the pixel coordinates under one viewpoint to another viewpoint) is used to calculate the mapping relationship between the images of the two cameras (the real camera and the hypothetical camera). Let the height of the camera from the ground be d.
[0058] 1. First, with the vehicle stationary, record the initial position information (x0, y0) of the vehicle.
[0059] 2. Place the target object near one of the four corners of the field of vision. The following example uses the target object in the upper right corner.
[0060] Move the vehicle over the target object and record its position information (x1, y1), then return to (x0, y0).
[0061] The distance (dx, dy) between the initial position of the vehicle and the first target object is denoted as: dx1 = x1 - x0, dy1 = y1 - y0.
[0062] The camera's intrinsic parameter matrix is:
[0063] The following can be calculated using the intrinsic parameter matrices fx, fy, cx, cy and the camera's height d above the ground:
[0064] ui1 = dx1 * fx / d + cx
[0065] vi1 = dy1 * fy / d + cy
[0066] The point (ui1, vi1) is the assumed center pixel coordinate of the target object under the camera.
[0067] 3. Further, based on neural networks, taking a bounding box recognition network as an example, an image of a target object at position (x1, y1) is captured at position (x0, y0) using a real camera. The bounding box recognition network then identifies the target object information (ltx, lty, wh0, dh0), where ltx and lty are the coordinates of the top-left corner of the target object's bounding box, and wh0 and dh0 are the width and height of the target object's bounding box. Therefore, the center pixel coordinates (ur1, vr1) of the target object under the real camera image are:
[0068] ur1 = ltx + wh0 / 2
[0069] vr1 = lty + dh0 / 2
[0070] In this way we obtain a pair of points (ui1,vi1) and (ur1,vr1).
[0071] By repeating the above method on three other points, we can obtain three more pairs of points: (ui2,vi2) and (ur2,vr2), (ui3,vi3) and (ur3,vr3), and (ui4,vi4) and (ur4,vr4).
[0072] With these four pairs of points, we can use OpenCV to calculate the homography matrix H between our actual camera view and the assumed camera view. This concludes the calibration process. Note that the 2D pixel coordinates need to be converted to homogeneous coordinates because the homography matrix H is a 3x3 matrix, and the corresponding coordinates are 3x1 vectors.
[0073] IV. Target Location
[0074] 1. First, the target objects in the scene are detected by using a bounding box recognition network and a key point detection network.
[0075] If the keypoint detection network identifies the keypoint, it calculates the center pixel coordinates (ur, vr) of the target object by using the position of the keypoint and the geometric relationship of the target object.
[0076] If the keypoint detection network cannot identify it, the center pixel coordinates (ur, vr) of the target object are identified by the bounding box recognition network.
[0077] Next, write the homogeneous coordinates (ur, vr, 1).
[0078] 2. Let each term of the homography matrix H be hij, where i and j range from 1 to 3. Then we can calculate:
[0079] ui=(h11*ur+h12*vr+h13) / (h31*ur+h32*vr+h33)
[0080] vi=(h21*ur+h22*vr+h23) / (h31*ur+h32*vr+h33)
[0081] Point (ui, vi) represents the assumed center pixel coordinates of the target object under the camera's view.
[0082] 3. Using the camera's height d above the ground and the intrinsic parameter matrix, calculate the distance from the vehicle to the center of the target object:
[0083] dx = (ui - cx) * d / fx
[0084] dy=(vi-cy)*d / fy
[0085] Based on (dx, dy), the spreader can be controlled to move to the corresponding position.
[0086] Example 1
[0087] In the simulated driving application, the target object in the scene was selected as a steel plate, and the key point identification information was the four corner points of the rectangular steel plate: top left, bottom left, top right, and bottom right. The camera was mounted on the crane of the lifting device, 4 meters above the ground. The positioning accuracy using the above method can reach within 2cm, which meets the application requirements.
[0088] The results of the comparative experiment are as follows:
[0089] (1) Using only the key point detection network combined with homography calculation, the target object will fail to be found when it is randomly placed.
[0090] (2) Using only Yolov5 combined with homography, the accuracy is about 5cm.
[0091] (3) Using only the key point detection network and YOLOv5, without using homography calibration, the accuracy is 10cm based on the existing camera angle.
[0092] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for locating driving targets based on machine vision, characterized in that: The camera on the vehicle acquires scene images of the target object, a neural network is used to obtain the center pixel coordinate information of the target object in the scene image, and then the distance between the vehicle and the target object is calculated by the height of the camera from the ground and the camera's intrinsic parameter matrix. The distance (dx, dy) between the vehicle and the target object: dx = (ui - cx) * d / fx dy = (vi - cy) * d / fy The known center pixel coordinates of the target object are (ui, vi), the camera's height above the ground is d, and the camera's intrinsic parameter matrix is: ; The positioning method also includes vertical calibration of the camera: assuming the camera is vertically downward, the hypothetical center pixel coordinate information is obtained, and a mapping relationship between the hypothetical center pixel coordinate information and the real center pixel coordinate information is established; the hypothetical center pixel coordinate information corresponding to the real center pixel coordinate information is calculated through the mapping relationship, which is the center pixel coordinate information of the target object as (ui, vi). The establishment of the mapping relationship includes: (1) Record the initial position information of the vehicle (x0, y0). (2) Place four targets within the assumed field of view of the camera, move the vehicle directly above the first target, and record the vehicle's position information (x1, y1). The distance between the initial position of the vehicle and the first target object is denoted as: dx1 = x1 - x0, dy1 = y1 - y0; The following can be calculated using the intrinsic parameter matrix and the camera's height d above the ground: ui1 = dx1 * fx / d + cx vi1 = dy1 * fy / d + cy The point (ui1, vi1) represents the center pixel coordinates of the first target object under the assumed camera view; (3) The real camera acquires scene images of the target object at the initial position of the vehicle, and uses a neural network to identify the center pixel coordinates (ur1, vr1) of the first target object; (4) Repeating the above process will yield four pairs of points (ui1, vi1) and (ur1, vr1), (ui2, vi2) and (ur2, vr2), (ui3, vi3) and (ur3, vr3), (ui4, vi4) and (ur4, vr4); (5) The homography matrix H between the real camera viewpoint and the assumed camera viewpoint is calculated through the above four pairs of points, which is the mapping relationship.
2. The vehicle target localization method based on machine vision according to claim 1, characterized in that: The four targets are located at the four corners of the assumed camera's field of view.
3. The vehicle target localization method based on machine vision according to claim 1, characterized in that: The neural network identifies the center pixel coordinates (ur, vr) of the target object, which are written as homogeneous coordinates (ur, vr, 1). Let each term of the homography matrix H be hij, where i and j range from 1 to 3. Then calculate: ui = (h11*ur + h12*vr+h13) / (h31*ur + h32*vr+h33) vi = (h21*ur + h22*vr+h23) / (h31*ur + h32*vr+h33) Then (ui, vi) are the assumed center pixel coordinates of the target object under the camera. Using the camera's height d above the ground and the intrinsic parameter matrix, the distance from the vehicle to the center of the target object is calculated as follows: dx = (ui - cx) * d / fx dy = (vi - cy) * d / fy Based on (dx, dy), control the vehicle to move to the corresponding position.
4. The vehicle target localization method based on machine vision according to claim 1, characterized in that: The neural network is a key point detection network or a frame recognition network.
5. The vehicle target localization method based on machine vision according to claim 1, characterized in that: The neural network consists of a keypoint detection network and a bounding box recognition network, with the keypoint detection network having a higher priority than the bounding box recognition network.
6. The vehicle target localization method based on machine vision according to claim 5, characterized in that: The positioning method further includes real-time correction, which includes: continuously acquiring scene images during the vehicle's movement toward the target object; first, performing a key point detection network on each scene image to obtain the center pixel coordinate information of the target object; and then using a bounding box recognition network to detect the target object when the key point detection network fails to detect it.
7. The vehicle target localization method based on machine vision according to any one of claims 4-6, characterized in that: The training of the keypoint detection network includes: acquiring scene images of the target object through a camera, cropping ROIs so that the target object appears in the center, labeling key points, and then using the keypoint detection network for training; the training of the bounding box recognition network includes: acquiring scene images of the target object through a camera, labeling the bounding boxes of the target object in the scene images, and then using the bounding box recognition network for training.