Target three-dimensional positioning method based on depth vision

Abstract

The embodiment of the invention provides a target three-dimensional positioning method based on depth vision, and the method comprises the steps: obtaining data, and obtaining a depth image and an RGB image through a depth camera; processing the RGB image by adopting a set target detection model to obtain a bounding box of each object in the RGB image; based on the bounding box, extracting a pixel coordinate of a central point in the bounding box; depth information fusion is carried out, a corresponding depth value is searched in the depth map based on the pixel coordinates of the center point, and the depth value represents the linear distance from the depth camera plane to the target center point; and converting the points in the two-dimensional image coordinate system into a three-dimensional world coordinate system by using the internal reference and the external reference of the depth camera. And restoring the target identified in the RGB image to the position of the target in the three-dimensional world through a YOLO target detection model.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and information processing, and in particular to a three-dimensional target positioning method based on depth vision. Background Art

[0002] Three-dimensional target positioning technology is a key technology in the field of machine vision and automation. It is widely used in many fields such as self-driving cars, environmental mapping, and augmented reality. Although current technology has been able to achieve three-dimensional spatial positioning of targets by using methods such as lidar, stereo vision, and structured light, these methods often require expensive equipment and complex algorithm support, and their adaptability and flexibility in dynamic environments are still limited. The following technical challenges are faced in the three-dimensional positioning of targets:

[0003] Computational resource limitations: The application of existing 3D positioning technologies on mobile devices is limited by the computing resources of the devices. Advanced image processing and object detection algorithms require a lot of computing power, which is a challenge on mobile devices with limited resources.

[0004] Coping with environmental complexity: Maintaining positioning accuracy in complex environments, such as different lighting conditions and weather changes, is one of the main problems faced by existing technologies. It is necessary to develop more powerful and robust algorithms that can adapt to these changes.

[0005] Data fusion and processing: When existing technologies process data from different sensors, how to effectively fuse this information is a key issue. In addition, using the differences between simulated data and real-world data to improve the reliability of algorithms in practical applications is a focus of current research. Summary of the invention

[0006] In view of the above problems existing in the prior art, an embodiment of the present invention provides a three-dimensional target positioning method based on depth vision, which restores the target identified in the RGB image to its position in the three-dimensional world through the YOLO target detection model.

[0007] The embodiment of the present invention provides a method for three-dimensional positioning of a target based on depth vision, comprising:

[0008] Get data, use the depth camera to get the depth map and RGB map;

[0009] The RGB image is processed using a set object detection model to obtain the bounding box of each object in the RGB image;

[0010] Based on the bounding box, extracting the pixel coordinates of the center point in the bounding box;

[0011] Performing depth information fusion, including searching a corresponding depth value in a depth map based on the pixel coordinates of the center point, wherein the depth value represents a straight-line distance from a depth camera plane to a center point of the target;

[0012] Using the intrinsic and extrinsic parameters of the depth camera, the points in the two-dimensional image coordinate system are transformed into the three-dimensional world coordinate system.

[0013] In some embodiments of the present invention, the method for three-dimensional positioning of a target based on depth vision further includes:

[0014] If the 3D world coordinate system used is inconsistent with the camera coordinate system, the coordinate system is further converted.

[0015] In some embodiments of the present invention, the depth map is used to provide distance information from each pixel to the camera;

[0016] The RGB image is used for target detection.

[0017] In some embodiments of the present invention, the step of processing the RGB image using a set target detection model to obtain a bounding box of each object in the RGB image includes:

[0018] The YOLO target detection model is used to process the RGB image, and objects of different categories are identified in the RGB image by the YOLO target detection model, and a bounding box is output for each detected object.

[0019] In some embodiments of the present invention, outputting a bounding box for each detected object includes:

[0020] Output bounding box coordinates and category labels for each detected object;

[0021] If the bounding box is a rectangle, the output is the coordinates of the four corners of the bounding box.

[0022] In some embodiments of the present invention, if the bounding box is a rectangle, the pixel coordinates of the center point in the extracted bounding box are obtained by calculating the average value of the coordinates of the upper left corner and the lower right corner of the bounding box.

[0023] In some embodiments of the present invention, the step of converting a point in a two-dimensional image coordinate system into a three-dimensional world coordinate system by using an intrinsic parameter and an extrinsic parameter of a depth camera includes:

[0024] According to the intrinsic parameters of the depth camera including at least the focal length and the optical center, the pixel coordinates are converted into normalized coordinates in the camera coordinate system, and the normalized coordinates are converted into actual three-dimensional space coordinates using the depth value;

[0025] If the depth camera also includes external parameters including at least the position and orientation of the camera, the pixel coordinates are converted into normalized coordinates in the camera coordinate system according to the internal parameters of the depth camera including at least the focal length and the optical center, and the external parameters including at least the position and orientation of the camera.

[0026] In some embodiments of the present invention, if the bounding box coordinates output by the YOLO model are

[0027] (x min ,y min ,x max ,y max ),

[0028] Among them, (x min, y min ), is the coordinate of the upper left corner of the bounding box, (x max, y max ) are the coordinates of the lower right corner of the bounding box;

[0029] The pixel coordinates (u, v) of the center point of the bounding box are calculated using the following formula:

[0030]

[0031] Let the depth value of the point (u, v) in the depth map be d, which represents the straight-line distance from the camera to the point;

[0032] Using the camera's intrinsic parameter matrix k, where k includes the focal length f x ,f y and light center c x ,c y , convert the pixel coordinates (u, v) to a three-dimensional point (X, Y, Z) in the camera coordinate system;

[0033] The internal parameter matrix is:

[0034]

[0035] It uses the inverse matrix of the intrinsic parameter matrix and the depth z to restore the coordinates of the three-dimensional point in the camera coordinate system;

[0036]

[0037] Since the camera device is installed at a certain position of the vehicle, including translation and rotation, the target 3D coordinates in the camera coordinate system need to be converted into the vehicle coordinate system coordinates;

[0038] Among them, R is the rotation matrix of the camera relative to the vehicle, and T is the translation matrix of the camera relative to the vehicle;

[0039]

[0040] Coordinate system (x v ,y v ,z v ) is in the camera coordinate system, the camera coordinate system x-axis faces right, y-axis faces down, and z-axis faces forward. The coordinate axis needs to be converted into NED coordinates or ENU coordinates used in the field of autonomous driving, depending on the settings of the specific project, the coordinate conversion matrix;

[0041]

[0042] For vehicles using the ENU coordinate system, use the following formula:

[0043]

[0044] To obtain the coordinates of the vehicle in the world coordinate system, it is necessary to process the rotation and translation of the vehicle coordinate system and the world coordinate system. R is the rotation matrix of the vehicle relative to the world coordinate system, which is constructed based on the pitch angle, yaw angle and roll angle, or based on quaternion. T is the displacement matrix of the vehicle relative to the world coordinate system.

[0045]

[0046] Compared with the prior art, the method for three-dimensional positioning of a target based on depth vision provided by the embodiment of the present invention has the following beneficial effects:

[0047] 1. Accurate three-dimensional positioning capability

[0048] It uses deep vision combined with advanced image processing algorithms to accurately calculate the position of the target in three-dimensional space. This accuracy is particularly important for application scenarios that require high position accuracy (such as automatic navigation, robot control, etc.), and can significantly improve the operating efficiency and safety of the system.

[0049] 2. Enhanced target recognition performance

[0050] By combining depth information with traditional RGB image data, this technology can improve the accuracy and reliability of object recognition. Depth information provides an additional dimension for object recognition, helping to solve the occlusion and similar background problems that may be encountered in traditional two-dimensional image recognition.

[0051] 3. Real-time processing and tracking

[0052] It supports real-time data processing, can instantly update the target position, and adapt to dynamically changing environments. This can effectively improve the response speed and environmental adaptability for applications that require real-time response (such as driverless cars and mobile robots).

[0053] 4. Applicable to various environments

[0054] This 3D positioning technology is not only applicable to indoor environments, but can also work effectively outdoors or in complex and changing environments. This environmental adaptability expands the application scope of the technology, including industrial automation, outdoor detection, disaster relief and other fields.

[0055] 5. Low Dependency

[0056] This technology mainly relies on data captured by the camera for processing. Compared with other three-dimensional positioning technologies that require additional hardware (such as lidar, etc.), it can reduce equipment dependence while reducing costs, making deployment more flexible and economical. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of the coordinates of a target projected on an image plane in a three-dimensional target positioning method based on depth vision provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0059] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0060] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0061] It should also be understood that, although the present application has been described with reference to some specific examples, those skilled in the art will be able to realize many other equivalent forms of the present application that have the features described in the claims and are therefore within the scope of protection defined thereby.

[0062] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

[0063] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to determine the true intent based on the user's historical operations and to avoid unnecessary or redundant details that make the present application unclear. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but are merely used as the basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.

[0064] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.

[0065] The embodiment of the present invention provides a method for three-dimensional positioning of a target based on depth vision, such as Figure 1 As shown, the method includes:

[0066] Acquire data and use a depth camera to acquire a depth map and an RGB map; the depth map is used to provide distance information from each pixel to the camera; the RGB map is used for target detection.

[0067] The RGB image is processed by a set target detection model to obtain a bounding box of each object in the RGB image; specifically, the RGB image is processed by a YOLO target detection model, objects of different categories are identified in the RGB image by the YOLO target detection model, and a bounding box is output for each detected object.

[0068] In this embodiment, outputting a bounding box for each detected object includes: outputting the coordinates and category label of the bounding box for each detected object; wherein, if the bounding box is a rectangle, the coordinates of the four corners of the bounding box are output.

[0069] Based on the bounding box, the pixel coordinates of the center point in the bounding box are extracted; if the bounding box is a rectangle, the pixel coordinates of the center point in the extracted bounding box are obtained by calculating the average value of the coordinates of the upper left corner and the lower right corner of the bounding box.

[0070] Performing depth information fusion, including searching a corresponding depth value in a depth map based on the pixel coordinates of the center point, wherein the depth value represents a straight-line distance from a depth camera plane to a center point of the target;

[0071] Using the intrinsic and extrinsic parameters of the depth camera, the points in the two-dimensional image coordinate system are transformed into the three-dimensional world coordinate system.

[0072] In this embodiment, the use of intrinsic parameters and extrinsic parameters of the depth camera to convert points in the two-dimensional image coordinate system into the three-dimensional world coordinate system includes: converting pixel coordinates into normalized coordinates in the camera coordinate system according to the intrinsic parameters of the depth camera including at least the focal length and the optical center, and converting the normalized coordinates into actual three-dimensional space coordinates using the depth value; if the depth camera also includes extrinsic parameters including at least the position and orientation of the camera, then converting the pixel coordinates into normalized coordinates in the camera coordinate system according to the intrinsic parameters of the depth camera including at least the focal length and the optical center, and the extrinsic parameters including at least the position and orientation of the camera.

[0073] If the 3D world coordinate system used is inconsistent with the camera coordinate system, the coordinate system is further converted.

[0074] In the actual calculation process, if the bounding box coordinates output by the YOLO model are

[0075] (x min ,y min ,x max ,y max ),

[0076] Among them, (xmin,y min ), is the coordinate of the upper left corner of the bounding box, (x max, y max ) are the coordinates of the lower right corner of the bounding box;

[0077] The pixel coordinates (u, v) of the center point of the bounding box are calculated using the following formula:

[0078]

[0079] Let the depth value of the point (u, v) in the depth map be d, which represents the straight-line distance from the camera to the point;

[0080] Using the camera's intrinsic parameter matrix k, where k includes the focal length f x ,f y and light center c x ,c y , convert the pixel coordinates (u, v) to a three-dimensional point (X, Y, Z) in the camera coordinate system;

[0081] The internal parameter matrix is:

[0082]

[0083] It uses the inverse matrix of the intrinsic parameter matrix and the depth z to restore the coordinates of the three-dimensional point in the camera coordinate system;

[0084]

[0085] Since the camera device is installed at a certain position of the vehicle, including translation and rotation, the target 3D coordinates in the camera coordinate system need to be converted into the vehicle coordinate system coordinates;

[0086] Among them, R is the rotation matrix of the camera relative to the vehicle, and T is the translation matrix of the camera relative to the vehicle;

[0087]

[0088] Coordinate system (x v ,y v ,z v) is in the camera coordinate system, the camera coordinate system x-axis faces right, y-axis faces down, and z-axis faces forward. The coordinate axis needs to be converted into NED coordinates or ENU coordinates used in the field of autonomous driving, depending on the settings of the specific project, the coordinate conversion matrix;

[0089]

[0090] For vehicles using the ENU coordinate system, use the following formula:

[0091]

[0092] To obtain the coordinates of the vehicle in the world coordinate system, it is necessary to process the rotation and translation of the vehicle coordinate system and the world coordinate system. R is the rotation matrix of the vehicle relative to the world coordinate system, which is constructed based on the pitch angle, yaw angle and roll angle, or based on quaternion. T is the displacement matrix of the vehicle relative to the world coordinate system.

[0093]

[0094] In order to facilitate the understanding of the above technical solution, the following is a description with reference to the accompanying drawings and specific examples, as follows:

[0095] 1. Experimental simulation platform configuration

[0096] Platform selection: Use the AirSim experimental simulation platform, which provides a highly realistic urban scene and simulates various environmental factors in the real world.

[0097] Drone configuration: The drone is equipped with a depth camera and an RGB camera, which can capture high-resolution images and depth data for subsequent image processing and target recognition.

[0098] 2. Data Collection

[0099] Topic subscription: Subscribe to the topics of RGB camera and depth camera through the ROS node (airsim_ros_node) provided by AirSim. This allows you to receive image and depth information from the simulation environment in real time.

[0100] Dataset generation: Collect RGB and depth image data to build a dataset for training the YOLOv10 model. The dataset includes images of vehicles in different environments and the corresponding depth information.

[0101] 3. Object detection and recognition

[0102] Model training: Use the collected dataset to train the YOLOv10 model. YOLOv10 is an advanced target detection model that can quickly and accurately identify targets such as vehicles in images and output the bounding box of the target.

[0103] Target recognition: In practical applications, the trained YOLOv10 model is deployed on the drone to process images acquired from the RGB camera in real time, identify vehicle targets in the image, and calculate the bounding box of each target.

[0104] 4. Three-dimensional positioning

[0105] Bounding box processing: The geometric center point of the target is extracted from the bounding box obtained by the YOLOv10 model as the center of mass of the target.

[0106] 3D coordinate acquisition: Calculate the 3D coordinates of the target using the depth information provided by the depth camera and the position of the centroid in the image. Use the camera's intrinsic matrix to convert the 2D image coordinates into 3D world coordinates.

[0107] 5. Target Tracking

[0108] Tracking algorithm implementation: Based on the identified 3D position of the target, the corresponding tracking algorithm is used to track the target. The tracking algorithm can continuously update the target position to ensure that the drone can effectively track the moving vehicle.

[0109] Through the implementation of the above detailed steps, pure vision-based target recognition and three-dimensional positioning can be efficiently performed in a simulation environment, providing technical support for the automatic navigation and target tracking of drones. The application of this technology is not limited to urban vehicles, but can also be extended to the recognition and tracking of other mobile targets, and has broad application prospects.

[0110] It can be seen from the above technical solution that the beneficial effects of the target three-dimensional positioning method based on depth vision provided by the above embodiment of the present invention are as follows:

[0111] 1. Accurate three-dimensional positioning capability

[0112] It uses deep vision combined with advanced image processing algorithms to accurately calculate the position of the target in three-dimensional space. This accuracy is particularly important for application scenarios that require high position accuracy (such as automatic navigation, robot control, etc.), and can significantly improve the operating efficiency and safety of the system.

[0113] 2. Enhanced target recognition performance

[0114] By combining depth information with traditional RGB image data, this technology can improve the accuracy and reliability of object recognition. Depth information provides an additional dimension for object recognition, helping to solve the occlusion and similar background problems that may be encountered in traditional two-dimensional image recognition.

[0115] 3. Real-time processing and tracking

[0116] It supports real-time data processing, can instantly update the target position, and adapt to dynamically changing environments. This can effectively improve the response speed and environmental adaptability for applications that require real-time response (such as driverless cars and mobile robots).

[0117] 4. Applicable to various environments

[0118] This 3D positioning technology is not only applicable to indoor environments, but can also work effectively outdoors or in complex and changing environments. This environmental adaptability expands the application scope of the technology, including industrial automation, outdoor detection, disaster relief and other fields.

[0119] 5. Low Dependency

[0120] This technology mainly relies on data captured by the camera for processing. Compared with other three-dimensional positioning technologies that require additional hardware (such as lidar, etc.), it can reduce equipment dependence while reducing costs, making deployment more flexible and economical.

[0121] The above embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention. The protection scope of the present invention is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.

Claims

1. A three-dimensional target positioning method based on depth vision, characterized in that: include: Get data, use the depth camera to get the depth map and RGB map; The RGB image is processed using a set object detection model to obtain the bounding box of each object in the RGB image; Based on the bounding box, extracting the pixel coordinates of the center point in the bounding box; Performing depth information fusion, including searching a corresponding depth value in a depth map based on the pixel coordinates of the center point, wherein the depth value represents a straight-line distance from a depth camera plane to a center point of the target; Using the intrinsic and extrinsic parameters of the depth camera, the points in the two-dimensional image coordinate system are transformed into the three-dimensional world coordinate system.

2. The method for three-dimensional positioning of a target based on depth vision according to claim 1, characterized in that: Also includes: If the 3D world coordinate system used is inconsistent with the camera coordinate system, the coordinate system is further converted.

3. The method for three-dimensional positioning of a target based on depth vision according to claim 2, characterized in that: The depth map is used to provide distance information from each pixel to the camera; The RGB image is used for target detection.

4. The method for three-dimensional positioning of a target based on depth vision according to claim 3, characterized in that: The method of processing the RGB image by setting the target detection model to obtain the bounding box of each object in the RGB image includes: The YOLO target detection model is used to process the RGB image, and objects of different categories are identified in the RGB image by the YOLO target detection model, and a bounding box is output for each detected object.

5. The method for three-dimensional positioning of a target based on depth vision according to claim 4, characterized in that: The method outputs a bounding box for each detected object, including: Output bounding box coordinates and category labels for each detected object; If the bounding box is a rectangle, the output is the coordinates of the four corners of the bounding box.

6. The method for three-dimensional positioning of a target based on depth vision according to claim 5, characterized in that: If the bounding box is a rectangle, the pixel coordinates of the center point in the extracted bounding box are obtained by calculating the average value of the coordinates of the upper left corner and the lower right corner of the bounding box.

7. The method for three-dimensional positioning of a target based on depth vision according to claim 6, characterized in that: The method of using the intrinsic and extrinsic parameters of the depth camera to transform the points in the two-dimensional image coordinate system into the three-dimensional world coordinate system includes: According to the intrinsic parameters of the depth camera including at least the focal length and the optical center, the pixel coordinates are converted into normalized coordinates in the camera coordinate system, and the normalized coordinates are converted into actual three-dimensional space coordinates using the depth value; If the depth camera also includes external parameters including at least the position and orientation of the camera, the pixel coordinates are converted into normalized coordinates in the camera coordinate system according to the internal parameters of the depth camera including at least the focal length and the optical center, and the external parameters including at least the position and orientation of the camera.

8. The method for three-dimensional positioning of a target based on depth vision according to claim 7, characterized in that: If the bounding box coordinates output by the YOLO model are (x min, y min, x max, y max ), Among them, (x min ,y min ), is the coordinate of the upper left corner of the bounding box, (x max, y max ) are the coordinates of the lower right corner of the bounding box; The pixel coordinates (u, v) of the center point of the bounding box are calculated using the following formula: Let the depth value of the point (u, v) in the depth map be d, which represents the straight-line distance from the camera to the point; Using the camera's intrinsic parameter matrix k, where k includes the focal length f x ,f y and light center c x ,c y , convert the pixel coordinates (u,v) to a three-dimensional point (X,Y,Z) in the camera coordinate system; The internal parameter matrix is: It uses the inverse matrix of the intrinsic parameter matrix and the depth z to restore the coordinates of the three-dimensional point in the camera coordinate system; Since the camera device is installed at a certain position of the vehicle, including translation and rotation, the target 3D coordinates in the camera coordinate system need to be converted into the vehicle coordinate system coordinates; Among them, R is the rotation matrix of the camera relative to the vehicle, and T is the translation matrix of the camera relative to the vehicle; Coordinate system (x v ,y v ,z v ) is in the camera coordinate system, the camera coordinate system x-axis faces right, y-axis faces down, and z-axis faces forward. The coordinate axis needs to be converted into NED coordinates or ENU coordinates used in the field of autonomous driving, depending on the settings of the specific project, the coordinate conversion matrix; For vehicles using the ENU coordinate system, use the following formula: To obtain the coordinates of the vehicle in the world coordinate system, it is necessary to process the rotation and translation of the vehicle coordinate system and the world coordinate system. R is the rotation matrix of the vehicle relative to the world coordinate system, which is constructed based on the pitch angle, yaw angle and roll angle, or based on quaternion. T is the displacement matrix of the vehicle relative to the world coordinate system.

Images