Method, device and storage medium for determining three-dimensional position of target

Abstract

The present application provides a method, device and storage medium for determining the three-dimensional position of a target, including: obtaining a theoretical transformation formula that characterizes the theoretical transformation relationship between a pixel coordinate system and a world coordinate system; judging the terrain state of the target location, and setting the vertical coordinate value of the target's centroid descent point in the world coordinate system according to the terrain state, wherein the centroid descent point is the mapping point of the target's centroid on the contact surface between the target and the ground; obtaining a target transformation formula according to the vertical coordinate value and the theoretical transformation formula, wherein the target transformation formula is used to characterize the actual transformation relationship between the pixel coordinate system and the world coordinate system for the terrain state; and substituting the pixel coordinates of the centroid descent point into the target transformation formula to determine the three-dimensional position information of the target in the world coordinate system. The present application provides a method that avoids the cost investment of collecting a large amount of depth truth data in the prior art by the above-mentioned method, and has low requirements for the operating platform and is universal.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, and storage medium for determining the three-dimensional position of a target. Background Art

[0002] In the field of computer vision research, it is often necessary to estimate the three-dimensional spatial coordinates corresponding to a marker point on a two-dimensional image frame. Currently, the approximate position of the target in the image can be estimated based on the target image information captured by the camera and related camera parameters. However, due to the unknown actual depth information of the target and the uncertainty of scale, it is difficult to determine the accurate three-dimensional position information of the target based on a monocular image. Based on this, the related technology inputs the monocular image into the depth estimation network to obtain a monocular depth map, and then uses the depth information to determine a unique scale to calculate the three-dimensional position information of the target.

[0003] However, the above-mentioned method of regressing scene depth using a neural network model requires the collection of a large amount of true depth values ​​for model training. The process of collecting true value data often requires expensive high-precision data acquisition equipment and a large amount of manpower investment, and the operating rate of the neural network model is closely related to the computing power of the carrier device. However, when running such a neural network model on a low-cost hardware platform, the frame rate will also be limited. In other words, the method of regressing scene depth using a neural network model is limited by the high cost of manpower and material resources, the model frame rate limited by computing power, and the limitations of the model's generalization ability, making it difficult to promote such solutions on a large scale. Summary of the Invention

[0004] The present application aims to solve at least one of the technical problems in the related art to a certain extent. To this end, one object of the present application is to provide a method, device and storage medium for determining the three-dimensional position of a target without the need to collect true depth value related data.

[0005] On the one hand, the present application provides a method for determining the three-dimensional position of a target, which may include: obtaining a theoretical transformation formula that characterizes the theoretical transformation relationship between a pixel coordinate system and a world coordinate system; judging the terrain state of the target location, and setting the vertical coordinate value of the target's center of mass descent point in the world coordinate system according to the terrain state, wherein the center of mass descent point is the mapping point of the target's center of mass on the contact surface between the target and the ground; obtaining a target transformation formula based on the vertical coordinate value and the theoretical transformation formula, wherein the target transformation formula is used to characterize the actual transformation relationship between the pixel coordinate system and the world coordinate system for the terrain state; and substituting the pixel coordinates of the center of mass descent point into the target transformation formula to determine the three-dimensional position information of the target in the world coordinate system.

[0006] In some embodiments, obtaining a theoretical transformation formula representing the theoretical transformation relationship between a pixel coordinate system and a world coordinate system may include: generating a first transformation formula representing the transformation relationship between a world coordinate system and a camera coordinate system; generating a second transformation formula representing the transformation relationship between a pixel coordinate system and a camera coordinate system based on a principle of similar triangles in a camera model of the camera, wherein the camera is used to photograph a target; and integrating the first transformation formula and the second transformation formula to obtain a theoretical transformation formula representing the theoretical transformation relationship between the pixel coordinate system and the world coordinate system, wherein the theoretical transformation formula includes a scale parameter.

[0007] In some embodiments, the theoretical transformation formula is: Among them, s is the scale parameter, u is the row value of the centroid drop point in the pixel coordinate system, v is the column value of the centroid drop point in the pixel coordinate system, K is the intrinsic parameter matrix of the camera, is the rotation transformation parameter between the camera coordinate system and the world coordinate system, P odom is the position matrix of the target in the world coordinate system, is the translation transformation parameter between the camera coordinate system and the world coordinate system.

[0008] In some embodiments, determining the terrain state of the target location and setting the vertical coordinate value of the target's center of mass descent point in the world coordinate system based on the terrain state may include: determining the terrain state of the target location to determine whether the target location and the camera's supporting device are in the same plane; and in response to the judgment result that the target location and the camera's supporting device are in the same plane, setting the vertical coordinate value of the target's center of mass descent point in the world coordinate system to zero, or in response to the judgment result that the target location and the camera's supporting device are not in the same plane, setting the vertical coordinate value of the center of mass descent point in the world coordinate system to an empirical value, wherein the center of mass descent point is the mapping point of the target's center of mass on the contact surface between the target and the ground.

[0009] In some embodiments, obtaining a target transformation formula based on the vertical coordinate value and the theoretical transformation formula may include: performing matrix expansion processing on the theoretical transformation formula to obtain a matrix expansion result; substituting the vertical coordinate value of the centroid descent point in the world coordinate system into the matrix expansion result to calculate the actual scale value of the scale parameter; and substituting the actual scale value into the matrix expansion result to obtain a target transformation formula that represents the actual transformation relationship between the pixel coordinate system and the world coordinate system.

[0010] On the other hand, the present application provides a device for determining the three-dimensional position of a target, which may include: a theoretical transformation formula generation module, a vertical coordinate value setting module, a target transformation formula generation module, and a three-dimensional position determination module. The theoretical transformation formula generation module is used to obtain a theoretical transformation formula that characterizes the theoretical transformation relationship between the pixel coordinate system and the world coordinate system; the vertical coordinate value setting module is used to determine the terrain state of the target location and set the vertical coordinate value of the target's center of mass descent point in the world coordinate system according to the terrain state, wherein the center of mass descent point is the mapping point of the target's center of mass on the contact surface between the target and the ground; the target transformation formula generation module is used to obtain a target transformation formula based on the vertical coordinate value and the theoretical transformation formula, wherein the target transformation formula is used to characterize the actual transformation relationship between the pixel coordinate system and the world coordinate system for the terrain state; the three-dimensional position determination module is used to substitute the pixel coordinates of the center of mass descent point into the target transformation formula to determine the three-dimensional position information of the target in the world coordinate system.

[0011] In some embodiments, the execution steps of the theoretical transformation formula generation module include: generating a first transformation formula representing the transformation relationship between the world coordinate system and the camera coordinate system; generating a second transformation formula representing the transformation relationship between the pixel coordinate system and the camera coordinate system based on the principle of similar triangles in the camera model of the camera, wherein the camera is used to shoot the target; and integrating the first transformation formula and the second transformation formula to obtain a theoretical transformation formula representing the theoretical transformation relationship between the pixel coordinate system and the world coordinate system, wherein the theoretical transformation formula includes a scale parameter.

[0012] In some embodiments, the theoretical transformation formula is: Among them, s is the scale parameter, u is the row value of the centroid drop point in the pixel coordinate system, v is the column value of the centroid drop point in the pixel coordinate system, K is the intrinsic parameter matrix of the camera, is the rotation transformation parameter between the camera coordinate system and the world coordinate system, P odom is the position matrix of the target in the world coordinate system, is the translation transformation parameter between the camera coordinate system and the world coordinate system.

[0013] In some embodiments, the execution steps of the vertical coordinate value setting module include: judging the terrain state of the target location to determine whether the target location and the camera's supporting device are in the same plane; and in response to the judgment result that the target location and the camera's supporting device are in the same plane, setting the vertical coordinate value of the target's center of mass descent point in the world coordinate system to zero, or in response to the judgment result that the target location and the camera's supporting device are not in the same plane, setting the vertical coordinate value of the center of mass descent point in the world coordinate system to an empirical value, wherein the center of mass descent point is the mapping point of the target's center of mass on the contact surface between the target and the ground.

[0014] In some embodiments, the execution steps of the target transformation formula generation module include: performing matrix expansion processing on the theoretical transformation formula to obtain a matrix expansion result; substituting the vertical coordinate value of the center of mass descent point in the world coordinate system into the matrix expansion result to calculate the actual scale value of the scale parameter; and substituting the actual scale value into the matrix expansion result to obtain a target transformation formula that represents the actual transformation relationship between the pixel coordinate system and the world coordinate system.

[0015] Another aspect of the present application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the methods for determining the three-dimensional position of a target provided in one aspect of the present application.

[0016] According to at least one embodiment of the present application, a method, device, and storage medium for determining the three-dimensional position of a target are provided. The vertical coordinate value of the target's centroid descent point in the world coordinate system is set according to the terrain state of the target's location, so as to determine the target transformation formula for different terrain states, and then substitute the pixel coordinates of the target's centroid descent point into the world coordinate system to obtain the target's three-dimensional position coordinates in the world coordinate system. The present application provides a method, which avoids the cost of collecting a large amount of depth truth data in the prior art, and has low requirements for the operating platform and universal applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of the hardware structure of a multi-legged robot according to one embodiment of the present application;

[0018] Figure 2 is a schematic diagram of the mechanical structure of a multi-legged robot according to one embodiment of the present application;

[0019] Figure 3 is a flow chart of a method for determining a three-dimensional position of a target according to one embodiment of the present application;

[0020] Figure 4 is a flow chart for generating a theoretical transformation formula according to one embodiment of the present application;

[0021] Figure 5 is a schematic diagram of a pinhole camera model according to one embodiment of the present application;

[0022] Figure 6A It is a schematic diagram showing that the target and the camera carrying device are on the same plane without slope;

[0023] Figure 6B It is a schematic diagram showing that the target and the camera-carrying device are on the same plane with a slope;

[0024] Figure 7A It is a schematic diagram in which the plane where the target is located is higher than the plane where the camera carrying device is located;

[0025] Figure 7B is a schematic diagram showing that the plane where the target is located is lower than the plane where the camera carrying device is located;

[0026] Figure 8 1 is a schematic structural diagram of a device for determining a three-dimensional position of a target according to one embodiment of the present application;

[0027] Figure 9 is a schematic diagram of an electronic device according to one embodiment of the present application; and

[0028] Figure 10 is a schematic diagram of a storage medium according to one embodiment of the present application. DETAILED DESCRIPTION

[0029] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0030] In the subsequent description, suffixes such as "module," "component," or "unit" used to represent components are used only to facilitate the description of the present invention and have no specific meaning. Therefore, "module," "component," or "unit" may be used interchangeably.

[0031] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of the hardware structure of a multi-legged robot 100 according to an embodiment of the present application. Figure 1 In the embodiment shown, the multi-legged robot 100 includes a mechanical unit 101, a communication unit 102, a sensor unit 103, an interface unit 104, a storage unit 105, a control module 110, and a power supply 111. The various components of the multi-legged robot 100 can be connected in any manner, including wired or wireless connections. It will be understood by those skilled in the art that Figure 1 The specific structure of the multi-legged robot 100 shown in the figure does not constitute a limitation of the multi-legged robot 100. The multi-legged robot 100 may include more or fewer components than shown in the figure. Some components are not necessary components of the multi-legged robot 100 and can be omitted or combined as needed without changing the essence of the invention.

[0032] Figure 2 This is a schematic diagram of the mechanical structure of a multi-legged robot according to an embodiment of the present application. Figure 1 and Figure 2 The various components of the multi-legged robot 100 are introduced in detail:

[0033] The mechanical unit 101 is the hardware of the multi-legged robot 100. Figure 1As shown, the mechanical unit 101 may include a driving plate 1011, a motor 1012, and a mechanical structure 1013. Figure 2 As shown, the mechanical structure 1013 may include a body 1014, extendable legs 1015, and a foot 1016. In other embodiments, the mechanical structure 1013 may further include an extendable mechanical arm (not shown), a rotatable head structure 1017, a swingable tail structure 1018, a cargo structure 1019, a saddle structure 1020, a camera structure 1021, etc. It should be noted that the various component modules of the mechanical unit 101 can be one or more, and can be arranged according to specific circumstances. For example, there can be four legs 1015, each leg 1015 can be configured with three motors 1012, and the corresponding number of motors 1012 is 12.

[0034] The communication unit 102 can be used to receive and send signals and can also communicate with the network and other devices. For example, after receiving instructions from a remote control or other multi-legged robot 100 to move in a specific direction at a specific speed according to a specific gait, the communication unit 102 is transmitted to the control module 110 for processing. The communication unit 102 includes, for example, a WiFi module, a 4G module, a 5G module, a Bluetooth module, an infrared module, etc.

[0035] The sensing unit 103 is used to obtain information data about the environment surrounding the multi-legged robot 100 and to monitor parameter data of various components within the multi-legged robot 100, and to send the data to the control module 110. The sensing unit 103 includes a variety of sensors, such as sensors for obtaining information about the surrounding environment: laser radar (for long-range object detection, distance determination and / or speed value determination), millimeter wave radar (for short-range object detection, distance determination and / or speed value determination), camera, infrared camera, global navigation satellite system (GNSS, Global Navigation Satellite System), etc. For example, sensors for monitoring various components within the multi-legged robot 100 include: inertial measurement unit (IMU) (for measuring velocity, acceleration and angular velocity values), plantar sensor (for monitoring plantar force point position, plantar posture, ground contact force magnitude and direction), temperature sensor (for detecting component temperature). As for other sensors that can be configured for the multi-legged robot 100, such as load sensor, touch sensor, motor angle sensor, torque sensor, etc., they will not be described in detail here.

[0036] The interface unit 104 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more components within the multi-legged robot 100, or may be used to output (e.g., data information, power, etc.) to an external device. The interface unit 104 may include a power port, a data port (e.g., a USB port), a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, etc.

[0037] The storage unit 105 is used to store software programs and various data. The storage unit 105 may primarily include a program storage area and a data storage area. The program storage area may store operating system programs, motion control programs, and application programs (such as a text editor), etc.; the data storage area may store data generated by the multi-legged robot 100 during use (such as various sensor data acquired by the sensor unit 103 and log file data). Furthermore, the storage unit 105 may include high-speed random access memory and non-volatile memory, such as disk storage, flash memory, or other volatile solid-state memory.

[0038] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0039] The input unit 107 can be used to receive input digital or character information. Specifically, the input unit 107 may include a touch panel 1071 and other input devices 1072. The touch panel 1071, also known as a touch screen, can collect user touch operations (such as operations performed by the user using the palm, finger or suitable accessories on the touch panel 1071 or near the touch panel 1071) and drive the corresponding connection device according to a pre-set program. The touch panel 1071 may include two parts: a touch detection device 1073 and a touch controller 1074. Among them, the touch detection device 1073 detects the user's touch direction and detects the signal caused by the touch operation, and transmits the signal to the touch controller 1074; the touch controller 1074 receives the touch information from the touch detection device 1073 and converts it into touch point coordinates, which are then sent to the control module 110. It can also receive commands sent by the control module 110 and execute them. In addition to the touch panel 1071, the input unit 107 may also include other input devices 1072. Specifically, other input devices 1072 may include but are not limited to one or more of a remote control operating handle, etc., which are not specifically limited here.

[0040] Furthermore, the touch panel 1071 may cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the control module 110 to determine the type of touch event. Subsequently, the control module 110 provides corresponding visual output on the display panel 1061 according to the type of touch event. Figure 1 In the embodiment, the touch panel 1071 and the display panel 1061 are used as two independent components to realize input and output functions respectively. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to realize input and output functions, which is not limited here.

[0041] The control module 110 is the control center of the multi-legged robot 100. It uses various interfaces and lines to connect the various components of the entire multi-legged robot 100, and controls the multi-legged robot 100 as a whole by running or executing software programs stored in the storage unit 105 and calling data stored in the storage unit 105.

[0042] The power supply 111 is used to supply power to each component. The power supply 111 may include a battery and a power control board. The power control board is used to control functions such as battery charging, discharging, and power consumption management. Figure 1 In the illustrated embodiment, the power supply 111 is electrically connected to the control module 110. In other embodiments, the power supply 111 may also be electrically connected to the sensor unit 103 (such as a camera, radar, speaker, etc.) and the motor 1012. It should be noted that each component can be connected to a different power supply 111 or powered by the same power supply 111.

[0043] Based on the above embodiments, specifically, in some embodiments, a terminal device can be used to establish a communication connection with the multi-legged robot 100. When the terminal device and the multi-legged robot 100 are communicating, the terminal device can send command information to the multi-legged robot 100. The multi-legged robot 100 can receive the command information through the communication unit 102, and upon receiving the command information, can transmit the command information to the control module 110, so that the control module 110 can process the command information to obtain a target speed value. The terminal device includes, but is not limited to, a mobile phone with an image capture function, a tablet computer, a server, a personal computer, a wearable smart device, and other electrical devices.

[0044] The command information can be determined based on preset conditions. In one embodiment, the multi-legged robot 100 may include a sensing unit 103, which can generate command information based on the current environment of the multi-legged robot 100. The control module 110 can use the command information to determine whether the current speed of the multi-legged robot 100 meets the corresponding preset conditions. If so, the multi-legged robot 100 will maintain its current speed and gait. If not, the multi-legged robot 100 will be controlled to move at the target speed and gait based on the corresponding preset conditions. Environmental sensors may include temperature sensors, air pressure sensors, visual sensors, and sound sensors. The command information may include temperature information, air pressure information, image information, and sound information. The communication method between the environmental sensors and the control module 110 may be wired communication or wireless communication. Wireless communication methods include, but are not limited to, wireless networks, mobile communication networks (3G, 4G, 5G, etc.), Bluetooth, and infrared.

[0045] Figure 3 This is a flow chart of a method for determining a three-dimensional position of a target according to one embodiment of the present application.

[0046] like Figure 3 As shown, on the one hand, the present application provides a method for determining the three-dimensional position of a target, which may include: step S1, obtaining a theoretical transformation formula that characterizes the theoretical transformation relationship between the pixel coordinate system and the world coordinate system; step S2, judging the terrain state of the target location, and setting the vertical coordinate value of the target's center of mass descent point in the world coordinate system according to the terrain state, wherein the center of mass descent point is the mapping point of the target's center of mass on the contact surface between the target and the ground; step S3, obtaining a target transformation formula according to the vertical coordinate value and the theoretical transformation formula, wherein the target transformation formula is used to characterize the actual transformation relationship between the pixel coordinate system and the world coordinate system for the terrain state; step S4, substituting the pixel coordinates of the center of mass descent point into the target transformation formula to determine the three-dimensional position information of the target in the world coordinate system.

[0047] The following will be combined Figures 4 to 7B The specific steps of the above method for determining the three-dimensional position of the target are described in detail.

[0048] Figure 4 It is a flow chart for generating a theoretical transformation formula according to one embodiment of the present application.

[0049] like Figure 4 As shown, first, obtain the offline calibration result of the camera, that is, the rotation transformation parameters from the camera coordinate system to the carrier coordinate system and translation transformation parameters Specifically, the carrier coordinate system is also called the IMU (Inertial Measurement Unit) coordinate system. The IMU is a device for measuring the three-axis attitude angle and acceleration of an object, including three single-axis accelerometers and three single-axis gyroscopes. The accelerometer is used to detect the acceleration signal of the object in the three independent axes of the carrier coordinate system, and the gyroscope detects the angular velocity signal of the object relative to the world coordinate system, and measures the angle and acceleration of the object in three-dimensional space. It should be noted that the IMU is arranged in the carrier device of the camera and is located at the center of mass of the carrier device; the camera is arranged on the outside of the carrier device. In this application, the carrier device of the camera is a robot horse, and the camera is arranged at the front chest position of the robot horse.

[0050] In some embodiments, the rotation transformation parameters from the world coordinate system input by the odometer module to the carrier coordinate system are obtained. and translation transformation parameters It should be noted that the odometer module converts the world coordinate system to the carrier coordinate system based on the collected pose information (including angle and acceleration) and speed information (including forward speed and turning speed) of the camera carrier device, and outputs the rotation transformation parameters from the world coordinate system to the carrier coordinate system. and translation transformation parameters The process can be implemented based on the existing odometer technology, and the specific implementation method is not limited here.

[0051] In some embodiments, a camera captures a camera image containing a target, and a perception module detects the pixel coordinates (u, v) of the target's center of mass descent point, where the center of mass descent point is a mapping point of the target's center of mass on the contact surface between the target and the ground; wherein u represents the row value of the target's center of mass descent point in the pixel coordinate system, and v represents the column value of the target's center of mass descent point in the pixel coordinate system.

[0052] In some embodiments, due to the rotation transformation parameters from the world coordinate system input by the odometer module to the vehicle coordinate system and translation transformation parameters There is a gap between the sampling frequency of the pixel coordinates of the centroid drop point detected by the perception module. In order to avoid calculation errors, the rotation transformation parameters from the world coordinate system to the carrier coordinate system are collected. and translation transformation parameters And the pixel coordinates of the centroid drop point, it is necessary to align the timestamps of the two.

[0053] Furthermore, according to the rotation transformation parameters from the camera coordinate system to the carrier coordinate system in the offline calibration result and translation transformation parameters And the rotation transformation parameters from the world coordinate system input by the odometer module to the carrier coordinate system and translation transformation parameters Get the rotation transformation parameters from the world coordinate system to the camera coordinate system And the translation transformation parameters from the world coordinate system to the camera coordinate system

[0054] Furthermore, let the three-dimensional coordinate matrix of the target in the world coordinate system be The rotation transformation parameters from the world coordinate system to the camera coordinate system are as follows: and shift transformation parameters The transformation of can obtain the first transformation relationship between the world coordinate system and the camera coordinate system:

[0055]

[0056] In formula (1), Indicates P odom The camera coordinate matrix in the camera coordinate system. The acquisition of the first transformation relationship paves the way for the subsequent steps to transform the pixel coordinate system into the world coordinate system according to the transformation relationship between the camera coordinate system and the pixel coordinate system.

[0057] Figure 5 Schematic diagram of a pinhole camera model according to one embodiment of the present application.

[0058] like Figure 5 As shown, this application takes a pinhole camera as an example. In the pinhole camera model, O1 represents the origin of the image coordinate system, and the image coordinate system has an image horizontal coordinate axis x_img axis and an image vertical coordinate axis y_img axis; O2 represents the projection of the target point on the camera optical axis; O3 represents the origin of the pixel coordinate system, and the pixel coordinate system has a u-axis representing the row value of the centroid descending point in the pixel coordinate system and a v-axis representing the column value of the centroid descending point in the pixel coordinate system; c represents the origin of the camera coordinate system, which has the camera transverse coordinate axis x_cam, the camera longitudinal coordinate axis y_cam and the camera vertical coordinate axis z_cam; P cam It's P odom The camera coordinate matrix in the camera coordinate system, Figure 5 As shown in (x cam ,y cam , z cam ) is P cam Coordinate form of P uv =(u, v) represents the pixel coordinates of the target's centroid drop point. cam and P uv These are two representations of the same point in different coordinate systems.

[0059] Specifically, under ideal imaging conditions, the origin of the image coordinate system and the origin of the camera coordinate system coincide, both being O1. At this point, let the pixel coordinates of the target's centroid descending point be (u, v). According to the principle of similar triangles, the triangle ΔO c O1P uv and triangle ΔO c O2P cam With the following relationship:

[0060]

[0061] In formula group (2), f represents the focal length of the camera.

[0062] However, in actual situations, the origin of the image coordinate system is often at position O3, so the coordinates of the image center point (i.e., the origin of the image coordinate system O1 under ideal imaging conditions) are (c x , c y ), then formula group (2) can be expressed as:

[0063]

[0064] In some embodiments, the pixel size of the camera is λ mm / pixel, where the pixel size represents the size of a pixel. The pixel size and the number of pixels (i.e., resolution) together determine the size of the camera's target surface. Furthermore, formula group (3) can be expressed in matrix form as follows:

[0065]

[0066] Furthermore, by simplifying formula (4), we can obtain the second transformation relationship that represents the transformation relationship between the camera coordinate system and the pixel coordinate system:

[0067]

[0068] In formula (5), s represents the scale parameter, and K represents the intrinsic parameter matrix of the camera. K can be obtained through camera calibration or factory settings of the camera, that is, K is known based on the properties of the camera.

[0069] Combining formula (1) and formula (5), that is, substituting formula (1) into formula (5), the coordinate transformation between the world coordinates and the pixel coordinates can be completed, and then the theoretical transformation formula that characterizes the theoretical transformation relationship between the pixel coordinate system and the world coordinate system can be obtained, which provides a theoretical basis for obtaining the three-dimensional position coordinates of the target in the world coordinate system according to the pixel points in the subsequent steps. The theoretical transformation formula is:

[0070]

[0071] In some implementations, formula (6) may be modified to obtain:

[0072]

[0073] Furthermore, formula (7) is simplified, that is, the first simplified matrix is Let the second simplified matrix Available:

[0074]

[0075] In formula (8), a0 represents the value of the first row of matrix A, a1 represents the value of the second row of matrix A, a2 represents the value of the third row of matrix A, b0 represents the value of the first row of matrix B, b1 represents the value of the second row of matrix B, and b2 represents the value of the third row of matrix B.

[0076] Furthermore, in formula (8), from the third row of the matrix, we can get: sa2-b2=z, that is,

[0077] Furthermore, Substituting into formula (8), we can obtain the matrix expansion result of the theoretical transformation formula:

[0078]

[0079] Figure 6A It is a schematic diagram showing that the target and the camera carrying device are on the same plane without slope; Figure 6B It is a schematic diagram showing that the target and the camera-carrying device are on the same plane with a slope; Figure 7A It is a schematic diagram in which the plane where the target is located is higher than the plane where the camera carrying device is located; Figure 7B This is a schematic diagram showing that the plane where the target is located is lower than the plane where the camera supporting device is located.

[0080] like Figure 6A and Figure 6B As shown, Figure 6A It shows that the target and the camera carrying device are on the same plane, and the plane has no slope. Figure 6B The target and the camera carrier are located on the same plane, and the plane has a certain slope. Specifically, the rotation transformation parameters from the world coordinate system input by the odometer module to the carrier coordinate system are: and translation transformation parameters It can characterize the terrain conditions between the target location and the camera carrying device, including whether the target location and the camera carrying device location are in the same plane, and the slope angle of the plane when the two are in the same plane. When the data input by the odometer module shows that the target and the camera carrying device are in the same plane without slope (that is, the slope angle is zero), the vertical coordinate value of the target's center of mass descent point in the world coordinate system is set to zero, that is, z = 0. When the data input by the odometer module shows that the target and the camera carrying device are in the same plane with a slope, the world coordinate system can be rotated according to the value of the slope angle to use the plane as the horizontal and vertical planes of the world coordinate system, that is, the xy plane; at this time, the vertical coordinate value of the target's center of mass descent point in the world coordinate system can be set to zero, that is, z = 0.

[0081] Based on this, the actual scale value of the scale parameter can be obtained Substituting the actual scale value into formula (9), we can obtain the target transformation formula when the camera's carrier and the target are in the same plane in the world coordinate system:

[0082] The values ​​of a0, a1, a2, b0, b1 and b2 are known and can be obtained by mathematically transforming the first simplified matrix A and the second simplified matrix B, which will not be detailed here.

[0083] Specifically, formula (10) is applicable to scenes with flat terrain, such as indoor large flat floors, parking lots, outdoor squares, lawns, etc. In these scenes, the target position can be estimated efficiently, simply and at low cost.

[0084] like Figure 7A and Figure 7B As shown, Figure 7A The target is on the steps, while the camera-carrying device is on the ground. The vertical coordinate value of the target's center of mass descending point in the world coordinate system is higher than the vertical coordinate value of the camera-carrying device in the world coordinate system. Figure 7B The target is in a pit, and the camera carrying device is on the ground. The vertical coordinate value of the target in the world coordinate system is lower than the vertical coordinate value of the center of mass drop point of the camera carrying device in the world coordinate system. In the above two examples, the terrain conditions between the position of the target input by the odometer module and the position of the camera carrying device are: the two are not in the same plane in the world coordinate system. At this time, the vertical coordinate value z of the center of mass drop point of the target in the world coordinate system can be obtained based on the experience value or the detection value, which is not limited here. After obtaining the vertical coordinate value z of the center of mass drop point of the target in the world coordinate system, the actual scale value s0 of the scale parameter can be obtained, and the actual scale value s0 is substituted into formula (9) to obtain the target transformation formula when the camera carrying device and the target are not in the same plane in the world coordinate system:

[0085]

[0086] In some embodiments, the three-dimensional position of the target can be obtained specifically according to the actual terrain conditions, making the present application universal.

[0087] It should be noted that the differences between formula (10) and formula (11) are due to the different actual scale values ​​of S, and their essence is the same as that shown in formula (9).

[0088] In some embodiments, the target may be a pedestrian, and the camera carrier may be a robotic horse, which is not limited here.

[0089] In some embodiments, any point at a height of the target other than the point where the centroid descends can be used as an estimation point, which is not limited here.

[0090] In some embodiments, the pixel coordinates of the target's centroid descent point are substituted into the target transformation formula, i.e., formula (9), to determine the three-dimensional position information of the target in the world coordinate system.

[0091] According to at least one embodiment of the present application, a method for determining the three-dimensional position of a target is provided. The vertical coordinate value of the target's centroid descent point in the world coordinate system is set based on the terrain state of the target's location. The target transformation formula for the target is determined for different terrain states. The pixel coordinates of the target's centroid descent point are then substituted into the world coordinate system to obtain the target's three-dimensional position coordinates in the world coordinate system. The present application provides a method that avoids the cost of collecting large amounts of true depth data required by existing technologies, has low requirements for the operating platform, and is universally applicable.

[0092] Figure 8 It is a schematic structural diagram of a device for determining the three-dimensional position of a target according to one embodiment of the present application.

[0093] like Figure 8As shown, the present application provides another aspect of a device 200 for determining a three-dimensional position of a target, which may include: a theoretical transformation formula generation module 210, a vertical coordinate value setting module 220, a target transformation formula generation module 230, and a three-dimensional position determination module 240. The theoretical transformation formula generation module 210 is used to obtain a theoretical transformation formula that represents the theoretical transformation relationship between the pixel coordinate system and the world coordinate system; the vertical coordinate value setting module 220 is used to determine the terrain state of the target location and set the vertical coordinate value of the target's centroid descent point in the world coordinate system based on the terrain state, wherein the centroid descent point is the mapping point of the target's centroid on the contact surface between the target and the ground; the target transformation formula generation module 230 is used to obtain a target transformation formula based on the vertical coordinate value and the theoretical transformation formula, wherein the target transformation formula is used to represent the actual transformation relationship between the pixel coordinate system and the world coordinate system for the terrain state; and the three-dimensional position determination module 240 is used to substitute the pixel coordinates of the centroid descent point into the target transformation formula to determine the three-dimensional position information of the target in the world coordinate system.

[0094] In some embodiments, the execution steps of the theoretical transformation formula generation module 210 include: generating a first transformation formula representing the transformation relationship between the world coordinate system and the camera coordinate system; generating a second transformation formula representing the transformation relationship between the pixel coordinate system and the camera coordinate system based on the principle of similar triangles in the camera model of the camera, wherein the camera is used to shoot the target; and integrating the first transformation formula and the second transformation formula to obtain a theoretical transformation formula representing the theoretical transformation relationship between the pixel coordinate system and the world coordinate system, wherein the theoretical transformation formula includes a scale parameter.

[0095] In some embodiments, the theoretical transformation formula is: Among them, s is the scale parameter, u is the row value of the centroid drop point in the pixel coordinate system, v is the column value of the centroid drop point in the pixel coordinate system, K is the intrinsic parameter matrix of the camera, is the rotation transformation parameter between the camera coordinate system and the world coordinate system, P odom is the position matrix of the target in the world coordinate system, is the translation transformation parameter between the camera coordinate system and the world coordinate system.

[0096] In some embodiments, the execution steps of the vertical coordinate value setting module 220 include: judging the terrain state of the target location to determine whether the target location and the camera's supporting device are in the same plane; and in response to the judgment result that the target location and the camera's supporting device are in the same plane, setting the vertical coordinate value of the target's center of mass descent point in the world coordinate system to zero, or in response to the judgment result that the target location and the camera's supporting device are not in the same plane, setting the vertical coordinate value of the center of mass descent point in the world coordinate system to an empirical value, wherein the center of mass descent point is the mapping point of the target's center of mass on the contact surface between the target and the ground.

[0097] In some embodiments, the execution steps of the target transformation formula generation module 230 include: performing matrix expansion processing on the theoretical transformation formula to obtain a matrix expansion result; substituting the vertical coordinate value of the target in the world coordinate system into the matrix expansion result to calculate the actual scale value of the scale parameter; and substituting the actual scale value into the matrix expansion result to obtain a target transformation formula that represents the actual transformation relationship between the pixel coordinate system and the world coordinate system.

[0098] According to at least one embodiment of the present application, a device for determining the three-dimensional position of a target is provided. The vertical coordinate value of the target's centroid descent point in the world coordinate system is set based on the terrain state of the target's location. The target transformation formula for the target is determined for different terrain states. The pixel coordinates of the target's centroid descent point are then substituted into the world coordinate system to obtain the target's three-dimensional position coordinates in the world coordinate system. The present application provides a method that avoids the cost of collecting large amounts of true depth data required by existing technologies, has low requirements for the operating platform, and is universally applicable.

[0099] Figure 9 is a schematic diagram of an electronic device according to one embodiment of the present application.

[0100] like Figure 9 According to another aspect of the present application, an electronic device is provided. The electronic device may include one or more processors and one or more memories. The memories may store computer-readable code that, when executed by the one or more processors, may execute the method for determining the three-dimensional position of a target as described above.

[0101] The method or device according to the embodiment of the present application can also be used by Figure 9 The electronic device architecture shown in FIG. Figure 9As shown, the electronic device may include a bus 301, one or more CPUs 302, a read-only memory (ROM) 303, a random access memory (RAM) 304, a communication port 305 connected to a network, an input / output component 306, a hard disk 307, and the like. The storage device in the electronic device, such as the ROM 303 or the hard disk 307, can store various data or files used for processing and communication of the method for determining the three-dimensional position of the target provided in this application, as well as program instructions executed by the CPU. The method for determining the three-dimensional position of the target may, for example, include obtaining a theoretical transformation formula that characterizes the theoretical transformation relationship between the pixel coordinate system and the world coordinate system; determining the terrain state of the target location, and setting the vertical coordinate value of the target's centroid descent point in the world coordinate system according to the terrain state, wherein the centroid descent point is the mapping point of the target's centroid on the contact surface between the target and the ground; obtaining a target transformation formula based on the vertical coordinate value and the theoretical transformation formula, wherein the target transformation formula is used to characterize the actual transformation relationship between the pixel coordinate system and the world coordinate system for the terrain state; substituting the pixel coordinates of the centroid descent point into the target transformation formula to determine the three-dimensional position information of the target in the world coordinate system. Furthermore, the electronic device may further include a user interface 308. Figure 9 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 9 One or more components of an electronic device are shown.

[0102] Figure 10 is a schematic diagram of a storage medium according to one embodiment of the present application.

[0103] like Figure 10 , a storage medium 400 according to one embodiment of the present application is shown. Computer-readable instructions are stored on the storage medium 400. When the computer-readable instructions are executed by a processor, the method for determining the three-dimensional position of a target according to an embodiment of the present application described with reference to the above figures can be executed. The storage medium 400 includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc.

[0104] In addition, according to the embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the present application provides a non-transitory machine-readable storage medium, which stores machine-readable instructions, and the machine-readable instructions can be executed by a processor to execute instructions corresponding to the method steps provided in the present application, such as: obtaining a theoretical transformation formula that characterizes the theoretical transformation relationship between the pixel coordinate system and the world coordinate system; judging the terrain state of the target location, and setting the vertical coordinate value of the target's centroid descent point in the world coordinate system according to the terrain state, wherein the centroid descent point is the mapping point of the target's centroid on the contact surface between the target and the ground; obtaining a target transformation formula based on the vertical coordinate value and the theoretical transformation formula, wherein the target transformation formula is used to characterize the actual transformation relationship between the pixel coordinate system and the world coordinate system for the terrain state; substituting the pixel coordinates of the centroid descent point into the target transformation formula to determine the three-dimensional position information of the target in the world coordinate system. When the computer program is executed by the central processing unit (CPU), the above functions defined in the method of the present application are executed.

[0105] The methods, apparatuses, and devices of the present application may be implemented in many ways. For example, the methods, apparatuses, and devices of the present application may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present application are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present application may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present application. Therefore, the present application also covers recording media that store programs for executing the methods according to the present application.

[0106] In addition, the parts of the above technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive redundancy.

[0107] The above description is merely an embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of protection involved in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the technical concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for determining a three-dimensional position of a target, characterized in that: include: Obtaining a theoretical transformation formula representing a theoretical transformation relationship between a pixel coordinate system and a world coordinate system; Determining the terrain state of the target location, and setting the vertical coordinate value of the target's center of mass drop point in the world coordinate system according to the terrain state, wherein the center of mass drop point is the mapping point of the target's center of mass on the contact surface between the target and the ground; Obtaining a target transformation formula according to the vertical coordinate value and the theoretical transformation formula, wherein the target transformation formula is used to represent an actual transformation relationship between the pixel coordinate system and the world coordinate system for the terrain state; and Substituting the pixel coordinates of the centroid descent point into the target transformation formula to determine the three-dimensional position information of the target in the world coordinate system; The theoretical transformation formula is: Where s is the scale parameter, u is the row value of the centroid drop point in the pixel coordinate system, v is the column value of the centroid drop point in the pixel coordinate system, and K is the intrinsic parameter matrix of the camera. is the rotation transformation parameter between the camera coordinate system and the world coordinate system, P odom is the position matrix of the target in the world coordinate system, is the translation transformation parameter between the camera coordinate system and the world coordinate system.

2. The method for determining the three-dimensional position of a target according to claim 1, wherein: The obtaining of the theoretical transformation formula representing the theoretical transformation relationship between the pixel coordinate system and the world coordinate system includes: Generate a first transformation relationship expression representing the transformation relationship between the world coordinate system and the camera coordinate system; generating a second transformation relationship expression representing a transformation relationship between the pixel coordinate system and the camera coordinate system based on a similar triangle principle in a camera model of a camera used to photograph the target; and The first transformation relationship and the second transformation relationship are integrated to obtain a theoretical transformation relationship representing a theoretical transformation relationship between the pixel coordinate system and the world coordinate system, wherein the theoretical transformation relationship includes a scale parameter.

3. The method for determining the three-dimensional position of a target according to claim 2, wherein: The determining of the terrain state of the target location and setting the vertical coordinate value of the target's center of mass descent point in the world coordinate system according to the terrain state includes: Determining the terrain state of the target location to determine whether the target location and the camera carrying device location are on the same plane; and In response to the judgment result that the position of the target and the position of the camera's supporting device are in the same plane, the vertical coordinate value of the target's center of mass descent point in the world coordinate system is set to zero, or in response to the judgment result that the target's position and the camera's supporting device are not in the same plane, the vertical coordinate value of the center of mass descent point in the world coordinate system is set to an empirical value, wherein the center of mass descent point is the mapping point of the target's center of mass on the contact surface between the target and the ground.

4. The method for determining the three-dimensional position of a target according to claim 3, wherein: The step of obtaining a target transformation formula according to the vertical coordinate value and the theoretical transformation formula includes: Performing matrix expansion processing on the theoretical transformation formula to obtain a matrix expansion result; Substituting the vertical coordinate value of the centroid descent point in the world coordinate system into the matrix expansion result to calculate the actual scale value of the scale parameter; and The actual scale value is substituted into the matrix expansion result to obtain a target transformation formula representing the actual transformation relationship between the pixel coordinate system and the world coordinate system.

5. A device for determining a three-dimensional position of a target, characterized in that: include: A theoretical transformation formula generation module is used to obtain a theoretical transformation formula representing the theoretical transformation relationship between the pixel coordinate system and the world coordinate system; a vertical coordinate value setting module, configured to determine the terrain state of the target location and set the vertical coordinate value of the target's centroid descent point in the world coordinate system according to the terrain state, wherein the centroid descent point is a mapping point of the target's centroid on the contact surface between the target and the ground; a target transformation formula generation module, configured to obtain a target transformation formula based on the vertical coordinate value and the theoretical transformation formula, wherein the target transformation formula is used to represent an actual transformation relationship between the pixel coordinate system and the world coordinate system for the terrain state; and a three-dimensional position determination module, configured to substitute the pixel coordinates of the centroid descent point into the target transformation formula to determine the three-dimensional position information of the target in the world coordinate system; The theoretical transformation formula is: Where s is the scale parameter, u is the row value of the centroid drop point in the pixel coordinate system, v is the column value of the centroid drop point in the pixel coordinate system, and K is the intrinsic parameter matrix of the camera. is the rotation transformation parameter between the camera coordinate system and the world coordinate system, P odom is the position matrix of the target in the world coordinate system, is the translation transformation parameter between the camera coordinate system and the world coordinate system.

6. The device for determining the three-dimensional position of a target according to claim 5, wherein: The execution steps of the theoretical transformation formula generation module include: Generate a first transformation relationship expression representing the transformation relationship between the world coordinate system and the camera coordinate system; generating a second transformation relationship expression representing a transformation relationship between the pixel coordinate system and the camera coordinate system based on a similar triangle principle in a camera model of a camera used to photograph the target; and The first transformation relationship and the second transformation relationship are integrated to obtain a theoretical transformation relationship representing a theoretical transformation relationship between the pixel coordinate system and the world coordinate system, wherein the theoretical transformation relationship includes a scale parameter.

7. The device for determining a three-dimensional position of a target according to claim 6, wherein: The execution steps of the vertical coordinate value setting module include: Determining the terrain state of the target location to determine whether the target location and the camera carrying device location are on the same plane; and In response to the judgment result that the position of the target and the position of the camera's supporting device are in the same plane, the vertical coordinate value of the target's center of mass descent point in the world coordinate system is set to zero, or in response to the judgment result that the target's position and the camera's supporting device are not in the same plane, the vertical coordinate value of the center of mass descent point in the world coordinate system is set to an empirical value, wherein the center of mass descent point is the mapping point of the target's center of mass on the contact surface between the target and the ground.

8. The device for determining the three-dimensional position of a target according to claim 7, wherein: The target transformation formula generation module executes the following steps: Performing matrix expansion processing on the theoretical transformation formula to obtain a matrix expansion result; Substituting the vertical coordinate value of the centroid descent point in the world coordinate system into the matrix expansion result to calculate the actual scale value of the scale parameter; and The actual scale value is substituted into the matrix expansion result to obtain a target transformation formula representing the actual transformation relationship between the pixel coordinate system and the world coordinate system.

9. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for determining the three-dimensional position of a target as described in any one of claims 1 to 4 is implemented.

Images