Self-feedback target point three-dimensional coordinate acquisition method based on monocular and binocular vision

Abstract

The invention discloses a self-feedback target point three-dimensional coordinate acquisition method based on monocular and binocular vision. According to the method, images are collected through a monocular camera, and accurate pose information of the camera during shooting of each image is obtained by using a specific three-dimensional manual mark. Based on a group of images which are shot by rotating around a plumb line, coordinates of a target point on the axis are calculated by combining a binocular vision principle, and horizontal two-direction coordinates of the target point are calculated by utilizing a monocular vision principle and camera pose information, so that acquisition of three-dimensional coordinates is realized. In addition, abnormal data are effectively eliminated through a statistical filtering technology, the accuracy of coordinate acquisition and the robustness of the algorithm are improved, and a reliability index capable of representing an algorithm result is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and engineering structure digital image detection, and in particular to a method for acquiring three-dimensional coordinates of a target point based on monocular and binocular vision with self-feedback. Background Art

[0002] my country's civil engineering structure construction is gradually shifting from the centralized construction stage to the service and operation stage, facing the challenge of aging. In particular, typical engineering structures such as residential buildings built in the last century are increasingly exposed to safety hazards due to factors such as material aging, environmental erosion, sudden loads, and improper use and management. Structural health monitoring can effectively and quantitatively obtain the response of the structure under complex working conditions and reflect the performance status of the structure in real time, which is of great significance for preventing the above-mentioned hidden dangers. Accurately obtaining the displacement and deformation of engineering structures is one of the key tasks of structural health monitoring. Although traditional displacement measurement methods (such as displacement meters, total stations, etc.) are mature in technology, they are costly and time-consuming to operate, and are difficult to meet the needs of large-scale and high-frequency monitoring. In view of this, finding efficient and economical monitoring methods has become a hot topic in current research. The rapid development of computer vision technology, especially its close integration with drone platforms, provides a new idea for solving this problem. The monocular camera positioning method can monitor structures from multiple angles and realize comprehensive data collection due to its effective integration with drone technology. At the same time, it has automation and intelligent analysis capabilities, making the acquisition and processing of monitoring data more efficient.

[0003] Existing research mainly focuses on two directions: point cloud reconstruction method and positioning technology based on manual marking. The former usually uses technologies such as drone tilt photography to construct point cloud models. This method has significant advantages in obtaining the three-dimensional coordinates of a large number of measurement points on the surface of the structure. However, this technology also has some limitations: ① The reconstruction process is highly dependent on the texture information of the object surface, and in the process of corner detection and matching, insufficient robustness may lead to a decrease in point cloud accuracy and prone to deviation and noise; ② It is often necessary to manually select specific points from the reconstructed point cloud model for positioning or length measurement, which limits the improvement of its automation level; ③ Camera positioning depends on the GPS system, and its accuracy is still insufficient, especially in the elevation direction, the error may reach about 10 cm (reference: Chun-Xu Qu, Jin-Zhao Jiang, Ting-Hua Yi, et al. Computer vision-based 3D coordinate acquisition of surface feature points of building structures [J]. Engineering Structures, 2024, 300: 117212.). The latter usually calculates the position of the camera relative to each Aruco code in the scene, and solves the relative position between all Aruco codes in the scene to achieve positioning (reference: Wang Yunfeng. Camera calibration and indoor positioning system based on Aruco code [D]. Zhejiang University, 2023.). In addition, some scholars have proposed a method of combining position estimation with binocular vision technology to perform three-dimensional positioning of the target point through a monocular camera (reference: Finding the world coordinates of a specified point based on the camera pose and its python implementation. https: / / www.jianshu.com / p / 77f7c0cd9ec7.). However, the above method has the following shortcomings: ① In the process of identifying Aruco codes and their corner points, problems such as misidentification, missed identification, and border identification deviation often occur, which affects the overall robustness; ② If the method of solving the relative position between two Aruco codes is adopted, the error may further accumulate; ③ If only a monocular camera is used to imitate binocular vision for one-time three-dimensional positioning, the requirements for the shooting angle are extremely strict. If the angle of the intersection ray is small, the error in the depth direction will increase significantly. Summary of the invention

[0004] The purpose of the present invention is to provide a method for acquiring three-dimensional coordinates of a target point based on monocular or binocular vision with self-feedback in view of the deficiencies in the prior art.

[0005] The object of the present invention is achieved through the following technical solution: A method for acquiring three-dimensional coordinates of a target point based on self-feedback of monocular and binocular vision, comprising the following steps:

[0006] (1) Setting at least one reference Aruco marker at the reference point and setting a target Aruco marker at the target point;

[0007] (2) In the horizontal plane where the target point is located, rotate and shoot around an axis perpendicular to the plane to obtain multiple photos containing the target Aruco marker and at least one reference Aruco marker; based on the coordinates of the corner points of the reference Aruco marker in the photo, obtain the position and posture of the camera relative to the reference Aruco marker during shooting; determine the ray pointing from the camera to the target point based on the position and posture information and the camera internal parameters; calculate the elevation coordinate z value of the midpoint of the shortest line segment between two rays, and take the average value as the elevation coordinate Z of the target point; determine the plane equation of the horizontal plane S passing through the point (0,0,Z);

[0008] (3) Shooting outside the horizontal plane where the target point is located, obtaining multiple photos containing the target Aruco marker and at least one reference Aruco marker; based on the coordinates of the corner points of the reference Aruco marker in the photos, obtaining the position and posture of the camera relative to the reference Aruco marker during shooting; determining the ray pointing from the camera to the target point according to the position and posture information and the camera internal parameters; and calculating the three-dimensional coordinates (x, y, Z) of the intersection of the ray and the horizontal plane S;

[0009] (4) Calculate multiple three-dimensional coordinate values ​​in the same way as step (3), and take the average value as the three-dimensional coordinate (X, Y, Z) of the target point.

[0010] Furthermore, the step (2) further includes:

[0011] In the step of "calculating the elevation coordinate z value of the midpoint of the shortest line segment between two rays", the situation where the intersection angle between two rays does not satisfy the first preset threshold α is excluded.

[0012] Furthermore, the step (2) further includes:

[0013] In the step of "calculating the elevation coordinate z value of the midpoint of the shortest line segment between any two rays", the case where the length of the shortest line segment between any two rays is greater than the second preset threshold β is excluded.

[0014] Furthermore, it also includes:

[0015] For the elevation coordinate z value solved in step (2), the 3σ criterion is used to eliminate outliers, and the average value of the remaining data is taken as the elevation coordinate Z of the target point.

[0016] Furthermore, it also includes:

[0017] For the obtained three-dimensional coordinates (x, y, Z), the 3σ criterion is used to eliminate outliers, and the average value of the remaining data is taken as the three-dimensional coordinates (X, Y, Z) of the target point.

[0018] Furthermore, the standard deviation σ of the remaining data is calculated as a reliability indicator feedback of the result.

[0019] Furthermore, the shooting outside the horizontal plane where the target point is located includes:

[0020] Shoot directly above or below your target point.

[0021] The present invention also provides a device for acquiring three-dimensional coordinates of a target point based on monocular and binocular vision, comprising:

[0022] The marker setting module is used to set a number of Aruco markers at the reference point, which are called reference Aruco markers; and set Aruco markers at the target point, which are called target Aruco markers;

[0023] The first calculation module is used to: in the horizontal plane where the target point is located, rotate and shoot around an axis perpendicular to the plane to obtain multiple photos containing the target Aruco mark and at least one reference Aruco mark; based on the corner coordinates of the reference Aruco mark in the photo, obtain the position and posture of the camera relative to the reference Aruco mark during shooting; determine the ray pointing from the camera to the target point according to the position and posture information and the camera internal parameters; calculate the elevation coordinate z value of the midpoint of the shortest line segment between two rays, and take the average value as the elevation coordinate Z of the target point; determine the plane equation of the horizontal plane S passing through the point (0,0,Z);

[0024] The second calculation module is used to: take photos outside the horizontal plane where the target point is located to obtain multiple photos containing the target Aruco mark and at least one reference Aruco mark; based on the corner coordinates of the reference Aruco mark in the photos, obtain the position and posture of the camera relative to the reference Aruco mark during shooting; determine the ray pointing to the target point from the camera according to the position and posture information and the camera internal parameters; calculate the three-dimensional coordinates (x, y, Z) of the intersection of the ray and the horizontal plane S; similarly calculate multiple three-dimensional coordinate values, and take the average value as the three-dimensional coordinates (X, Y, Z) of the target point.

[0025] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned method for acquiring three-dimensional coordinates of a self-feedback target point based on monocular and binocular vision.

[0026] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for acquiring three-dimensional coordinates of a target point based on monocular and binocular vision and self-feedback is implemented.

[0027] The beneficial effects of the present invention are: ① Improved robustness: The accuracy of axial positioning using binocular vision in the same plane is used to improve the accuracy of positioning; according to the preset ray intersection angle, binocular images with high matching degree are automatically screened out to optimize the image matching process; after the monocular vision and binocular vision operations are completed, statistical noise reduction technology is used to remove abnormal data, significantly reducing the error. ② Improved accuracy: Under the same instrument use conditions, the method of the present invention is compared with the method in the prior art that realizes positioning by solving the relative posture between all Aruco codes in the scene, and the method of performing binocular vision positioning by only two images combined with posture estimation, and can reduce the error, especially the error in the depth direction, from the centimeter or even decimeter level to the millimeter level. ③ Reliability feedback mechanism: The present invention provides a quantitative indicator for the reliability of the final result by calculating the standard deviation of the data list, thereby further enhancing the reliability of the algorithm.

[0028] In addition, the present invention is suitable for combination with drone technology. It only requires one-time arrangement of permanent manual markings and planning of drone routes, and the existing camera can be used to quickly monitor the displacement of multiple target points. Compared with the several hours required to establish a high-precision point cloud model, the present invention only takes a few seconds to calculate each measuring point, realizing automatic identification of target points and avoiding the tedious manual point selection process based on point cloud data. Therefore, the present invention not only achieves a high degree of automation of the process, but also reduces costs and improves monitoring efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly explain the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings involved. It should be noted that the following drawings only show some embodiments of the present invention, and those skilled in the art can conceive other implementation methods based on these drawings without creative work.

[0030] Figure 1 An implementation scheme for obtaining photos in an embodiment of the present invention is shown. Figure 1 (a) shows the method of indoor point positioning using a monocular camera, while Figure 1 (b) shows the technology of mounting a monocular mapping camera on a UAV for outdoor point positioning;

[0031] Figure 2 Provided are an arrangement scheme for manually marking reference points and target points and examples of shooting angles in an embodiment of the present invention. Figure 2 (a) shows an example of a ring photo taken in a plane. Figure 2 (b) shows an example of a bird's-eye view of the plumb line.

[0032] Figure 3 The measurement point arrangement scheme of the indoor monocular camera experiment in the embodiment of the present invention is demonstrated;

[0033] Figure 4 Shows a list of results after binocular vision operations. Figure 4 (a) Figure 4 (b) and Figure 4 (c) in the figure corresponds to the results of point 1, point 2, and point 3. By applying the 3σ criterion to eliminate outliers, it can be seen that the data error along the axis in the dashed box is reduced to less than 1mm. The average value of these data will be used for subsequent monocular vision operations;

[0034] Figure 5 is a list of results after monocular vision operation. Figure 5 (a) Figure 5 (b) and Figure 5 (c) in the figure corresponds to the results of point 1, point 2, and point 3, respectively. After using the 3σ criterion to remove outliers and data in the solid-line box, it can be seen that the data errors in the two directions in the plane marked by the dotted box are as low as mm level. The average value of the data in the plane is used as the coordinates in the two directions. The standard deviation of the two columns of data is the reliability feedback indicator of the final result, reflecting the self-feedback capability of the entire process algorithm.

[0035] Figure 6 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0036] Figure 7 A schematic flow chart of a method for acquiring three-dimensional coordinates of a target point based on monocular and binocular vision with self-feedback provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0038] The present invention provides a method for acquiring three-dimensional coordinates of a target point based on self-feedback of monocular and binocular vision, such as Figure 7 As shown, the following steps are included:

[0039] (1) Setting at least one reference Aruco marker at the reference point and setting a target Aruco marker at the target point;

[0040] In one embodiment, a high-precision artificial three-dimensional marking system is produced: Figure 1 (a) and (b) in Figure 3, a high-precision cubic object is selected as a reference mark, and Aruco marks are pasted or laser-printed on each surface of the object to ensure that the three-dimensional coordinates of all corner points of the Aruco mark are accurately known in the cubic reference system during the production process. A flat Aruco mark can be used as a target mark. The reference mark and the target mark are fixed to the reference point and the target point respectively. One edge of the reference mark cube is along the plumb line direction; the target mark is set to be tilted at an angle of 45 degrees to the plumb line, and the mark can be clearly seen from the top. Specifically, the following sub-steps are included;

[0041] (1.1) Use high-quality, thick and smooth acrylic sheet, accurately process it into a 5-sided acrylic cube box with a side length of 21 cm to ensure the stability and accuracy of the reference mark attachment, and cut a 7 cm side length flat acrylic plate as the target mark attachment;

[0042] (1.2) Accurately paste or laser print a 2×2 arrangement of Aruco marker plates on the four sides and top of the cube. Ensure that each marker plate has a white border of sufficient width, that is, the side length of the marker plate is a certain proportion less than the corresponding side length of the cube to meet the visual identification requirements. Accurately paste or laser print an Aruco marker with a different ID number on the flat acrylic plate as the target marker; make a 2×2 Aruco marker plate with a side length of 9 cm, a spacing of 0.5 cm, and a 0.75 cm white border around the marker plate. Stick these different Aruco marker plates to different faces of the reference label cube, and stick an Aruco marker with a side length of 5 cm and a 1 cm white border around it with non-repeating numbers to the target label acrylic plate;

[0043] (1.3) Through high-precision measurement methods, the three-dimensional coordinates of each corner point of the Aruco marker in the cube reference system are recorded in detail to provide an accurate data basis for subsequent spatial positioning; these data need to be recorded and passed into the program;

[0044] (1.4) The reference mark and target mark are fixed to the reference point and target point respectively. One edge of the reference mark cube is along the plumb line. The target mark is set to be tilted at an angle of 45 degrees to the plumb line and can be clearly seen from the top.

[0045] (2) In the horizontal plane where the target point is located, rotate and shoot around an axis perpendicular to the plane to obtain multiple photos containing the target Aruco marker and at least one reference Aruco marker; based on the coordinates of the corner points of the reference Aruco marker in the photo, obtain the position and posture of the camera relative to the reference Aruco marker during shooting; determine the ray pointing from the camera to the target point based on the position and posture information and the camera internal parameters; calculate the elevation coordinate z value of the midpoint of the shortest line segment between two rays, and take the average value as the elevation coordinate Z of the target point; determine the plane equation of the horizontal plane S passing through the point (0,0,Z);

[0046] In one embodiment, it also includes: in the step of "calculating the elevation coordinate z value of the midpoint of the shortest line segment between two rays", excluding the situation where the intersection angle of two rays does not meet the first preset threshold value α; that is, the two rays are no longer involved in the step of "calculating the elevation coordinate z value of the midpoint of the shortest line segment between two rays".

[0047] In one embodiment, it also includes: in the step of "calculating the elevation coordinate z value of the midpoint of the shortest line segment between two rays", excluding the situation where the length of the shortest line segment between two rays is greater than the second preset threshold β; that is, the two rays are no longer involved in the step of "calculating the elevation coordinate z value of the midpoint of the shortest line segment between two rays".

[0048] In one embodiment, the method further includes: using one or two passes of the 3σ criterion to eliminate abnormal values ​​of the elevation coordinate z value solved in the step (2), and taking the average value of the remaining data as the elevation coordinate Z of the target point.

[0049] In one embodiment, step (2) includes the following sub-steps:

[0050] (2.1) Figure 2 As shown in (a), in the horizontal plane where the target point is located, rotate and shoot around an axis perpendicular to the plane to obtain a set of circular photos: when shooting, ensure that each photo clearly presents the target point mark and at least one reference point mark, so as to accurately calculate the posture information in subsequent processing; specifically, it includes the following sub-steps;

[0051] (2.1.1) Move the monocular camera precisely to a position on the same horizontal plane as the target point, ensuring that the camera can clearly aim at the target mark and that at least one reference mark is within the field of view for shooting; move the camera to a position about 3m away from the target point for shooting;

[0052] (2.1.2) With the plumb line at the target point as the rotation center, control the monocular camera to perform compound motion near the horizontal plane: first rotate 15°-20° to change the viewing angle, then revolve the same angle on a larger circular path, keeping the camera accurately aimed at the target point throughout the process and taking pictures;

[0053] (2.1.3) Repeat step (2.2) and take continuous photos around the plumb line at the target point until a set of circular photos is completed. To ensure the reliability and accuracy of data processing, it is recommended to take 4 or more photos, each of which must clearly show the target point mark and at least one reference point mark, so that they can be verified and possible bad data can be eliminated; in this case, 5 photos were actually taken.

[0054] (2.2) Using the images in step (2.1), traverse the photo pairs one by one, use the manual marks on the photos to calculate the pose of each photo, and then use binocular vision technology to determine the rays pointing to the common target point based on the pose information, solve and record the midpoint coordinates of the shortest line segment between the rays, and measure the ray spacing for subsequent analysis: For each image, use OpenCV to extract the Aruco marker corner points, estimate the camera pose based on the reference markers and the PNP algorithm, and map the target Aruco marker upper left corner point P to the three-dimensional space in combination with the camera internal parameters to form the current pose of the camera pointing to P. Rays are traversed through a pair of photos in pairs, and a preset angle threshold [θ1, θ2] (i.e., the first preset threshold) is set. If the intersection angle of the rays of the pair of photos does not meet the threshold, the pair is skipped to ensure that the binocular vision error is controllable; if the threshold is met, the shortest distance between the corresponding rays of the pair of photos is calculated. If the distance is less than the preset length threshold D (i.e., the second preset threshold), it is determined that the rays intersect well, and the elevation coordinate z of the midpoint of the shortest line segment in the direction of the plumb line is recorded in the data list data1 (i.e., the preset angle threshold and the preset length threshold must be met at the same time); specifically, the following sub-steps are included;

[0055] (2.2.1) For each image, you can use the opencv aruco.detectMarkers function to read the pixel coordinates of all the Aruco markers in the image and record them in a csv file in the form of a list;

[0056] (2.2.2) Pass the coordinates of all the Aruco marker corner points belonging to the benchmark marker obtained in step (2.2.1) into the solvePnP function of opencv to obtain the position and posture information of the camera relative to the benchmark marker during shooting;

[0057] (2.2.3) Using the camera pose data obtained in step (2.2.2) and the known camera intrinsic parameter matrix, map the upper left corner point P of the target Aruco marker obtained in step (2.2.1) to three-dimensional space to form a ray pointing from the camera in the current pose to P;

[0058] (2.2.4) Traverse a pair of photos pairwise, with a preset angle threshold [θ1, θ2] (i.e., the first preset threshold, preferably [30°, 150°]). If the included angle of the rays of this pair of photos is within this threshold, calculate the length value distance of the shortest line segment between the spatial rays corresponding to this pair of photos; if it is not within this threshold, skip it to ensure that the binocular vision error is controllable.

[0059] (2.2.5) Set a length threshold D (i.e., the second preset threshold, preferably 1 cm). If distance < D, it is considered that the intersection of this group of rays is good, and record the elevation coordinate z of the midpoint of this shortest line segment in the direction of the plumb line to the data list data1. If distance > D, it is determined that the binocular intersection is poor and skip without recording.

[0060] (2.2.6) Traverse all photo pairs and repeat steps (2.2.4) to (2.2.5) to obtain a data list data1 containing all spare elevation coordinate data.

[0061] (2.3) Statistically denoise and use the mean value of the remaining data as the elevation of the target point to obtain the plane equation of the horizontal plane where the target point is located: Use the 3σ criterion one or two times to remove outliers from the data list data1 obtained in step (2.2.6), and take the average value of the remaining data as the elevation coordinate Z of the target point, and find the plane equation of the horizontal plane S passing through this point Z; specifically, it includes the following sub-steps;

[0062] (2.3.1) Statistically denoise the data list obtained in step (2.2.6), use the 3σ criterion one or two times to remove outliers, and record the average value of the remaining data as the elevation coordinate Z of the target point; (in this embodiment, the 3σ criterion is used once for statistical noise reduction);

[0063] (2.3.2) Find the plane equation S of the horizontal plane passing through the point (0, 0, Z).

[0064] (3) Take pictures outside the horizontal plane where the target point is located to obtain multiple photos containing the target Aruco marker and at least one reference Aruco marker; based on the corner coordinates of the reference Aruco marker in the photo, obtain the pose of the camera relative to the reference Aruco marker during shooting; determine the ray of the camera pointing to the target point according to this pose information and the camera internal parameters; find the three-dimensional coordinates (x, y, Z) of the intersection point of this ray and the horizontal plane S; that is, find the horizontal coordinates, and at this time the Z value has been determined;

[0065] Among them, when taking pictures outside the horizontal plane where the target point is located, preferably, take pictures directly above or below the target point, and adjust the pose of the camera multiple times (such as rotating the camera and / or adjusting the camera height) to obtain several photos.

[0066] In one embodiment, such as Figure 2 As shown in (b), aim at the target point from above the target point and change the camera posture for many times to take a group of pictures, and use the monocular vision method to intersect with the horizontal plane in step (2) to obtain horizontal two-dimensional coordinates: For each picture, use OpenCV to extract the Aruco marker corner points, estimate the camera posture based on the reference marker and the PNP algorithm, and combine the camera internal parameters to map the target Aruco marker upper left corner point P to the three-dimensional space, forming a ray pointing to P in the current posture of the camera, and calculate the three-dimensional coordinates (x, y, z) of the intersection of the ray and the horizontal plane S in step (2) and write them into the data list data2, which specifically includes the following sub-steps:

[0067] (3.1) Sight the target point and take a group of pictures in the area directly above the target point and within the intersection angle of 30° with the plumb line. Since data processing means are needed to verify each other and eliminate bad data, it is recommended to use 4 or more photos, and each photo must contain the target point mark and at least one reference point mark; in this embodiment, 5 photos are actually taken;

[0068] (3.2) For each image, you can use the opencv aruco.detectMarkers function to read the pixel coordinates of all the Aruco markers in the image and record them in a csv file in the form of a list;

[0069] (3.3) Pass the coordinates of all the Aruco marker corner points belonging to the benchmark marker obtained in step (3.2) into the solvePnP function of opencv to obtain the position and posture information of the camera relative to the benchmark target during shooting;

[0070] (3.4) Using the camera pose data obtained in step (3.3) and the known camera intrinsic parameter matrix, the upper left corner point P of the target Aruco marker obtained in step (3.2) is mapped to the three-dimensional space to form a ray pointing from the camera in the current pose to P;

[0071] (3.5) Calculate the three-dimensional coordinates (x, y, z) of the intersection of the ray and the horizontal plane S in step (2) and write them into the data list data2.

[0072] (4) Similarly, calculate several three-dimensional coordinates according to step (3) and take the average value as the three-dimensional coordinate of the target point.

[0073] In one embodiment, the method further includes: using one or two passes of the 3σ criterion to eliminate outliers on the three-dimensional coordinates obtained in step (4), and taking the average value of the remaining data as the three-dimensional coordinates of the target point.

[0074] In one embodiment, one or two passes of the 3σ criterion are used to remove outliers from data2, the average value of the remaining data is used as the three-dimensional coordinates (X, Y, Z) of the target point, and the standard deviation [σ] of the remaining data is used as the reliability indicator of the result.

[0075] In summary, the present invention firstly proposes a three-dimensional artificial marking production scheme, so that monocular and binocular vision pictures can be taken by two shooting methods: in-plane panoramic shooting and vertical line overhead shooting, which can greatly reduce the positioning error.

[0076] Then, a self-feedback target point three-dimensional coordinate acquisition process based on monocular and binocular vision is proposed. This method uses a monocular camera throughout the process, and the camera pose information when each photo is taken is obtained through stereo manual marking. The elevation coordinates of the target point are obtained based on a set of binocular vision between two photos around the axis, and then the monocular vision is used along the axis to obtain the two-way coordinates in the horizontal plane. Data processing methods are used in the process;

[0077] Finally, a monocular and binocular vision positioning method involving data processing is proposed. After the monocular vision or binocular vision operation is completed, the abnormal data is eliminated by statistical noise reduction to reduce the error and enhance the robustness of the algorithm. At the end of the algorithm, the reliability feedback index is obtained by calculating the standard deviation of the data list.

[0078] In addition, a solution using only a monocular camera was proposed, which was tested and found to be able to be effectively combined with a drone.

[0079] Implementation Examples

[0080] The images were taken using a 5120×5120 resolution MARS-2621-42GTM monocular camera from Daheng Image, and the implementation example of the present invention was implemented on a machine equipped with a GeForce RTX 3060 graphics processor and 16GB of memory. All parameter values ​​listed in the specific implementation were used to obtain the experimental results shown in the accompanying drawings.

[0081] Figure 4 is the result list after binocular vision operation, Figure 4 (a) in the figure is point 1. Figure 4 (b) is the second point. Figure 4 (c) is point 3. After removing the outliers in the solid-line frame using the 3σ criterion, it can be clearly seen that the data error along the axis marked by the dotted-line frame is as low as 1 mm. The average value of the data in the frame is used as the coordinate along the axis for subsequent monocular vision calculations.

[0082] Figure 5 is a list of results after monocular vision operation. Figure 5 (a) in the figure is point 1. Figure 5 (b) is the second point. Figure 5(c) in the figure is point 3. After using the 3σ criterion to remove outliers and the data in the solid box, we can see that the data errors in the two directions in the plane marked by the dotted box are as low as mm level. The average value of the data in the plane is used as the coordinates in the two directions. The standard deviation of the two columns of data is the reliability feedback indicator of the final result, reflecting the self-feedback capability of the entire process algorithm.

[0083] The present invention also provides a device for acquiring three-dimensional coordinates of a target point based on monocular and binocular vision, comprising:

[0084] The marking setting module is used to set a number of Aruco markings at the reference point, referred to as the reference markings; and set an Aruco marking at the target point, referred to as the target marking;

[0085] The first calculation module is used to perform rotation shooting around the vertical direction of the horizontal plane containing the target point to obtain a number of photos, each of which presents the target mark and at least one reference mark; based on the coordinates of the corner points of the reference mark in the photo, obtain the position and posture of the camera relative to the reference mark during shooting; determine the ray pointing from the camera to the target point according to the position and posture information and the camera internal parameters; solve the elevation coordinate z value of the midpoint of the shortest line segment between the two rays in the direction of the plumb line and take the average value as the elevation coordinate Z of the target point; and calculate the plane equation of the horizontal plane S passing through the point (0,0,Z);

[0086] The second calculation module is used to shoot outside the horizontal plane containing the target point to obtain a number of photos, each photo presenting the target mark and at least one reference mark; based on the corner point coordinates of the reference mark in the photo, the position and posture of the camera relative to the reference mark during shooting are obtained; according to the position and posture information and the camera internal parameters, the ray pointing from the camera to the target point is determined; the three-dimensional coordinates of the intersection of the ray and the horizontal plane S are calculated; similarly, several three-dimensional coordinates are calculated and the average value is taken as the three-dimensional coordinates of the target point.

[0087] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.

[0088] Figure 6 This is a schematic diagram of an electronic device structure provided by an embodiment of the present invention. Figure 6 The electronic device provided in this embodiment includes: a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions. When the program instructions are loaded and executed by the processor, a method for acquiring three-dimensional coordinates of a target point based on monocular and binocular vision and self-feedback is implemented according to the present invention.

[0089] It should be noted that, in addition to Figure 6In addition to the memory and processor shown, the electronic device may also include other hardware according to its actual functions, which will not be described in detail.

[0090] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a method for acquiring three-dimensional coordinates of a target point based on monocular and binocular vision and self-feedback in the above embodiment is implemented.

[0091] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0092] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A method for acquiring three-dimensional coordinates of a target point based on self-feedback of monocular and binocular vision, characterized in that: The following steps are involved: (1) Setting at least one reference Aruco marker at the reference point and setting a target Aruco marker at the target point; (2) In the horizontal plane where the target point is located, rotate and shoot around an axis perpendicular to the plane to obtain multiple photos containing the target Aruco marker and at least one reference Aruco marker; based on the coordinates of the corner points of the reference Aruco marker in the photos, obtain the position and posture of the camera relative to the reference Aruco marker during shooting; determine the ray pointing from the camera to the target point based on the position and posture information and the camera internal parameters; Calculate the elevation coordinate z value of the midpoint of the shortest line segment between two rays, and take the average value as the elevation coordinate Z of the target point; determine the plane equation of the horizontal plane S passing through the point (0,0,Z); (3) photographing outside the horizontal plane where the target point is located to obtain multiple photos containing the target Aruco marker and at least one reference Aruco marker; Based on the coordinates of the corner points of the reference Aruco marker in the photo, obtain the pose of the camera relative to the reference Aruco marker during shooting; determine the ray pointing from the camera to the target point based on the pose information and the camera internal parameters; and calculate the three-dimensional coordinates (x, y, Z) of the intersection of the ray and the horizontal plane S; (4) Calculate multiple three-dimensional coordinate values ​​in the same way as step (3), and take the average value as the three-dimensional coordinate (X, Y, Z) of the target point.

2. The method for acquiring three-dimensional coordinates of a target point based on monocular and binocular vision self-feedback according to claim 1, characterized in that: The step (2) further includes: In the step of "calculating the elevation coordinate z value of the midpoint of the shortest line segment between two rays", the situation where the intersection angle between two rays does not satisfy the first preset threshold α is excluded.

3. The method for acquiring three-dimensional coordinates of a target point based on monocular and binocular vision self-feedback according to claim 1, characterized in that: The step (2) further includes: In the step of "calculating the elevation coordinate z value of the midpoint of the shortest line segment between any two rays", the case where the length of the shortest line segment between any two rays is greater than the second preset threshold β is excluded.

4. The method for acquiring three-dimensional coordinates of a target point based on monocular and binocular vision self-feedback according to claim 1, characterized in that: Also includes: For the elevation coordinate z value solved in step (2), the 3σ criterion is used to eliminate outliers, and the average value of the remaining data is taken as the elevation coordinate Z of the target point.

5. The method for acquiring three-dimensional coordinates of a target point based on monocular and binocular vision self-feedback according to claim 1, characterized in that: Also includes: For the obtained three-dimensional coordinates (x, y, Z), the 3σ criterion is used to eliminate outliers, and the average value of the remaining data is taken as the three-dimensional coordinates (X, Y, Z) of the target point.

6. The method for acquiring three-dimensional coordinates of a target point based on monocular and binocular vision self-feedback according to claim 5, characterized in that: The standard deviation σ of the remaining data is calculated as the reliability indicator feedback of the result.

7. The method for acquiring three-dimensional coordinates of a target point based on monocular and binocular vision self-feedback according to claim 1, characterized in that: The shooting outside the horizontal plane where the target point is located includes: Shoot directly above or below the target point.

8. A device for acquiring three-dimensional coordinates of a target point based on monocular and binocular vision, characterized in that: include: A marker setting module, used to set a number of Aruco markers at the reference point, called reference Aruco markers; Setting Aruco markers at target points is called target Aruco markers; The first calculation module is used to: in a horizontal plane where the target point is located, rotate and shoot around an axis perpendicular to the plane to obtain multiple photos containing the target Aruco marker and at least one reference Aruco marker; Based on the coordinates of the corner points of the reference Aruco marker in the photo, the position and posture of the camera relative to the reference Aruco marker during shooting are obtained; based on the position and posture information and the camera internal parameters, the ray pointing from the camera to the target point is determined; Calculate the elevation coordinate z value of the midpoint of the shortest line segment between two rays, and take the average value as the elevation coordinate Z of the target point; determine the plane equation of the horizontal plane S passing through the point (0,0,Z); The second calculation module is used to: take photos outside the horizontal plane where the target point is located to obtain multiple photos containing the target Aruco mark and at least one reference Aruco mark; based on the corner coordinates of the reference Aruco mark in the photos, obtain the position and posture of the camera relative to the reference Aruco mark during shooting; determine the ray pointing to the target point from the camera according to the position and posture information and the camera internal parameters; calculate the three-dimensional coordinates (x, y, Z) of the intersection of the ray and the horizontal plane S; similarly calculate multiple three-dimensional coordinate values, and take the average value as the three-dimensional coordinates (X, Y, Z) of the target point.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement a method for acquiring three-dimensional coordinates of a target point based on monocular and binocular vision and self-feedback as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a method for acquiring three-dimensional coordinates of a target point based on self-feedback monocular or binocular vision as described in any one of claims 1 to 7 is implemented.

Images