Partition detection positioning method and device based on marker fusion PnP, and medium

Abstract

The invention relates to a partition detection positioning method and device based on marker fusion PnP and a medium, and the method comprises the following steps: obtaining a scene marker video frame to be detected and positioned collected by a camera in real time, and extracting the ORB feature points of the video frame; matching the video frame ORB feature points with image ORB feature points extracted in advance according to a scene marker image partition, wherein the scene marker image comprises a positioned part marker; judging whether the matching is successful or not, if yes, calculating a homography matrix, and if not, obtaining a next video frame to perform feature point matching again; calculating an RT pose matrix of the video frame by using a PnP algorithm based on the homography matrix; calculating three-dimensional information of the ORB feature points of the video frame on an image plane based on the RT pose matrix; positioning information of the part marker is obtained based on the three-dimensional information, and the detection positioning process is completed. Compared with the prior art, the method has the advantages of improving the positioning effect and the like.

Description

Technical Field

[0001] The present invention relates to the field of intelligent auxiliary assembly, and in particular to a partition detection and positioning method, equipment and medium based on marker fusion PnP. Background Art

[0002] In the industrial field, the traditional assembly that completely relies on the memory and experience of workers has gradually failed to meet the actual production needs for assembly. The augmented reality (AR) intelligent assisted assembly system can help workers remember the assembly positions of parts, significantly improve the assembly efficiency of workers and thus improve energy efficiency. It has become a research hotspot in the field of parts assembly. Humans cannot remember the positions where parts need to be assembled as accurately as computers, so they can assist in assembly by using the method of computer recognition and prompts. However, computers are limited by camera equipment, and the areas that can be recognized are always limited. However, in actual assembly scenarios, there is often a problem that the camera cannot fully recognize the entire work area.

[0003] The mainstream method of computer recognition now is to process point cloud features based on deep learning. This method is suitable for robots to build or map three-dimensional scenes in unfamiliar environments. However, in the field of intelligent assembly, when using this method, computers will process a large amount of useless environmental feature information, resulting in a waste of limited computing power, making the equipment run slowly and the actual reliability low, which seriously affects the operating experience of workers in actual scenes and fails to achieve the expected results. In fact, under the premise of ensuring fast operation, the recognition success rate only needs to ensure a certain degree of reliability. Therefore, the recognition of landmarks is more suitable for this scenario than the recognition of point cloud features.

[0004] Computer recognition relies on feature point extraction and matching algorithms. The earliest algorithm SIFT (Scale-invariant feature transform) has now been gradually replaced. The current mainstream algorithm is SURF (Speeded Up Robust Features). It uses integral images to efficiently approximate Harr wavelets to extract features. The amount of information processed is very large. Although the accuracy can be guaranteed, the disadvantage is that the algorithm runs for a long time and has very high requirements for the processor. In the field of industrial assembly, cost performance and efficiency are often the most important factors. Obviously, the SURF algorithm is not suitable for this scenario. Considering these practical needs, the fast detection sub-FAST (Features from Accelerated Segment Test) that uses the gray value comparison within the neighborhood and uses the binary features of the gray value relationship of random points in the local image neighborhood, in addition to integrating the rotation invariance of the ORB (Oriented FAST and Rotated BRIEF) algorithm, has become the best choice in this scenario. Its accuracy is lower than SURF, but the operation speed is very fast, and the algorithm stability is very high, which is very in line with practical needs.

[0005] In summary, in actual assembly scenarios, when dealing with the problem that the camera cannot fully identify the entire working area, many researchers often choose to let the operator slowly turn around the assembly parts several times in order to collect more feature points for identification and positioning. However, this method is very difficult to operate in practice and cannot guarantee high accuracy. Summary of the invention

[0006] The purpose of the present invention is to provide a partition detection and positioning method, device and medium based on marker fusion PnP to improve the positioning effect.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A partition detection and positioning method based on marker fusion PnP, comprising the following steps:

[0009] Acquire the video frames of the scene landmarks to be detected and located collected by the camera in real time, and extract the ORB feature points of the video frames;

[0010] Matching the video frame ORB feature points with the image ORB feature points pre-extracted based on the scene marker image partitions, wherein the scene marker image includes the part markers that have been positioned;

[0011] Determine whether the match is successful. If so, calculate the homography matrix. If not, obtain the next scene landmark video frame and re-match the feature points.

[0012] Based on the homography matrix, the RT pose matrix of the video frame is calculated using the PnP algorithm;

[0013] Based on the RT pose matrix, calculating the three-dimensional information of the ORB feature points of the video frame on the image plane;

[0014] The positioning information of the part marker is obtained based on the three-dimensional information to complete the detection and positioning process.

[0015] Furthermore, the ORB feature extraction method is combined with the image pyramid method to extract the ORB feature points of the image, wherein the number of feature points in each layer is allocated in a geometric progression manner in the image pyramid method, and the expression is:

[0016]

[0017] Where allocation is the number of feature points in each layer of the image pyramid, nfeatures is the number of feature points, factor is the scaling factor, and n is the number of image pyramid layers.

[0018] Furthermore, it also includes: correcting the position of the ORB feature point of the image by correcting radial distortion and tangential distortion, and the correction expressions of the radial distortion and tangential distortion are respectively:

[0019]

[0020]

[0021] Where x, y are the original pixel coordinates; x', y' are the radial distortion correction parameters; k1, k2 are the pixel coordinates after radial distortion correction; x", y" are the pixel coordinates after tangential distortion correction; p1, p2 are the tangential distortion correction parameters.

[0022] Furthermore, the specific steps of calculating the homography matrix include:

[0023] The plane equation of the image plane where the image is located is constructed based on the matched image ORB feature points:

[0024] A(x-x0)+B(y-y0)+C(z-z0)=0

[0025] Where, (A, B, C) is the image plane normal vector n, (x0, y0, z0) is a known point on the image plane, and (x, y, z) is any point P on the image plane;

[0026] Simplifying, we get:

[0027] Ax+By+Cz+(-Ax0-By0-Cz0)=0

[0028] Let (-Ax0-By0-Cz0=d), we get:

[0029] n T P+d=0

[0030] Further sorting gives:

[0031]

[0032] Calculate the projection of the matched video frame feature points to the image plane. The projection expression is:

[0033] p=K(RP+t)

[0034] Where p is the matched video frame feature point, K is the camera's intrinsic parameter matrix, R is the rotation matrix, and t is the translation matrix;

[0035] We can get:

[0036]

[0037] In the formula, p1 and p2 are both matching video frame feature points;

[0038] remember Let it be H, and we get p2=Hp1, which can be expanded to:

[0039]

[0040] The homography matrix H is calculated by iteratively solving at least 4 pairs of image ORB feature points and corresponding matching video frame feature points.

[0041] Furthermore, the iterative solution step includes:

[0042] The initial homography matrix is ​​obtained by preliminarily solving the image ORB feature points and the corresponding matching video frame ORB feature points;

[0043] The image ORB feature points are reconstructed onto the matching video frame through affine reconstruction, and it is calculated whether the actual error is greater than the expected error. If so, the image feature points are removed. If not, the image ORB feature points and the corresponding matching video frame ORB feature points are screened, and the homography matrix is ​​recalculated based on the remaining image ORB feature points and the corresponding matching video frame ORB feature points after screening. The steps are repeated until the iteration ends to obtain the final homography matrix.

[0044] Furthermore, the RANSAC algorithm is used to screen the image ORB feature points and the corresponding matching video frame ORB feature points.

[0045] Furthermore, the step of calculating the RT pose matrix of the video frame includes:

[0046] Projecting the image ORB feature points into the matching video frame using the homography matrix to calculate the projection;

[0047] Based on the projection, a PnP algorithm is used to calculate the RT pose matrix of the video frame relative to the scene marker image.

[0048] Furthermore, the expression for calculating the three-dimensional information is:

[0049]

[0050] Where realX, realY and realZ are the three-dimensional information of the feature point in the video frame under the spatial coordinates, x and y are the two-dimensional coordinates of the feature point in the video frame, and X and Y are half the size of the marker.

[0051] The present invention also provides an electronic device, comprising: one or more processors; a memory; and one or more programs stored in the memory, wherein the one or more programs include instructions for executing the partition detection and positioning method based on marker fusion PnP as described above.

[0052] The present invention also provides a computer-readable storage medium, comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the partition detection and positioning method based on marker fusion PnP as described above.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] (1) In order to solve the problem that the augmented reality system cannot fully identify the working area in the intelligent assisted assembly scene because the working area is too large, the present invention can detect the feature points that are relatively matched between the video frame and the image by matching the feature points of the video frame with the scene marker image that has been pre-divided into areas, thereby obtaining the positioning information of the part marker. This partition detection method only needs to integrate and process all the information collected by the camera, and does not need to collect more feature points for identification and positioning, which can improve the positioning effect.

[0055] (2) The present invention adopts an iterative solution method to solve the optimized homography matrix. By continuously eliminating matching points with insufficient accuracy in the homography matrix, the error is reduced and the algorithm accuracy is improved. This enables the method to have a better detection and positioning effect when facing problems such as image occlusion, severe tilt, scale change, and metal material reflection. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0057] Figure 2 is a scene landmark diagram in an embodiment of the present invention;

[0058] Figure 3 A diagram of a plate-shaped part marker in an embodiment of the present invention;

[0059] Figure 4 This is a diagram showing the detection effect of tilt occlusion and scale change in an embodiment of the present invention;

[0060] Figure 5 This is a diagram showing the effect of occlusion detection affected by illumination in an embodiment of the present invention;

[0061] Figure 6 The following are the effect diagrams of the detection of the work area 1 in the embodiment of the present invention, wherein (a) is the effect diagram of the scene marker after the detection partition, and (b) is the effect diagram of the scene marker fixed in the three-dimensional scene after the detection is removed;

[0062] Figure 7 The following are the effect diagrams of the detection of the work area 2 in the embodiment of the present invention, wherein (a) is the effect diagram of the scene marker after the detection partition, and (b) is the effect diagram of the scene marker fixed in the three-dimensional scene after the detection is removed;

[0063] Figure 8 The following are the effect diagrams of the detection of the work area 3 in the embodiment of the present invention, wherein (a) is the effect diagram of the scene marker after the detection partition, and (b) is the effect diagram of the scene marker fixed in the three-dimensional scene after the detection is removed;

[0064] Fig. 9 The following are the effect diagrams of the detection of the work area 4 in the embodiment of the present invention, wherein (a) is the effect diagram of the scene marker after the detection partition, and (b) is the effect diagram of the scene marker fixed in the three-dimensional scene after the detection is removed;

[0065] Fig.10 1 is a comparison diagram of the registration effect of the part marker in the embodiment of the present invention, wherein (a) is a diagram of the registration result of the part marker in the theoretical working scene marker, and (b) is a diagram of the registration effect of the part marker of the detection and positioning method;

[0066] Fig.11 This is a diagram of the actual registration effect of the oblique placement in the embodiment of the present invention. DETAILED DESCRIPTION

[0067] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0068] Example 1

[0069] This embodiment provides a partition detection and positioning method based on marker fusion PnP, such as Figure 1 As shown, the method specifically comprises the following steps:

[0070] Step 1: Obtain the video frame of the scene landmark to be detected and located captured by the camera in real time, and extract the ORB feature points of the video frame.

[0071] Figure 2 For the scene markers used in this embodiment, a camera is used to collect the video stream of the scene marker to be detected and located in real time, and the video stream is switched to a video frame. According to the obtained video frame, an ORB feature extraction method is used to extract the ORB feature points of the video frame.

[0072] Step 2: Match the video frame ORB feature points with the image ORB feature points extracted in advance based on the scene marker image partitioning, wherein the scene marker image includes the part markers that have been positioned.

[0073] First, the scene marker image needs to be preloaded. The image contains part markers for positioning small parts. The part markers used in this embodiment are as follows: Figure 3 As shown in the figure. After the scene marker image is partitioned into four equal parts, the ORB features of the marker image are extracted and the positions of the feature points are corrected. The image pyramid method is used to solve the scale deficiency of the image ORB feature extraction method. Here, the geometric progression method is used to allocate the number of feature points in each layer.

[0074]

[0075] In formula (1), allocation is the number of feature points in each layer of the image pyramid; nfeatures is the number of feature points; factor is the scaling factor; and n is the number of layers in the image pyramid.

[0076] Because when light hits the camera, it will definitely be distorted to varying degrees. The distortion will cause a relatively large error in the experimental data, and this error from radial distortion and tangential distortion needs to be corrected.

[0077]

[0078] In formula (2), x, y are the original pixel coordinates; x', y' are the pixel coordinates after correction by radial distortion parameters; k1, k2.

[0079] Tangential distortion correction uses the following formula:

[0080]

[0081] In formula (3), x”, y” are the pixel coordinates after secondary correction; p1, p2 are the tangential distortion correction parameters.

[0082] To know the actual size of the marker, use a ruler to measure the actual size of the marker. Use the PnP algorithm to obtain the RT pose matrix between the camera and the marker. The specific steps are as follows:

[0083] Let spatial point , correspond to feature point in the image. We can get the following formula:

[0084]

[0085] Eliminating s in the third line, we can get the following two constraint expressions:

[0086]

[0087]

[0088] Where: t1 = (t1, t2, t3, t4) T , t2=(t5,t6,t7,t8) T , t3=(t9,t 10 , t 11 , t 12 ) T

[0089] Simplified:

[0090]

[0091]

[0092] Here t is an unknown number. The following linear equations with N characteristic points can be obtained:

[0093]

[0094] At least 6 sets of feature points are needed to solve t, which has 12 unknowns. When the corresponding feature points collected are greater than 6, multiple different t values ​​can be solved. In order to make the calculated value closer to reality, it is necessary to use the knowledge of SVD decomposition to solve overdetermined equations to find the least squares solution.

[0095] Saving the image feature points and the camera's zero-frame video pose is a supplementary process for program initialization after the algorithm framework is built. After saving, the first detection will be very fast, which will save a lot of time in the test code stage.

[0096] Next, the image ORB feature points and the video frame ORB feature points are matched. Specifically, the preset image ORB feature point data is read, which is equivalent to a read and write instruction replacing the computer's recognition and judgment process, and the marker can be directly restored. The program starts running, extracting the ORB features from the current frame image obtained by the camera and matching them with the image ORB feature points in the preset marker image, which is equivalent to giving the computer the ability to combine memory and recall. More specifically, the smaller the error of feature matching, the more similar the two feature points are.

[0097] Step 3: Determine whether the match is successful. If so, calculate the homography matrix. If not, obtain the next scene marker video frame and re-match the feature points.

[0098] In this implementation, whether the match is successful is determined by the number of matching points reaching a preset threshold (for example, there are at least 10 pairs of matching points), and the homography matrix is ​​calculated after the match is successful.

[0099] The homography matrix describes the position mapping relationship between the object in the world coordinate system and the pixel coordinate system.

[0100] Assume that the feature points p1 and p2 of the video frame are matching points on two image planes. Let M0(x0, y0, z0) be a known point on the plane, P(x, y, z) be any point on the plane, and vector n(A, B, C) be the plane normal vector. The equation of the plane where the feature points are located is:

[0101] A(x-x0)+B(y-y0)+C(z-z0)=0 (10)

[0102] Simplified:

[0103] Ax+By+Cz+(-Ax0-By0-Cz0)=0 (11)

[0104] Let (-Ax0-By0-Cz0=d), we get:

[0105] n T P+d=0 (12)

[0106] After a little tidying up:

[0107]

[0108] Projection formula from a point to a plane image:

[0109] p=K(RP+t) (14)

[0110] We can get:

[0111]

[0112] Will Let it be H, and we get p2=Hp1. Expand it and we get:

[0113]

[0114] Let, at this time there are 8 unknown parameters in the H matrix, and at least 4 pairs of matching feature points need to be solved.

[0115] Since the pairing of feature points may not meet the experimental requirements, the RANSAC algorithm is used to iteratively solve the H matrix to obtain an optimal solution.

[0116] Since the homography matrix directly determines the accuracy of the algorithm, the optimization of the posture ultimately returns to the homography matrix in mathematical terms, so the homography matrix needs to be further optimized, and the matching points with insufficient accuracy in the homography matrix obtained in the previous step need to be removed. The whole process is to first reconstruct the feature points extracted from the landmark onto the current image frame through affine; then use professional measurement tools to measure the actual error. If the actual error is greater than expected, remove the feature point; finally, use the RANSAC algorithm to screen all the matching points that meet the expectations. After the screening is completed, use the remaining points to calculate the homography matrix, and the error becomes very small at this time.

[0117] Step 4: Based on the homography matrix, use the PnP algorithm to calculate the RT pose matrix of the video frame.

[0118] The homography matrix is ​​used to project the known points in the landmarks to the first successfully matched video frame, and the PnP is used to calculate the RT pose matrix of this frame relative to the scene landmarks.

[0119] Step 5: Based on the RT pose matrix, calculate the three-dimensional information of the ORB feature points of the video frame on the image plane.

[0120] Calculate the 3D points on the plane and save the information in the 3D scene. The calculation formula is as follows:

[0121]

[0122] In formula (17), x and y are the coordinates of the feature point; realX, realY and realZ are the spatial coordinates of the feature point; col and row are the original marker sizes; and X and Y are half-sized markers.

[0123] Step 6: Obtain positioning information of the part marker based on the three-dimensional information to complete the detection and positioning process.

[0124] By matching the video frame with the scene marker image containing the part marker and performing pose conversion, the three-dimensional information of the ORB feature points of the video frame is obtained, which also includes the positioning information of the part marker. The part marker is placed in the scene marker for recognition, so that the registration effect is complete.

[0125] This embodiment also uses experiments to verify the effectiveness of the above method. The hardware configuration environment used in the experiment is Intel (R) Pentium (R) CPU G4600, graphics card NVIDIA GeForce GTX 1050, operating system Windows 10, software environment Microsoft Visual Studio 2017, Opencv3.4.15, an online recognition display module USB external 480p low-precision camera, and the Nadejia NEW G2 PRO glasses used to measure accuracy are combined with the Snapdragon 870 processor. When the present invention compares the algorithms, a scene video with a marker recorded by a USB camera is selected, and the detection is performed by intercepting the number of frames in different postures. If successful, the four edges of the detected marker will be drawn to intuitively show the effectiveness of the algorithm. When the error of the four drawn edges exceeds 50mm, the actual detection of this data is lost, so it is included in the recognition loss rate, not in the average error.

[0126] In view of the situation that workers may fail to track and identify the markers due to tilt, occlusion, scale change, and reflection of metal materials during movement, the three most classic algorithms (SIFT, SURF, ORB) and Gracker algorithm were selected to compare the time consumption, recognition loss rate, and average error with the Ours algorithm in this study. The results are shown in Table 1. In this scene detection, ORB and Ours have faster recognition speed, and SURF and Ours have higher recognition accuracy. Therefore, the Ours algorithm is more suitable for use in this detection scenario.

[0127] Table 1 Comparison of time consumption, recognition loss rate and average error of each algorithm

[0128]

[0129] Generally speaking, the accuracy of the two algorithms is not much different, so this experiment mainly compares the problems of illumination, scale change, occlusion and motion blur. Figure 4As shown in the figure, because the recognition effect can be intuitively expressed by the positions of the four edges, it can be seen that when the marker occupies only about 1 / 6 of the screen, ORB tracking fails, while Ours can track it with a small loss of accuracy. Considering the limited computing power of the processor, the low accuracy of the USB camera, and the blurry markers when the scale changes greatly, it can be seen that Ours is more efficient in processing markers under low computing power conditions. Ours real-time tracking of five occluded markers with different lighting effects is shown in the figure. Figure 5 It can be seen that Ours algorithm has high accuracy for real-time tracking of occluded environments with different lighting.

[0130] like Figure 6-9 The detection effect diagrams of the four work areas shown in the figure, where (a) is the effect of directly detecting the scene markers after partitioning, and (b) is the effect of removing the scene markers after detection and fixing them in the three-dimensional scene. It can be seen that after the work area detection is successful, the required registration effect can be successfully positioned in the three-dimensional virtual scene, which can effectively assist workers in positioning the work area.

[0131] like Fig.10 As shown, here is a simulation of a worker installing a block-shaped part at a location in the work area. (a) is a diagram of the placement of the marker in the work scene in theory in the embodiment, and (b) is a diagram of the registration effect in theory in the embodiment. Fig.11 It can be seen that the present invention can realize the functions of partitioning the working area and locating the parts in the partition by detecting the scene markers and part markers in the working area.

[0132] In the above specific embodiment, under the condition of USB camera display, no more equipment is added for experiment, which shows that the cost is quite low. In actual application, the actual conditions will not be simpler than the experimental conditions. Under the display conditions of Nadejia NEW G2 PRO glasses combined with Snapdragon 870 processor, the accuracy can be controlled below 3mm, which is a relatively good performance. Therefore, the above algorithm will not add too much extra overhead in actual application and is easy to implement.

[0133] In summary, the present invention can solve the problem that the augmented reality system cannot fully identify the working area in the intelligent assisted assembly scene because the working area is too large, and can assist workers to divide the entire working area into sections before performing assembly operations. The algorithm of the present invention runs fast, and the tracking accuracy and efficiency are guaranteed. It can effectively solve the problems of screen jitter, screen occlusion, severe tilt, scale change, metal material reflection, etc. in actual assembly scenes. And in practice, it has the advantages of saving computing power, high cost performance, and easy implementation.

[0134] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0135] Example 2

[0136] This embodiment provides an electronic device, including: one or more processors; a memory; and one or more programs stored in the memory, wherein the one or more programs include instructions for executing the partition detection and positioning method based on marker fusion PnP as described in Example 1.

[0137] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0138] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0141] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0142] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A partition detection and positioning method based on marker fusion PnP, It is characterized in that The following steps are involved: Acquire the video frames of the scene landmarks to be detected and located collected by the camera in real time, and extract the ORB feature points of the video frames; Matching the video frame ORB feature points with the image ORB feature points pre-extracted based on the scene marker image partitions, wherein the scene marker image includes the part markers that have been positioned; Determine whether the match is successful. If so, calculate the homography matrix. If not, obtain the next scene landmark video frame and re-match the feature points. Based on the homography matrix, the RT pose matrix of the video frame is calculated using the PnP algorithm; Based on the RT pose matrix, calculating the three-dimensional information of the ORB feature points of the video frame on the image plane; The positioning information of the part marker is obtained based on the three-dimensional information to complete the detection and positioning process.

2. A method for partition detection and positioning based on marker fusion PnP according to claim 1, It is characterized in that The ORB feature extraction method is combined with the image pyramid method to extract the ORB feature points of the image, wherein the number of feature points in each layer is allocated in a geometric progression manner in the image pyramid method, and the expression is: Where allocation is the number of feature points in each layer of the image pyramid, nfeatures is the number of feature points, factor is the scaling factor, and n is the number of image pyramid layers.

3. A method for partition detection and positioning based on marker fusion PnP according to claim 2, It is characterized in that Also includes: The positions of the ORB feature points of the image are corrected by correcting the radial distortion and the tangential distortion. The correction expressions of the radial distortion and the tangential distortion are respectively: Where x, y are the original pixel coordinates; x', y' are the radial distortion correction parameters; k 1 , k 2 is the pixel coordinate after radial distortion correction; x”, y” is the pixel coordinate after tangential distortion correction; p 1 , p 2 is the tangential distortion correction parameter.

4. A method for partition detection and positioning based on marker fusion PnP according to claim 1, It is characterized in that The specific steps of calculating the homography matrix include: The plane equation of the image plane where the image is located is constructed based on the matched image ORB feature points: A(x-x 0 )+B(y-y 0 )+C(z-z 0 )=0 Where (A, B, C) is the image plane normal vector n, (x 0 ,y 0 , z 0 ) is a known point on the image plane, (x, y, z) is any point P on the image plane; Simplifying, we get: Ax+By+Cz+(-Ax 0 -By 0 -Cz 0 )=0 Let (-Ax 0 -By 0 -Cz 0 =d), we get: n T P+d=0 Further sorting gives: Calculate the projection of the matched video frame feature points to the image plane. The projection expression is: p=K(RP+t) Where p is the matched video frame feature point, K is the camera's intrinsic parameter matrix, R is the rotation matrix, and t is the translation matrix; We can get: In the formula, p 1 、p 2 All are matching video frame feature points; Will Let it be H, and we get p2=Hp1, which can be expanded to: The homography matrix H is calculated by iteratively solving at least 4 pairs of image ORB feature points and corresponding matching video frame feature points.

5. A method for partition detection and positioning based on marker fusion PnP according to claim 4, It is characterized in that The iterative solution step comprises: The initial homography matrix is ​​obtained by preliminarily solving the image ORB feature points and the corresponding matching video frame ORB feature points; The image ORB feature points are reconstructed onto the matching video frame through affine reconstruction, and it is calculated whether the actual error is greater than the expected error. If so, the image ORB feature points are removed. If not, the image ORB feature points and the corresponding matching video frame ORB feature points are screened, and the homography matrix is ​​recalculated based on the remaining image ORB feature points and the corresponding matching video frame ORB feature points after screening. The steps are repeated until the iteration ends to obtain the final homography matrix.

6. A method for partition detection and positioning based on marker fusion PnP according to claim 5, It is characterized in that The RANSAC algorithm is used to screen the image ORB feature points and the corresponding matching video frame ORB feature points.

7. A method for partition detection and positioning based on marker fusion PnP according to claim 1, It is characterized in that The step of calculating the RT pose matrix of the video frame comprises: Projecting the image ORB feature points into the matching video frame using the homography matrix to calculate the projection; Based on the projection, a PnP algorithm is used to calculate the RT pose matrix of the video frame relative to the scene marker image.

8. A method for partition detection and positioning based on marker fusion PnP according to claim 1, It is characterized in that The expression for calculating the three-dimensional information is: Where realX, realY and realZ are the three-dimensional information of the feature point in the video frame under the spatial coordinates, x and y are the two-dimensional coordinates of the feature point in the video frame, and X and Y are half the size of the marker.

9. An electronic device, It is characterized in that include: one or more processors; Memory; One or more programs stored in a memory, wherein the one or more programs include instructions for executing the marker fusion PnP-based partition detection and positioning method as described in any one of claims 1-8.

10. A computer-readable storage medium, It is characterized in that It includes one or more programs for execution by one or more processors of an electronic device, and the one or more programs include instructions for executing the partition detection and positioning method based on marker fusion PnP as described in any one of claims 1-8.

Images