Space positioning method and device based on wing contour feature constraint

Abstract

The invention provides a space positioning method and device based on wing contour feature constraint, relates to the technical field of complex three-dimensional reconstruction, and is used for solving the technical problem that the accuracy and reliability of existing wing part assembly are relatively low. The method comprises the following steps: initializing a binocular camera; a binocular camera is adopted to carry out image acquisition on the wing part fixed by the mechanical arm, and a target image is obtained; matching the wing contour in the target image with a pre-stored mathematical model to obtain a matching result; according to the matching result, determining whether the plurality of matching pairs are successfully matched; and if it is determined that the multiple matching pairs are matched successfully, the mechanical arm is controlled to assemble the wing parts. On the basis, a binocular camera is combined with an image processing and matching technology, and a solution with relatively high accuracy and reliability is provided for automatic identification of the aircraft wing contour.

Description

Technical Field

[0001] This application relates to the field of complex 3D reconstruction technology, and provides a spatial positioning method and device based on wing contour feature constraints. Background Technology

[0002] Currently, in aircraft assembly, the use of automatic control technology and advanced image processing technology for the automatic assembly of wing components has become a trend, and the complexity of the wing profile requires the system to have high-precision recognition and matching capabilities.

[0003] Based on this, existing technologies mainly use hoisting or mechanical tooling to assist manual operations to assemble wing components, and use monocular cameras to locate structural features on the wing. However, due to the complex structure and narrow space of aircraft, it is difficult to achieve automatic assembly operations. Furthermore, since traditional monocular cameras lack depth information, it is difficult to achieve accurate three-dimensional positioning.

[0004] Therefore, improving the accuracy and reliability of wing component assembly has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a spatial positioning method and apparatus based on wing profile feature constraints to solve the technical problem of low accuracy and reliability in existing wing component assembly.

[0006] On the one hand, a spatial positioning method based on wing profile feature constraints is provided, the method comprising: Initialize the stereo camera; A binocular camera is used to acquire images of the fixed wing components of the robotic arm to obtain target images; The wing contour in the target image is matched with a pre-stored mathematical model to obtain a matching result; wherein the matching result contains multiple matching pairs; each matching pair contains one image feature point of the target image and one mathematical model feature point of the pre-stored mathematical model; Based on the matching results, determine whether the multiple matching pairs are successfully matched; If the multiple matching pairs are determined to be successfully matched, the robotic arm is controlled to assemble the wing components.

[0007] Optionally, the step of initializing the stereo camera includes: Calibrate the activated binocular camera; The focal length and viewing angle of the binocular camera are adjusted according to the working area of ​​the robotic arm; Determine whether the position of the wing component is within the predetermined assembly area; If it is determined that the wing component is not located within the predetermined assembly area, the position of the automated guided vehicle (AGV) is adjusted.

[0008] Optionally, the step of using a binocular camera to acquire images of the wing component fixed to the robotic arm and obtain target images includes: The binocular camera is used to acquire images of the wing component from both left and right perspectives, obtaining left-view and right-view images; The left-view image and the right-view image are fused to obtain an initial fused image; A preset image processing algorithm is used to determine whether all key features of the wing contour exist in the initial fused image; If it is determined that all the key features of the wing profile exist in the initial fused image, then the initial fused image is preprocessed to obtain the target image.

[0009] Optionally, the step of using the Canny edge detection algorithm to determine whether all key features of the wing contour exist in the initial fused image includes: The Canny edge detection algorithm is used to determine whether the initial fused image contains all predefined key feature regions; If it is determined that the initial fused image contains all predefined key feature regions, then the image feature points in the initial fused image are compared with the mathematical model feature points in the pre-stored mathematical model to determine whether all the key features of the wing contour exist in the initial fused image.

[0010] Optionally, the step of matching the wing contour in the target image with a pre-stored mathematical model to obtain a matching result includes: The target image is corrected using a preset stereo correction matrix to obtain a corrected image; Feature extraction is performed on the wing contour in the corrected image to obtain the image feature vector; Based on the similarity between the image feature vector and the mathematical model feature vector of the pre-stored mathematical model, multiple initial matching pairs are filtered to obtain multiple first-filtered matching pairs; Calculate the first alignment error of each first screening matching pair according to the preset alignment error calculation formula; Determine whether each of the first alignment errors meets the preset alignment threshold; If it is determined that each of the first alignment errors meets the preset alignment threshold, then the matching is successful and the matching result is obtained.

[0011] Optionally, after determining whether each of the first alignment errors meets a preset alignment threshold, the method further includes: If it is determined that the various first alignment errors do not meet the preset alignment threshold, then the RANSAC algorithm is used to filter the multiple first screening matching pairs again to obtain multiple second screening matching pairs. According to the preset alignment error calculation formula, the alignment parameters between the corrected image and the pre-stored mathematical model are calculated; wherein, the alignment parameters include a rotation matrix and a translation vector; Based on the alignment parameters, the image feature points in the plurality of second filtering matching pairs are transformed into the digital model feature point coordinate system to obtain a plurality of image transformed feature points; Calculate the second alignment error between multiple image transformation feature points and multiple digital model feature points; Determine whether the second alignment error meets the preset alignment threshold.

[0012] Optionally, the step of determining whether the plurality of matching pairs are successfully matched based on the matching results includes: The matching error of the matching result is obtained by calculating the matching error of the multiple matching pairs; Determine whether the matching error is within a preset acceptable range; If the matching error is determined to be within a preset acceptable range, then the matching is successful; If the matching error is determined to be outside the preset acceptable range, the matching fails.

[0013] Optionally, after a match fails, the method further includes: The focal length and angle of the binocular camera are adjusted; The adjusted binocular camera was used to re-acquire images of the wing components, resulting in re-acquired images. The wing contour in the resampled image is matched with a pre-stored mathematical model to obtain the matching result.

[0014] Optionally, the step of controlling the robotic arm to assemble the wing components if it is determined that the plurality of matching pairs are successfully matched includes: If the multiple matching pairs are determined to be successfully matched, then the spatial position and attitude information of the wing component and the motion path of the robotic arm are determined based on the matching results. The binocular camera is calibrated according to a preset calibration formula; Based on the spatial position and attitude information of the wing components and the movement path of the robotic arm, the robotic arm is controlled to assemble the wing components.

[0015] On the one hand, a spatial positioning device based on wing profile feature constraints is provided, the device comprising: a binocular camera, a robotic arm, an automated guided vehicle (AGV), and a host computer; The binocular camera is used to acquire images of the wing components fixed to the robotic arm; The robotic arm is used to assemble fixed wing components; The automated guided vehicle (AGV) is used to adjust the position of the wing component to a predetermined assembly area; The host computer is used to control the binocular camera to initialize; control the binocular camera to acquire images of the wing component fixed by the robotic arm to obtain a target image; match the wing contour in the target image with a pre-stored mathematical model to obtain a matching result; wherein, the matching result contains multiple matching pairs; each matching pair contains one image feature point of the target image and one mathematical model feature point of the pre-stored mathematical model; based on the matching result, determine whether the multiple matching pairs are successfully matched; if it is determined that the multiple matching pairs are successfully matched, control the robotic arm to assemble the wing component.

[0016] On the one hand, a computer storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement any of the methods described above.

[0017] Compared with the prior art, the beneficial effects of this application are as follows: In this application, when assembling wing components, firstly, the binocular camera can be initialized; then, the binocular camera can be used to acquire images of the wing components fixed by the robotic arm to obtain target images; next, the wing contour in the target image can be matched with a pre-stored mathematical model to obtain matching results; wherein, the matching results contain multiple matching pairs; each matching pair contains one image feature point of the target image and one mathematical model feature point of the pre-stored mathematical model; then, based on the matching results, it can be determined whether the multiple matching pairs are successfully matched; finally, if it is determined that the multiple matching pairs are successfully matched, the robotic arm is controlled to assemble the wing components.

[0018] Based on this, in this application, since a binocular camera is used to acquire images of the wing component fixed by the robotic arm, compared with the traditional monocular camera for image acquisition, this application can simultaneously capture images from two perspectives using a binocular camera, thereby calculating the depth information of the wing component to achieve three-dimensional spatial positioning of the wing profile. Obviously, this application not only improves the accuracy of image recognition, the accuracy and reliability of wing component assembly, but also can adapt to different wing shapes and structures. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 A schematic diagram of a space positioning device based on wing profile feature constraints provided in an embodiment of this application; Figure 2 A schematic diagram of a host computer provided in an embodiment of this application; Figure 3 This is a schematic flowchart of a spatial positioning method based on wing profile feature constraints provided in an embodiment of this application.

[0021] The diagram is labeled as follows: 20-Host computer, 201-Processor, 202-Memory, 203-I / O interface, 204-Database. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0023] Currently, in aircraft assembly, the use of automatic control technology and advanced image processing technology for the automatic assembly of wing components has become a trend, and the complexity of the wing profile requires the system to have high-precision recognition and matching capabilities.

[0024] Based on this, existing technologies mainly use hoisting or mechanical tooling to assist manual operations to assemble wing components, and use monocular cameras to locate structural features on the wing. However, due to the complex structure and narrow space of aircraft, it is difficult to achieve automatic assembly operations. Furthermore, since traditional monocular cameras lack depth information, it is difficult to achieve accurate three-dimensional positioning.

[0025] Based on this, this application provides a spatial positioning method based on wing contour feature constraints. In this method, firstly, a binocular camera can be initialized; then, the binocular camera can be used to acquire images of the wing component fixed by the robotic arm to obtain a target image; next, the wing contour in the target image can be matched with a pre-stored mathematical model to obtain a matching result; wherein, the matching result contains multiple matching pairs; each matching pair contains one image feature point of the target image and one mathematical model feature point of the pre-stored mathematical model; then, based on the matching result, it can be determined whether the multiple matching pairs are successfully matched; finally, if it is determined that the multiple matching pairs are successfully matched, the robotic arm is controlled to assemble the wing component. Furthermore, in this application, since a binocular camera is used to acquire images of the wing components fixed to the robotic arm, compared to using a traditional monocular camera for image acquisition, this application can simultaneously capture images from two perspectives using a binocular camera, thereby calculating the depth information of the wing components to achieve three-dimensional spatial positioning of the wing contour. Obviously, this application not only improves the accuracy of image recognition, the accuracy and reliability of wing component assembly, but also can adapt to different wing shapes and structures.

[0026] After introducing the design concept of the embodiments of this application, the following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application can be applied. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0027] like Figure 1 The diagram shown is a schematic of a spatial positioning device based on wing profile feature constraints provided in an embodiment of this application. The device includes a binocular camera, a robotic arm, an automated guided vehicle (AGV), and a host computer. The AGV serves as the base of the device. The binocular camera and the robotic arm are both fixedly connected to the AGV. The host computer can be installed on the AGV or remotely installed in other suitable locations.

[0028] like Figure 1As shown, a binocular camera is used to acquire images of the wing component fixed by the robotic arm; the robotic arm is used to assemble the fixed wing component; an automated guided vehicle (AGV) is used to adjust the position of the wing component to the predetermined assembly area; a host computer is used to control the binocular camera for initialization; the binocular camera is controlled to acquire images of the wing component fixed by the robotic arm to obtain target images; the wing contour in the target image is matched with a pre-stored mathematical model to obtain matching results; the matching results contain multiple matching pairs; each matching pair contains one image feature point of the target image and one mathematical model feature point of the pre-stored mathematical model; based on the matching results, it is determined whether the multiple matching pairs are successfully matched; if the multiple matching pairs are determined to be successfully matched, the robotic arm is controlled to assemble the wing component.

[0029] like Figure 2 The diagram shown is a schematic of a host computer provided in an embodiment of this application.

[0030] The host computer 20 can be used to fuse images of surface defects on an aircraft. For example, it can be an in-vehicle computer, a personal computer (PC), a server, or a laptop. The host computer 20 may include one or more processors 201, memory 202, I / O interfaces 203, and a database 204. Specifically, the processor 201 can be a central processing unit (CPU) or a digital processing unit, etc. The memory 202 can be volatile memory, such as random-access memory (RAM); it can also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. The memory 202 can be a combination of the above-mentioned memories. The memory 202 can store some program instructions of the spatial positioning method based on wing contour feature constraints provided in the embodiments of this application. When these program instructions are executed by the processor 201, they can be used to implement the steps of the spatial positioning method based on wing contour feature constraints provided in the embodiments of this application, so as to solve the technical problem that the accuracy and reliability of existing wing component assembly are both low. The database 204 can be used to store data such as left-view image, right-view image, initial fused image, target image, corrected image, matching result, pre-stored mathematical model, image feature vector and mathematical model feature vector involved in the solution provided in the embodiments of this application.

[0031] In this embodiment, the host computer 20 can obtain the wing component assembly instructions input by the operator through the I / O interface 203. Then, the processor 201 of the host computer 20 will solve the technical problem of low accuracy and reliability of existing wing component assembly by following the program instructions of the spatial positioning method based on wing contour feature constraints provided in this embodiment, which is stored in the memory 202. In addition, data such as left-view image, right-view image, initial fused image, target image, corrected image, matching result, pre-stored mathematical model, image feature vector, and mathematical model feature vector can be stored in the database 204.

[0032] Of course, for Figure 1 The functions of each unit of the illustrated device will be described in subsequent method embodiments, and will not be elaborated upon here. The method of this application embodiment will now be described in conjunction with the accompanying drawings.

[0033] like Figure 3 The diagram shown is a flowchart illustrating a spatial positioning method based on wing profile feature constraints provided in this application. This method can... Figure 2 The method is executed by the host computer 20. The specific process is described below.

[0034] Step 301: Initialize the stereo camera.

[0035] Specifically, first, turn on the power of the binocular camera and ensure that the binocular camera software is running normally. Use a calibration board or a calibration tool of known size and place it within the field of view of the binocular camera. Start the binocular camera calibration software and capture images of the calibration board at different positions in sequence to calibrate the started binocular camera. This ensures that the binocular camera can correctly capture objects in space. In addition, the robotic arm can lift the wing component to the vicinity of the installation position (working surface).

[0036] Then, the focal length and viewing angle of the binocular camera can be adjusted manually or automatically via a host computer, depending on the size of the working area of ​​the robotic arm, to ensure that the binocular camera can clearly capture the image of the entire working area. At the same time, an appropriate image resolution is also required to ensure that the entire wing and the wing components to be assembled can be covered simultaneously.

[0037] Next, a binocular camera can be used to capture images of the current wing component to determine whether the wing component is located within the predetermined assembly area.

[0038] Finally, if it is determined that the wing component is not within the predetermined assembly area, the position of the automated guided vehicle (AGV) is adjusted; otherwise, if it is determined that the wing component is within the predetermined assembly area, the position of the AGV does not need to be adjusted.

[0039] Step 302: Use a binocular camera to acquire images of the fixed wing components of the robotic arm to obtain target images.

[0040] Specifically, firstly, by moving the AGV or adjusting the angle of the binocular camera, comprehensive image data of the wing components from multiple angles can be collected. When collecting images, the binocular camera can be used to collect images of the wing components from both the left and right perspectives to obtain left-view and right-view images.

[0041] Then, the left-view image and the right-view image can be fused together to obtain an initial fused image.

[0042] Next, a preset image processing algorithm can be used to determine whether all key features of the wing contour exist in the initial fused image. For example, the Canny edge detection algorithm can be used to identify the edge features of the wing contour in the initial fused image. Then, based on whether the region containing the wing contour edge features is a predefined key feature region, it can be determined whether the initial fused image contains all predefined key feature regions. In this embodiment, the Canny edge detection formula is as follows:

[0043] in, Gx and Gy These are the gradients of the initial fused image in the x-direction and y-direction in the image coordinate system, respectively.

[0044] Then, if it is determined that the initial fused image contains all predefined key feature regions, the image feature points in the initial fused image can be compared with the mathematical model feature points in the pre-stored mathematical model to determine whether all key features of the wing contour exist in the initial fused image. Then, if it is determined that the initial fused image does not contain all predefined key feature regions, images of the wing components are re-acquired using a binocular camera.

[0045] Finally, if all key features of the wing contour are determined to exist in the initial fused image, the initial fused image is preprocessed to obtain the target image. That is, if all key features of the wing contour are determined to exist in the initial fused image, the initial fused image is transmitted to the processor of the host computer. In the processor, the initial fused image can be preprocessed, for example, by denoising and contrast enhancement, thus preparing for subsequent image processing and feature extraction.

[0046] Among these methods, image smoothing algorithms such as Gaussian filtering can be used to reduce noise interference. In this embodiment, the Gaussian filtering formula is as follows:

[0047] Here, σ represents the standard deviation, which determines the strength of the filter. Based on this, noise information can be eliminated by performing a convolution operation between the Gaussian filter formula and the initial fused image.

[0048] In addition, histogram equalization can be performed on the initial fused image after denoising to improve image contrast and make image feature points more prominent. In this embodiment, the histogram equalization formula is as follows:

[0049] Where H(i) is the new pixel value of the i-th pixel, CDF is the cumulative distribution function, N is the total number of pixels, and L is the maximum pixel value.

[0050] Step 303: Match the wing contour in the target image with the pre-stored mathematical model to obtain the matching result.

[0051] In the embodiments of this application, the matching result contains multiple matching pairs; each matching pair contains one image feature point of the target image and one mathematical model feature point of the pre-stored mathematical model.

[0052] Specifically, firstly, a preset stereo correction matrix is ​​used to correct the target image to obtain a corrected image, thereby minimizing the reprojection distortion of the target image; in this embodiment, the preset stereo correction matrix is ​​as follows:

[0053] in, The stereo correction matrix of the target image. This is the normal vector of the pole direction after stereo correction. for The direction normal vector relative to the camera's optical axis plane, As an auxiliary parameter, This is the projection matrix.

[0054] Then, feature extraction can be performed on the wing contour in the corrected image to obtain the image feature vector; among which, the key image feature points in the image feature vector include edge lines, corner points, surface features, etc.

[0055] In this embodiment of the application, the Canny edge detection algorithm can be used to identify edge lines in the corrected image and highlight the outline of the wing.

[0056] The Harris corner detection formula can be used to detect corners in the wing profile, which are typically located at points of significant profile change. In this embodiment, the Harris corner detection formula is as follows:

[0057] Where M is the second-order matrix of the corrected image, and k is an empirical constant.

[0058] By analyzing the grayscale changes in the corrected image, surface features can be extracted, thereby determining the geometric properties of the wing surface.

[0059] Next, image matching algorithms can be used for feature point matching. During the matching process, Euclidean distance or cosine similarity can be used to calculate the similarity between the image feature vector of the corrected image and the mathematical model feature vector of the pre-stored mathematical model. In this embodiment, the Euclidean distance formula is as follows:

[0060] in, and These are the components of the two feature descriptors.

[0061] Then, the similarity between the optimal and second-best matches can be determined based on the similarity between the image feature vector and the pre-stored mathematical model's feature vector. Based on this, multiple initial matching pairs can be filtered by comparing the similarity ratio of the optimal and second-best matches to obtain multiple first-filtered matching pairs, thus eliminating unreliable matches. For example, among multiple initial matching pairs, the similarity of initial matching pair A can be divided by the similarity of the optimal match, and the similarity of initial matching pair A can be divided by the similarity of the second-best match. If the difference between the results of these two divisions is higher than 5%, then initial matching pair A can be considered an unreliable match and thus filtered out.

[0062] Next, the alignment can be determined by calculating the distance and angle between the image feature points and the digital model feature points. Specifically, the first alignment error of each first screening matching pair can be calculated according to a preset alignment error calculation formula; in this embodiment, the preset error calculation formula is as follows:

[0063] in,( , ) represents the coordinates of image feature points, ( , ) represents the coordinates of the feature points in the digital model, and N represents the number of matching pairs.

[0064] Then, it can be determined whether the first alignment error of each first filtered matching pair meets the preset alignment threshold.

[0065] Based on this, if it is determined that the first alignment error of each first screening matching pair meets the preset alignment threshold (the alignment error is less than the preset alignment threshold), then the matching is determined to be successful and the matching result is obtained.

[0066] Conversely, if it is determined that the first alignment error of each first screening match does not meet the preset alignment threshold, the RANSAC algorithm is used to screen multiple first screening match pairs again, remove outliers with larger errors, and obtain multiple second screening match pairs. Then, the alignment parameters between the corrected image and the pre-stored mathematical model can be calculated according to the preset alignment error calculation formula; wherein, the alignment parameters include a rotation matrix and a translation vector; in this embodiment, the alignment matrix calculation formula is as follows:

[0067] in, and R represents the coordinates of feature points in the corrected image and the pre-stored mathematical model, respectively; R is the rotation matrix; and T is the translation vector.

[0068] Next, based on the alignment parameters, the image feature points in multiple second-selection matching pairs can be transformed into the digital model feature point coordinate system to obtain multiple image transformed feature points.

[0069] Then, based on the aforementioned preset alignment error calculation formula, the second alignment error between multiple image transformation feature points and multiple digital model feature points is calculated again.

[0070] Next, the alignment accuracy can be verified by determining whether the second alignment error between multiple image transformation feature points and multiple digital model feature points meets the preset alignment threshold. If the alignment accuracy meets the requirements, the transformation matrix R and translation vector T can be recorded to complete the feature point matching and alignment process.

[0071] Step 304: Based on the matching results, determine whether multiple matching pairs have been successfully matched.

[0072] Specifically, firstly, matching errors can be calculated for multiple matching pairs to obtain the matching error of the matching result; in this embodiment, the average error Emean or standard deviation σ can be used to calculate the matching error, wherein the formula for the average error Emean is as follows:

[0073] The formula for standard deviation is as follows:

[0074] Where N is the number of matching pairs.

[0075] Then, error and accuracy detection can be performed on the target image based on the Sobel operator template expression, where the Sobel operator is as follows: ,

[0076] in, and This serves as the template for the horizontal and vertical directions of the Sobel operator.

[0077] Next, we can determine whether the matching error is within the preset acceptable range.

[0078] Then, if the matching error is determined to be within the preset acceptable range, the matching is successful; Conversely, if the matching error is determined to be outside the preset acceptable range, the matching will fail.

[0079] Furthermore, staff can be alerted to matching failures and the reasons can be analyzed, such as inaccurate feature point recognition, low image quality, or feature point occlusion.

[0080] Therefore, after a matching failure, image acquisition and binocular camera adjustments can be performed again to improve assembly accuracy. Specifically, the focal length and angle of the binocular camera, or the position of the wing components, can be adjusted based on the analysis results to optimize matching precision.

[0081] Furthermore, after the adjustment, the binocular camera can be used again to reacquire images of the wing components and obtain reacquired images; Then, the wing profile in the resampled image can be matched with a pre-stored mathematical model to obtain the matching result. Of course, to improve the accuracy of assembly, the matching result needs to be evaluated again.

[0082] Step 305: If multiple matching pairs are found to be successfully matched, control the robotic arm to assemble the wing components.

[0083] Specifically, if multiple matching pairs are found to be successfully matched, then based on the matching results, motion commands for the robotic arm are generated, and information such as the spatial position and attitude information of the wing component and the motion path of the robotic arm (including rotation matrix and translation vector) are determined to ensure that the wing component can accurately reach the designated position.

[0084] Then, the binocular camera can be calibrated according to the preset calibration formula (based on the calibration formula of calibration board corner points and image points), and the calibration board image can be acquired; in this embodiment of the application, the preset calibration formula is as follows:

[0085] in, The scaling factor for the i-th image point. Let i be the coordinates of the corner point of the i-th image point. The intrinsic parameter matrix, Let i be the coordinates of the i-th image point. and Let be the image rotation matrix and translation vector.

[0086] Next, after camera calibration, the robotic arm can be controlled to assemble the wing components based on their spatial position, attitude information, and the robotic arm's movement path. Specifically, the robotic arm can move to a predetermined position according to its movement commands and path, ensuring precise alignment between the end effector and the contact surface of the wing component. During the movement, the robotic arm can adjust its attitude in real time based on the wing component's spatial position and attitude information to adapt to the wing component's spatial position, attitude information, and assembly requirements. Simultaneously, the monitoring system provides real-time feedback on the adjustment status of the wing component's spatial position and attitude information to ensure consistency with the robotic arm's movement commands.

[0087] In summary, this application has the following advantages: (1) Since this application uses a binocular camera to collect data on the wing components, the depth information of the wing components is calculated to achieve three-dimensional spatial positioning of the wing contour. Obviously, this application not only improves the accuracy of image recognition, the accuracy and reliability of wing component assembly, but also can adapt to different wing shapes and structures.

[0088] (2) By combining pre-stored digital model data and utilizing feature point matching algorithms, this application can also achieve efficient and accurate contour matching. Furthermore, the automatic detection and feedback mechanism during the matching process can correct any potential errors in real time, ensuring the accuracy and reliability of the matching.

[0089] (3) Since this application uses a binocular camera to achieve real-time tracking of the wing component at the end of the robotic arm, it ensures that the position and attitude of the wing component can be accurately captured during the assembly process, providing high-precision spatial positioning. This real-time tracking technology realizes closed-loop control of the robotic arm's movement, significantly reducing assembly deviations caused by robotic arm movement errors.

[0090] (4) After the matching is completed, combined with automatic control technology, the robotic arm can be controlled to perform automatic and precise assembly of the wing contour. The entire assembly process is highly automated, reducing human error and improving assembly efficiency and accuracy.

[0091] It is evident that this application provides a new approach to complex assembly tasks in the field of intelligent assembly. By combining image registration and target tracking technologies and comprehensively considering multiple processes and data parameters, it achieves spatial positioning of the wing contour across all processes and multiple data parameters, ensuring the accuracy of contour positioning. This solves the problems of difficult assembly, inaccurate structural docking, and low reliability in existing technologies, providing reliable technical support for the aviation manufacturing industry.

[0092] In some possible implementations, various aspects of the methods provided in this application can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of the methods according to the various exemplary embodiments of this application described above. For example, the computer device may perform actions such as... Figure 3 The method performed by the spatial positioning device based on wing profile feature constraints in the illustrated embodiment.

[0093] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. Alternatively, if the integrated units of this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0094] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0095] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A spatial positioning method based on wing profile feature constraints, characterized in that, The method includes: Initialize the stereo camera; A binocular camera is used to acquire images of the fixed wing components of the robotic arm to obtain target images; The wing contour in the target image is matched with a pre-stored mathematical model to obtain a matching result; wherein the matching result contains multiple matching pairs; each matching pair contains one image feature point of the target image and one mathematical model feature point of the pre-stored mathematical model; Based on the matching results, determine whether the multiple matching pairs are successfully matched; If the multiple matching pairs are determined to be successfully matched, the robotic arm is controlled to assemble the wing components.

2. The method as described in claim 1, characterized in that, The steps for initializing the stereo camera include: Calibrate the activated binocular camera; The focal length and viewing angle of the binocular camera are adjusted according to the working area of ​​the robotic arm; Determine whether the position of the wing component is within the predetermined assembly area; If it is determined that the wing component is not located within the predetermined assembly area, the position of the automated guided vehicle (AGV) is adjusted.

3. The method as described in claim 1, characterized in that, The step of acquiring images of the target image by using a binocular camera to fix the wing component of the robotic arm includes: The binocular camera is used to acquire images of the wing component from both left and right perspectives, obtaining left-view and right-view images; The left-view image and the right-view image are fused to obtain an initial fused image; A preset image processing algorithm is used to determine whether all key features of the wing contour exist in the initial fused image; If it is determined that all the key features of the wing profile exist in the initial fused image, then the initial fused image is preprocessed to obtain the target image.

4. The method as described in claim 3, characterized in that, The step of using the Canny edge detection algorithm to determine whether all key features of the wing contour exist in the initial fused image includes: The Canny edge detection algorithm is used to determine whether the initial fused image contains all predefined key feature regions; If it is determined that the initial fused image contains all predefined key feature regions, then the image feature points in the initial fused image are compared with the mathematical model feature points in the pre-stored mathematical model to determine whether all the key features of the wing contour exist in the initial fused image.

5. The method as described in claim 1, characterized in that, The step of matching the wing contour in the target image with a pre-stored mathematical model to obtain a matching result includes: The target image is corrected using a preset stereo correction matrix to obtain a corrected image; Feature extraction is performed on the wing contour in the corrected image to obtain the image feature vector; Based on the similarity between the image feature vector and the mathematical model feature vector of the pre-stored mathematical model, multiple initial matching pairs are filtered to obtain multiple first-filtered matching pairs; Calculate the first alignment error of each first screening matching pair according to the preset alignment error calculation formula; Determine whether each of the first alignment errors meets the preset alignment threshold; If it is determined that each of the first alignment errors meets the preset alignment threshold, then the matching is successful and the matching result is obtained.

6. The method as described in claim 5, characterized in that, After determining whether each of the first alignment errors meets a preset alignment threshold, the method further includes: If it is determined that the various first alignment errors do not meet the preset alignment threshold, then the RANSAC algorithm is used to filter the multiple first screening matching pairs again to obtain multiple second screening matching pairs. According to the preset alignment error calculation formula, the alignment parameters between the corrected image and the pre-stored mathematical model are calculated; wherein, the alignment parameters include a rotation matrix and a translation vector; Based on the alignment parameters, the image feature points in the plurality of second filtering matching pairs are transformed into the digital model feature point coordinate system to obtain a plurality of image transformed feature points; Calculate the second alignment error between multiple image transformation feature points and multiple digital model feature points; Determine whether the second alignment error meets the preset alignment threshold.

7. The method as described in claim 1, characterized in that, The step of determining whether the plurality of matching pairs are successfully matched based on the matching results includes: The matching error of the matching result is obtained by calculating the matching error of the multiple matching pairs; Determine whether the matching error is within a preset acceptable range; If the matching error is determined to be within a preset acceptable range, then the matching is successful; If the matching error is determined to be outside the preset acceptable range, the matching will fail.

8. The method as described in claim 7, characterized in that, After a match fails, the method further includes: The focal length and angle of the binocular camera are adjusted; The adjusted binocular camera was used to re-acquire images of the wing components, resulting in re-acquired images. The wing contour in the resampled image is matched with a pre-stored mathematical model to obtain the matching result.

9. The method as described in claim 1, characterized in that, The step of controlling the robotic arm to assemble the wing components if the multiple matching pairs are determined to be successfully matched includes: If the multiple matching pairs are determined to be successfully matched, then the spatial position and attitude information of the wing component and the motion path of the robotic arm are determined based on the matching results. The binocular camera is calibrated according to a preset calibration formula; Based on the spatial position and attitude information of the wing components and the movement path of the robotic arm, the robotic arm is controlled to assemble the wing components.

10. A spatial positioning device based on wing profile feature constraints, characterized in that, The device includes: a binocular camera, a robotic arm, an automated guided vehicle (AGV), and a host computer; The binocular camera is used to acquire images of the wing components fixed to the robotic arm; The robotic arm is used to assemble fixed wing components; The automated guided vehicle (AGV) is used to adjust the position of the wing component to a predetermined assembly area; The host computer is used to control the binocular camera to initialize; control the binocular camera to acquire images of the wing component fixed by the robotic arm to obtain a target image; match the wing contour in the target image with a pre-stored mathematical model to obtain a matching result; wherein, the matching result contains multiple matching pairs; each matching pair contains one image feature point of the target image and one mathematical model feature point of the pre-stored mathematical model; based on the matching result, determine whether the multiple matching pairs are successfully matched; if it is determined that the multiple matching pairs are successfully matched, control the robotic arm to assemble the wing component.

Images