A Visual Detection Method for Misfitting or Missing Aircraft Wing Connectors

Abstract

A visual detection method for misassembly and omission of aircraft wing surface connectors is proposed. First, deep learning-based part identification and localization are performed. Then, a part measurement algorithm based on image segmentation is constructed. Next, the algorithm is integrated and verified. Then, image normalization based on transmission transform is performed. Finally, an assembly error identification algorithm based on data matching is applied. Finally, assembly pose estimation is performed. This invention achieves accurate part identification and measurement, efficient location and detection of assembly errors, and performance verification testing of the algorithm, improving both assembly inspection efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of small parts misassembly and omission detection technology, specifically to a visual detection method for misassembly and omission of aircraft wing surface connectors. Background Technology

[0002] Advanced aviation equipment, as a crucial component of modern national defense, accounts for approximately 65% ​​of the total manufacturing cycle. Currently, this assembly cycle is a significant factor contributing to insufficient production capacity. Assembly quality inspection, an indispensable part of the assembly process, is primarily conducted manually. For example, aircraft wing surfaces contain tens of thousands of connecting parts, and about 30% of rejection orders are related to misassembly or omissions. Similarly, aircraft have approximately 150 types of corner pieces, with an average of 60 misassemblies per aircraft. Current inspection methods rely on workers using manual tools to inspect each part individually, which is not only inefficient and requires high skill levels but also introduces human error. Therefore, significantly improving the efficiency of detecting misassemblies and omissions in small aircraft parts can boost the production capacity of advanced aviation equipment.

[0003] Improving the detection approach for mis-assembled or missing parts can shift from manual, one-by-one inspection to batch inspection methods. Visual and image algorithms can extract visual features of parts and assemblies, and perform processing, classification, and combination of these features to ultimately achieve part classification, identification, measurement, and location. Specifically, deep learning-based object detection algorithms can locate and identify parts, while image processing algorithms can measure part dimensions through edge detection and image segmentation. Combining these two methods allows for the simultaneous measurement of multi-dimensional information about parts in a single input, thereby enabling the detection of mis-assembled or missing parts. This significantly improves both assembly inspection efficiency and accuracy. Currently, no relevant literature has been published. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the present invention aims to provide a visual detection method for misassembly and omission of aircraft wing surface connectors, which realizes accurate identification and measurement of parts, efficient location and detection of assembly errors, and performance verification testing of algorithms, thereby improving assembly inspection efficiency and accuracy.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A visual detection method for misinstallation or omission of aircraft wing surface connectors includes the following steps:

[0007] Step 1), Deep learning-based part identification and localization: Build a deep learning network framework and use a public dataset to train the model as pre-training weights; collect and label the part dataset, and enrich the part dataset using image enhancement methods; use transfer learning to train and test the deep learning network on the pre-training weights to identify the pixel coordinates and type information of the parts;

[0008] Step 2), Construction of part measurement algorithm based on image segmentation: The image is preprocessed, and a depth-first search algorithm is used to perform connected component analysis and mark the parts and background; for each mark, an image segmentation algorithm is used to obtain the segmented region of the part instance; the minimum circumcircle is calculated for each segmented region of the part instance, and the radius of the minimum circumcircle is used as the scale feature descriptor;

[0009] Step 3), Algorithm Integration and Verification: Integrate the two algorithms described in Step 1) and Step 2) to form an integrated algorithm for part identification, positioning and measurement. The algorithm can obtain the pixel coordinates, category information and minimum circumscribed circle size of the part at one input. Experiments are conducted to verify the part type recognition rate and recall rate, and the accuracy range of size measurement.

[0010] Step 4), Image normalization processing based on transmission transformation: The pixel coordinates of the parts are obtained from the integrated algorithm for part identification, positioning and measurement in Step 3), and the coordinate transformation is achieved through transmission projection transformation; the assembly area is segmented, and quadrilaterals are fitted to the segmented areas. The parameters of the transmission transformation matrix are solved using the corner information of the fitted quadrilaterals; the transmission transformation matrix is ​​used to perform calculations to achieve image normalization processing, and at the same time, the conversion between the pixel coordinate system and the object coordinate system is realized.

[0011] Step 5), Assembly error identification algorithm based on data matching: Establish a database, define part information, and pre-store the correct assembly part information template in the database; develop an information transmission interface between the algorithm and the database to realize real-time data interaction between the algorithm and the database; store the part information measured in Step 4) in the database in real time, match it with the template information in the database, return the part information that fails to match, and define the assembly error information;

[0012] Step 6), Assembly pose estimation: Pose estimation is performed on thin-walled, weakly textured targets using an end-to-end deep learning approach, utilizing the RGB-D information of the image for direct pose estimation.

[0013] The beneficial effects of this invention are as follows:

[0014] (1) This invention proposes an image measurement algorithm based on image segmentation, which solves the problem that deep learning cannot perform high-precision measurement of targets, and integrates it with deep learning algorithms to form an integrated algorithm that can calculate multi-dimensional information of parts in a single input.

[0015] (2) This invention proposes a coordinate transformation method based on transmission transformation, thereby realizing the transformation between the pixel coordinates of the part and the coordinate system of the assembly object. This is beneficial for quickly locating the real position of the part in combination with the assembly model, and to a certain extent realizing the normalization processing of the input image.

[0016] (3) The present invention uses database template matching to accurately obtain the location information of assembly errors, and can adaptively select the matching accuracy by adjusting the parameter values ​​in the matching algorithm to adapt to various scenarios.

[0017] (4) The present invention calculates the position of the assembly error relative to the assembly and the pose of the assembly relative to the camera in two steps, which can indirectly and accurately locate the spatial position of the assembly error. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the process of the present invention.

[0019] Figure 2 This is a flowchart illustrating the database template matching method of the present invention;

[0020] Figure 3 The diagrams show the input and output of the integrated algorithm for part identification, positioning, and measurement; (a) is the input image, (b) is the output image of the YOLO_v5 algorithm, (c) is the output image of the part measurement algorithm, and (d) is the output image of the integrated algorithm.

[0021] Figure 4 The diagrams show the image normalization process based on transmission transformation; (a) is the input image; (b) is the quadrilateral fitting result; (c) is the corner selection diagram; and (d) is the transmission transformation diagram. Detailed Implementation

[0022] The present invention will now be described in detail with reference to embodiments and accompanying drawings. The present invention can be implemented in many different forms and should not be considered limited to the embodiments described herein. Rather, these embodiments are provided so that the present invention will be thoroughly and completely disclosed and will fully express the scope of the invention to those skilled in the art.

[0023] like Figure 1 As shown, a visual inspection method for misassembly or omission of aircraft wing surface connectors includes the following steps (the assembly is an aircraft wing surface, with 7 types of parts selected, each with multiple sizes and specifications, and a total of 32 connection holes):

[0024] Step 1), Deep Learning-Based Part Recognition and Localization: A deep learning network framework is built, and a publicly available dataset is used to train the model as pre-trained weights; a parts dataset is collected and labeled, and image enhancement techniques are used to enrich the parts dataset; transfer learning is used to train and test the deep learning network on the pre-trained weights to identify the pixel coordinates and type information of the parts; the specific steps are as follows:

[0025] Step 1.1) Construct the YOLO_v5 algorithm framework, train the model based on the public dataset, obtain the pre-trained weights, and use transfer learning to port the model trained on the public dataset to this scene; collect photos from different angles and under different lighting conditions as the parts dataset, and use the LabelImg open-source image annotation tool for annotation; the annotation information includes target type and location information;

[0026] Step 1.2) The part dataset includes 8 types: hole, black1, black2, red1, red2, yellow1, yellow2, and blue. If the algorithm identifies the part as hole, it means that the part is not assembled here, which is a missing part. If the algorithm identifies the part as other, it means that the part is assembled here.

[0027] Step 1.3) Create and store the collected old mechanical parts images in TXT format. To address the problem of insufficient training samples, image enhancement techniques such as flipping, rotating, cropping, occlusion, tone transformation, and brightness transformation are used to enrich the parts dataset. The deep learning network is then retrained to ultimately achieve the recognition of parts pixel coordinates and types.

[0028] Step 2), Construction of a part measurement algorithm based on image segmentation: To address the challenge of identifying parts of the same type but different scales due to the commonality of scale invariance in deep learning, the image is preprocessed, and a depth-first search algorithm is used for connected component analysis, with parts and background labeled. For each label, an image segmentation algorithm is applied to obtain the segmented region of the part instance. The minimum circumcircle is calculated for each segmented region of the part instance, and the radius of the minimum circumcircle is used as the scale feature descriptor. The specific steps are as follows:

[0029] Step 2.1) Based on the OpenCV open-source library functions, threshold segmentation is performed on the preprocessed filtered and denoised image to obtain a binarized image. Multiple erosion and dilation iteration operations are used to eliminate pixel holes and boundary interference inside the part.

[0030] Step 2.2): The image processed in Step 2.1) is first subjected to a depth-first search algorithm to mark and sort the background and different individual parts; the image watershed algorithm is then used to segment each mark to obtain the segmented regions of each part instance.

[0031] Step 2.3): Correct the boundary of the segmented region of the above part instance using the following formula:

[0032]

[0033]

[0034] Where i is the x-coordinate of the current integer edge point, j is the y-coordinate of the current integer edge point, and R... 左 R is the gradient value to the left of the edge point. 右 R0 is the gradient value to the right of the edge point, ω is the distance from the adjacent pixel to the edge point, and θ is the direction of the gradient.

[0035] Step 2.4) Correct the results to obtain closed contours, extract the minimum circumcircle of each contour, and use the minimum circumcircle radius as the dimensional feature descriptor of the part to realize the size measurement and sorting of the part.

[0036] Step 3), Algorithm Integration and Verification: Integrate the two algorithms described in Step 1) and Step 2) to form an integrated algorithm for part identification, positioning, and measurement. This algorithm obtains the pixel coordinates, category information, and minimum circumscribed circle dimension of the part with a single input. Multiple experiments are conducted under complex environments with varying lighting conditions to verify indicators such as part type recognition rate, recall rate, and dimensional measurement accuracy range. The specific steps are as follows:

[0037] Step 3.1) Integrate the part type identification and localization algorithm designed in Step 1) and the part measurement algorithm designed in Step 2) in parallel integration. The input image is first processed by the YOLO_v5 deep learning algorithm, and then fed into the part measurement algorithm of Step 2) at the output layer for calculation and sorting. The sorted parts are then marked.

[0038] Step 3.2): For the first marked part, calculate the conversion ratio between its actual size and pixel size based on its actual size and the size measured by the algorithm.

[0039]

[0040] Where r 真实 r is the minimum circumscribed circle radius of the actual part. 测量 The minimum circumscribed circle radius of the part is obtained by the algorithm; based on this ratio, the formula r is used. 真实 =r 测量The ratio is used to obtain the true minimum circumcircle radius of the remaining parts, ultimately forming an integrated algorithm for part identification, positioning, and measurement.

[0041] Step 3.3) The precision P and recall R are used to evaluate the algorithm performance of the category identification and positioning module. The calculation formulas are as follows. The repeatability accuracy of multiple measurements is used to evaluate the algorithm performance of the part measurement module:

[0042]

[0043]

[0044] Where TP represents correct results found, FN represents correct results not found; FP represents results found but incorrect, and TN represents results not found and incorrect.

[0045] Step 4), Image normalization processing based on transmission transformation: The pixel coordinates of the parts are obtained from the integrated algorithm for part recognition, positioning, and measurement in Step 3), and coordinate transformation is achieved through transmission projection transformation; the assembly region is segmented, and quadrilaterals are fitted to the segmented regions. The parameters of the transmission transformation matrix are solved using the corner information of the fitted quadrilaterals; image normalization processing is achieved using the transmission transformation matrix, and the conversion between the pixel coordinate system and the object coordinate system is realized simultaneously; the specific steps are as follows:

[0046] Step 4.1), solve for the projection transformation matrix, as follows:

[0047] ① Solve for the target: the 8 parameters of the transmission transformation matrix M (3×3);

[0048] ② Analysis of solution conditions: Solving for 8 parameters requires 8 known conditions to construct 8 equations, therefore the coordinates of the four points known in the figure are needed;

[0049] ③ Solution process: The input image is segmented into the assembly region using Mask_rcnn. The segmented region is then fitted with quadrilaterals using OpenCV library functions to solve for the coordinates of the four corner points of the quadrilateral. The remaining known conditions are the width and height of the transformed image. The transmission transformation matrix is ​​then solved based on the above known conditions.

[0050] Step 4.2) The transformation is completed using the solved transmission transformation matrix, as follows:

[0051] First, project the 2D image onto a 3D view plane, and then transform it back to 2D coordinates; let the transformation matrix be M:

[0052]

[0053]

[0054] Where x2, y2, and z2 are the three coordinates after the transmission transformation, x1 and y1 are the two-dimensional coordinates before the transmission transformation, M is the transmission transformation matrix, and a1, a2, a3, b1, b2, b3, c1, c2, and c3 are the nine parameters of the transmission transformation matrix; the following formula can be obtained:

[0055] x2=a1x1+b1y1+c1

[0056] y2 = a2x1 + b2y1 + c2

[0057] z2 = a3x1 + b3y1 + c3

[0058] Since the calculation result is a three-dimensional coordinate, the obtained three-dimensional coordinate values ​​need to be converted to two-dimensional coordinates using the following formula:

[0059]

[0060]

[0061] In the calculation, c3 is usually set to 1. Therefore, there are 8 parameters to solve. Thus, the above steps obtain the coordinates of four points, construct a system of equations, use Cramer's rule to calculate the corresponding parameters, and finally complete the transformation to realize the conversion between the pixel coordinate system and the object coordinate system, while realizing the normalization of the image.

[0062] Step 5), Assembly error identification algorithm based on data matching: Establish a database, define part information, and pre-store correct assembly part information templates in the database; develop an information transmission interface between the algorithm and the database to achieve real-time data interaction between the algorithm and the database; store the part information measured in Step 4) in the database in real time, match it with the template information in the database, and return the part information that fails to match to define the assembly error information; the specific steps are as follows:

[0063] Step 5.1) Establish a MySQL database and define the part information, which is defined as follows: each part is described by five attributes, namely serial number (No), center point x-coordinate (x_coor), center point y-coordinate (y_coor), type (kind), and size (size); after setting the part attributes, establish a database model in the MySQL database to store each part.

[0064] Step 5.2) Use the PyMySQL third-party library to establish the algorithm and database interface; first connect to the database and create a real-time interaction interface between the program and the database. During the execution of the algorithm in steps 2) and 3), store the data in the data table in real time.

[0065] Step 5.3), the database template matching process is as follows: Figure 2 As shown, the algorithm runs sequentially from beginning to end according to the data table. For each part, it first compares its type (kind) with the template information. If the types are different, the part information is returned directly. If the types are the same, the size information (size) of the two is compared. According to the test, the ratio of the pixel size measured by the algorithm to the actual size is about 9:1. Considering that the minimum difference of the actual size is 1mm, in order to make the resolution of the algorithm sufficient to distinguish the parts, a fluctuation of 0.2mm is selected as the threshold for template matching. If the size difference between the two exceeds 0.2mm, they are regarded as different parts, that is, this is defined as an assembly error, and the part information is returned.

[0066] Step 6), Assembly pose estimation: Pose estimation is performed on the thin-walled, weakly textured target using an end-to-end deep learning approach, utilizing the RGB-D information of the image for direct pose estimation; the specific steps are as follows:

[0067] Step 6.1) Construct the EfficientPose network structure and set up the EfficientPose algorithm runtime environment;

[0068] Step 6.2) Create the Linemod dataset using a video stream. The target pose in the first frame is determined using Aruco code, and each subsequent frame is obtained by ICP registration with the previous frame.

[0069] Step 6.3) Train the EfficientPose network and quantitatively evaluate the pose estimation accuracy of the algorithm.

[0070] Step 6.4) Position the assembly relative to the camera based on the calculated pose matrix, and indirectly obtain the spatial position of the assembly error by combining the position of the assembly error relative to the assembly determined in Step 5).

[0071] In summary, this invention discloses a visual detection method for misassembly and omission of aircraft wing surface connectors. This method mainly includes six steps: deep learning-based part identification and localization, image segmentation-based part measurement, algorithm integration and verification, image normalization based on transmission transformation, assembly error identification based on data matching, and assembly pose estimation. This invention first proposes an image measurement algorithm based on image segmentation, thus solving the deficiency of deep learning in high-precision target measurement. It integrates with the deep learning algorithm to form an integrated algorithm for part identification, localization, and measurement, which can calculate multi-dimensional information of parts in a single input. It proposes a coordinate transformation method based on transmission transformation, thereby realizing the conversion between the pixel coordinates of parts and the coordinate system of the assembly object. This facilitates rapid localization of the actual position of parts using the assembly model and achieves a certain degree of normalization of the input image. It employs a database template matching method to accurately obtain the location information of assembly errors, and the matching accuracy can be adaptively selected by adjusting the parameter values ​​in the matching algorithm to adapt to various scenarios.

[0072] To verify the feasibility and effectiveness of the method of the present invention, an experiment was conducted on a typical scenario, and the accuracy of the method was evaluated.

[0073] like Figure 3 As shown, Figure 3 This is an integrated algorithm for part recognition, positioning, and measurement based on deep learning and image segmentation. The input image contains 32 targets, which are divided into 8 categories. After testing, the accuracy and recall of deep learning target detection in this scenario generally reached about 95%; while the part measurement algorithm had a part measurement error range of less than 0.5mm.

[0074] like Figure 4 As shown, Figure 4 For image normalization processing based on transmission transformation, the transformation between pixel coordinate system and object coordinate system is completed through transmission transformation.

[0075] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0076] It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A visual detection method for misinstallation or omission of aircraft wing surface connectors, characterized in that, Includes the following steps: Step 1), Deep learning-based part identification and localization: Build a deep learning network framework and use a public dataset to train the model as pre-trained weights; A parts dataset was collected and labeled, and image enhancement techniques were used to enrich the parts dataset. A deep learning network was trained and tested on pre-trained weights using transfer learning to identify the pixel coordinates and type information of the parts. Step 2), Construction of part measurement algorithm based on image segmentation: The image is preprocessed, connected component analysis is performed using the depth-first search algorithm, and the parts and background are marked; for each mark, the image segmentation algorithm is used to obtain the segmented region of the part instance; For each part instance segmentation region, calculate the minimum circumcircle and use the minimum circumcircle radius as the scale feature descriptor; Step 3), Algorithm Integration and Verification: Integrate the algorithms from Step 1) and Step 2) to form an integrated algorithm for part identification, positioning, and measurement. This algorithm obtains the pixel coordinates, category information, and minimum circumscribed circle dimension of the part with a single input. Experiments are then conducted to verify the part type recognition rate and recall rate, as well as the accuracy range of dimensional measurement. Specifically: Step 3.1) Integrate the part type identification and localization algorithm designed in Step 1) and the part measurement algorithm designed in Step 2) in parallel integration. The input image is first processed by the YOLO_v5 deep learning algorithm, and then fed into the part measurement algorithm of Step 2) at the output layer for calculation and sorting. The sorted parts are then marked. Step 3.2), for the first marked part, calculate the conversion ratio between the actual size and the pixel size based on its actual size and the size measured by the algorithm: in The minimum circumscribed circle radius of the actual part. The minimum circumscribed circle radius of the part is obtained by the algorithm; based on this ratio, the formula is used... The true minimum circumcircle radius of the remaining parts is obtained, and finally an integrated algorithm for part identification, positioning and measurement is formed. Step 3.3) The precision P and recall R are used to evaluate the algorithm performance of the category identification and positioning module. The calculation formula is as follows. The repeatability accuracy of multiple measurements is used to evaluate the algorithm performance of the part measurement module: in, TP It is the correct result retrieved. FN The correct result was not found. FP It was found but the result was incorrect. TN The result was not found and was incorrect. Step 4), Image normalization processing based on transmission transformation: The pixel coordinates of the parts are obtained from the integrated algorithm for part identification, positioning and measurement in Step 3), and coordinate transformation is achieved through transmission projection transformation; the assembly area is segmented, and quadrilateral fitting is performed on the segmented areas. The parameters of the transmission transformation matrix are solved using the corner information of the fitted quadrilateral; the transmission transformation matrix is ​​used to perform calculations to achieve image normalization processing, and at the same time, the conversion between the pixel coordinate system and the object coordinate system is realized. Step 5), Assembly error identification algorithm based on data matching: Establish a database, define part information, and pre-store correct assembly part information templates in the database; develop an information transmission interface between the algorithm and the database to realize real-time data interaction between the algorithm and the database; store the part information measured in Step 4) in the database in real time, match it with the template information in the database, return the part information that fails to match, and define the assembly error information; Step 6), Assembly pose estimation: Pose estimation is performed on thin-walled, weakly textured targets using an end-to-end deep learning approach, utilizing the RGB-D information of the image for direct pose estimation.

2. The method according to claim 1, characterized in that, Step 1) Specifically: Step 1.1) Construct the YOLO_v5 algorithm framework, train the model based on the public dataset, obtain the pre-trained weights, and use transfer learning to port the model trained on the public dataset to this scene; collect photos from different angles and under different lighting conditions as the parts dataset, and use the LabelImg open-source image annotation tool for annotation; the annotation information includes target type and location information; Step 1.2) The part dataset includes 8 types: hole, black1, black2, red1, red2, yellow1, yellow2, and blue. If the algorithm identifies the part as hole, it means that the part is not assembled here, which is a missing part. If the algorithm identifies the part as other, it means that the part is assembled here. Step 1.3) Create and store txt-type tags for the collected old mechanical parts images. Enrich the parts dataset by using image enhancement methods such as flipping, rotating, cropping, occlusion, tone transformation, and brightness transformation. Retrain the deep learning network to finally achieve the recognition of the pixel coordinates and types of the parts.

3. The method according to claim 1, characterized in that, Step 2) specifically involves: Step 2.1) Based on the OpenCV open-source library functions, threshold segmentation is performed on the preprocessed filtered and denoised image to obtain a binarized image. Multiple erosion and dilation iteration operations are used to eliminate pixel holes and boundary interference inside the part. Step 2.2) First, a depth-first search algorithm is used to mark and sort the background and different individual parts of the image processed in Step 2.1); then, the image watershed algorithm is used to segment each mark to obtain the segmented region of each part instance. Step 2.3) Correct the boundary of the segmented region of the above part instance using the following formula: in, It is the x-coordinate of the current integer coordinate edge point. It is the y-coordinate of the current integer edge point. It is the gradient value to the left of the edge point. It is the gradient value to the right of the edge point. These are the gradient values ​​at the edge points. It is the distance from adjacent pixels to edge points. It is the direction of the gradient; Step 2.4) Correct the results to obtain closed contours, extract the minimum circumcircle of each contour, and use the minimum circumcircle radius as the dimensional feature descriptor of the part to realize the size measurement and sorting of the part.

4. The method according to claim 1, characterized in that, Step 4) specifically involves: Step 4.1), solve for the projection transformation matrix, as follows: ① Solve for the objective: the 8 parameters of the transmission transformation matrix M (3×3); ② Analysis of solution conditions: Solving for 8 parameters requires 8 known conditions to construct 8 equations, therefore the coordinates of the four points known in the figure are needed; ③ Solution process: Mask_rcnn is used to segment the assembly region of the input image. The segmented region is fitted with quadrilaterals using OpenCV library functions, and the coordinates of the four corner points of the quadrilaterals are solved. The remaining known conditions are the length (width) and height (height) of the transformed image. The transmission transformation matrix is ​​solved based on the known conditions. Step 4.2) uses the solved transmission transformation matrix to complete the transformation, as follows: First, project the 2D image onto a 3D view plane, and then transform it back to 2D coordinates; let the transformation matrix be... M : in , , These are the three coordinates after the transmission transformation. , These are the two-dimensional coordinates before the transmission transformation. It is the transmission transformation matrix. , , , , , , , , These are the nine parameters of the transmission transformation matrix; the following formula is obtained: Since the calculation result is a three-dimensional coordinate, the obtained three-dimensional coordinate values ​​need to be converted to two-dimensional coordinates using the following formula: In the calculation, let = 1, requiring the solution of 8 parameters, thus obtaining the coordinates of four points, constructing a system of equations, using Cramer's rule to calculate the corresponding parameters, and finally completing the transformation to realize the conversion between the pixel coordinate system and the object coordinate system, while also realizing the normalization processing of the image.

5. The method according to claim 1, characterized in that, Step 5) specifically involves: Step 5.1) Establish a MySQL database and define the part information, which is defined as follows: each part is described by five attributes, namely serial number (No), center point x-coordinate (x_coor), center point y-coordinate (y_coor), type (kind), and size (size); after setting the part attributes, establish a database model in the MySQL database to store each part. Step 5.2) uses the PyMySQL third-party library to establish the algorithm and database interface; first, connect to the database and create a real-time interaction interface between the program and the database. During the execution of the algorithm in steps 2) and 3), the data is stored in the data table in real time. Step 5.3) The database template matching process is as follows: it runs sequentially from beginning to end according to the order of the data table. For each part, it is first compared with the type (kind) of the template information. If the types are different, the information of the part is returned directly. If the types are the same, the size information (size) of the two is compared. The threshold for template matching is selected to fluctuate by 0.2mm. If the size difference between the two exceeds 0.2mm, they are regarded as different parts, that is, this is defined as an assembly error, and the information of the part is returned.

6. The method according to claim 1, characterized in that, Step 6) specifically refers to: Step 6.1) Construct the EfficientPose network structure and set up the EfficientPose algorithm runtime environment; Step 6.2) Create the Linemod dataset using a video stream. The target pose in the first frame is determined using Aruco code, and each subsequent frame is obtained by ICP registration with the previous frame. Step 6.3) Train the EfficientPose network and quantitatively evaluate the pose estimation accuracy of the algorithm; Step 6.4) Position the assembly relative to the camera based on the calculated pose matrix, and indirectly obtain the spatial position of the assembly error by combining the position of the assembly error relative to the assembly determined in Step 5).

Images