A method for assembly inspection of aerospace products based on visual recognition

Abstract

The present invention discloses a method for assembly inspection of aerospace products based on visual recognition, which includes the following steps: performing tilt correction, text detection and recognition on the flexible cable image to obtain coded content; matching the coded content with parts in a database to obtain a flexible cable assembly inspection result; obtaining the position and posture of the rigid structural part in a camera coordinate system based on the rigid structural part image, converting the position and posture into a world coordinate system and comparing the position and posture with the real position and posture data of the structural part to obtain a rigid structural part assembly inspection result; transmitting the flexible cable and rigid structural part assembly inspection result to a human-computer interaction interface of a software system, guiding an operator to gradually complete the assembly status inspection of each workpiece of the product. The present invention completes the assembly result inspection of flexible cables and rigid structural parts in aerospace products through machine vision, which can reduce the professional knowledge requirements for operators, save labor costs, and reduce detection errors and omissions caused by human factors.

Description

Technical Field

[0001] The present invention belongs to the technical field of aerospace assembly detection, and in particular relates to an aerospace product assembly detection method based on visual recognition. Background Art

[0002] Aerospace product assembly is a crucial component of modern aerospace engineering. Traditionally, aerospace products such as aircraft, aero engines, and launch vehicles have numerous components and complex assembly processes, with over half of their production cycle devoted to assembly. Complex operating environments can easily lead to misunderstandings of process documentation and assembly errors, resulting in lengthy assembly times and difficulties in ensuring quality. Therefore, testing and verification of assembly results are essential. However, this testing process primarily relies on manual inspection by workers, which is inefficient and relies heavily on subjective judgment.

[0003] With the development of visual inspection technology, machine vision provides effective technical support for aerospace product assembly quality inspection. Deep learning-based text recognition technology can effectively identify irregular or partially occluded text. Keypoint matching and model matching techniques can accurately extract and track the 3D pose of target objects from images. Multi-view geometric relationships can solve positioning problems that are difficult to determine using a single view.

[0004] However, the diverse and complex components of aerospace product assemblies, coupled with their rapid evolution, present challenges in developing visual recognition algorithms for these parts. Traditional approaches to developing visual recognition models, relying on individual machine learning training for each part, are ill-suited to the rapidly evolving nature of aerospace product assembly inspection. Furthermore, most developed models require CAD models of the parts and real-time depth information, hindering the widespread adoption of visual recognition technology in assembly inspection. Therefore, achieving efficient, accurate, and convenient inspection of aerospace product assembly results remains a pressing challenge. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned existing technologies and to provide a method for assembly inspection of aerospace products based on visual recognition. The parts assembled in aerospace products are divided into flexible cable parts and rigid structural parts for assembly inspection. For flexible cable parts with irregular shapes, a detection method based on coded text recognition is adopted. The coded text on the cable label is matched with the assembly task through computer vision recognition, and the assembly result inspection is realized by comparing with the process file. For rigid structural parts, a detection method based on visual recognition posture recognition algorithm is adopted. At the same time, the identified rigid part posture is compared with the part coordinate posture in the real-world coordinate system to complete the part assembly result inspection.

[0006] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0007] A method for detecting aerospace product assembly based on visual recognition, the method comprising:

[0008] S1: Inspection of flexible cable assembly for aerospace products: A visual recognition-based text recognition algorithm is used to perform cable code tilt correction, text detection, and recognition on flexible cable images captured by the camera to obtain the cable code content in the image.

[0009] S2: Match the code content identified in S1 with the parts in the assembly process database to obtain the flexible cable assembly inspection result;

[0010] S3: Inspection of the assembly status of rigid structural parts of aerospace products: Based on the images of rigid structural parts captured by the camera, the position and posture of the rigid structural parts in the camera coordinate system are obtained using the visual recognition pose estimation algorithm and the least squares regression algorithm;

[0011] S4: Convert the pose of the rigid structural component in the camera coordinate system to the pose in the world coordinate system, and compare the converted pose data with the actual pose data of the structural component to obtain the assembly inspection result of the rigid structural component;

[0012] S5: The inspection results of the assembly status of flexible cables and rigid structural parts are transmitted to the human-computer interaction interface of the software system. The workpieces that have completed the inspection and are assembled accurately are marked in the process library, and the operator is guided to gradually complete the assembly status inspection of each workpiece of the product.

[0013] To optimize the above technical solutions, specific measures taken also include:

[0014] The above-mentioned S1 performs cable code tilt correction, text detection and recognition on the flexible cable image captured by the camera based on the machine vision text recognition algorithm to obtain the cable code content in the image, including:

[0015] S1.1: Use the Hough transform to extract the straight line information in the cable image. Obtain the tilt angle of the extracted straight line by averaging. Then solve the rotation matrix M. Perform an affine transformation with the image center as the rotation center to complete the tilt correction of the cable code in the image.

[0016] S1.2: Use a convolutional network to extract features from the tilt-corrected cable image. Use a recurrent neural network to detect text boxes of variable length. Divide the entire coding area in the image into small text candidate boxes for cable coding text detection. Finally, the detected small boxes are spliced ​​together to obtain the complete cable coding text box, which is the cable coding area.

[0017] S1.3: The detected cable coding area is cropped and input into the CRNN network, and recognition is performed based on the image sequence to obtain the cable coding content.

[0018] The solution formula for the above rotation matrix M is as follows:

[0019]

[0020] Where α=cos(angle), β=sin(angle), center.x and center.y are the center coordinates of the image.

[0021] In the above S2, the edit distance between the cable code content string and all the part name strings in the assembly process database is calculated; and the part name string with the smallest edit distance value is selected as the matching result.

[0022] The above edit distance calculation formula is:

[0023]

[0024] where lev a,b (i, j) is the edit distance between the first i characters of the cable code content string a and the first j characters of b; 1(a i ≠b j ) is the indicator function, a i ≠b j It is 1 when it is, otherwise it is 0; the three formulas in the min operation represent deleting, inserting and replacing characters in a to achieve b.

[0025] In the above S3, the pose estimation algorithm based on RGB data is used to estimate the pose of the rigid structure in the camera coordinate system for the image of the rigid structure. The least squares regression algorithm is used to perform linear fitting on the experimentally measured data to obtain the pose of the rigid structure in the camera coordinate system at the real scale:

[0026] f(x i )=wx i +b

[0027] where f(x i ) is the converted coordinate data, that is, the position and orientation of the rigid structure of the real scale in the camera coordinate system; x i is the pose coordinate data of the rigid structure in the camera coordinate system output by the recognition algorithm before conversion, w is the conversion weight, and b is the recognition deviation.

[0028] The above w and b are obtained by minimizing the sum of the Euclidean distances from the experimental samples to the straight line. The formula for the sum of the Euclidean distances from the experimental samples to the straight line is as follows:

[0029]

[0030] Among them E (w,b) That is the sum of the Euclidean distances from the sample to the fitting line, y i is the real coordinate data of the structure, m is the number of experimental data samples;

[0031] E (w,b) Calculate the partial derivatives of w and b respectively, and make the partial derivatives equal to 0, and we get:

[0032]

[0033]

[0034] The mean

[0035] The above S4 pose conversion formula is as follows:

[0036]

[0037] Where T1 is the pose matrix of the rigid structure in the world coordinate system after the transformation is completed; T2 is the pose matrix of the rigid structure in the camera coordinate system before the transformation; The transformation matrix from the camera coordinate system to the world coordinate system.

[0038] In the above S4, if the deviation between the converted pose data and the actual pose data of the structural component does not exceed the error threshold, the assembly result of the rigid structural component is correct.

[0039] The present invention has the following beneficial effects:

[0040] The present invention proposes to divide aerospace product assembly inspection into two inspection methods: flexible cable parts inspection and rigid structural parts inspection, which can cover the vast majority of aerospace product assembly scenarios.

[0041] For cable parts identification, a detection scheme based on coded text recognition is proposed. This combines the CTPN text detection algorithm and the CRNN text recognition algorithm for cable code recognition. Image angle adjustment preprocessing optimization enhances the robustness of the text recognition algorithm for tilted text detection. A text similarity matching mechanism is used to associate text recognition results with cable assembly tasks. Assembly inspection is implemented in conjunction with a software system containing process documentation. This improved preprocessing scheme for the text recognition algorithm allows operators to perform cable recognition operations at various angles, enhancing the robustness of the existing recognition algorithm.

[0042] For the detection of rigid parts assembly results, a rigid structural parts detection method based on pose estimation is proposed. The visual recognition pose estimation algorithm based on RGB images is used to estimate the pose of rigid structural parts. The least squares regression algorithm is used to perform curve fitting on the pose estimation results, which solves the scale distortion problem of pose estimation without depth data. The correctness of the rigid structure assembly state is judged by comparing the pose results after scale conversion with the true pose. It does not require neural network training for specific rigid parts, nor does it require CAD models and depth information. It can effectively adapt to the assembly environment of aerospace products with fast parts update speed, wide variety and complex structure. The scale restoration improvement of the pose recognition model can effectively make up for the recognition result deviation caused by the lack of depth data, which has certain reference significance for the pose estimation algorithm based on RGB data.

[0043] The present invention uses machine vision to complete the assembly result inspection of flexible cables and rigid structural parts in aerospace products, which can reduce the professional knowledge requirements of operators, save labor costs, and reduce inspection errors caused by human factors.

[0044] The assembly inspection method proposed in this invention can be used for assembly inspection in the fields of aerospace, automobiles, machine tools, etc. The computer vision-based inspection method can be applied to PCs or mobile terminals, and can also be integrated into industrial robot control systems to guide robots to complete complex work processes. It has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the process of the present invention;

[0046] Figure 2 Assemble a test experimental platform for the present invention;

[0047] Figure 3 This is the flexible cable identification process in the present invention;

[0048] Figure 4 Schematic diagram of various coordinate systems for visual recognition pose estimation based on RGB data in the present invention;

[0049] Figure 5 The scale restoration fitting curve for the pose recognition data in the present invention;

[0050] Figure 6 The present invention assembles the detection software human-computer interaction interface. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] Although the steps in the present invention are arranged with numbers, they are not intended to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0053] The present invention provides a method for detecting the assembly of aerospace products based on visual recognition. It uses machine vision technology to detect the assembly status of flexible cables and rigid structural parts in aerospace products, thereby reducing the workload of operators, reducing the training cycle and training costs of assembly workers, and avoiding assembly errors caused by human subjective factors. Figure 1 The steps in the invention are further described. Figure 2 The verification platform of the present invention is shown, which consists of an assembly wall, a camera, flexible cables and rigid parts. The specific implementation steps are as follows:

[0054] S1: Inspection of flexible cable assembly for aerospace products: Using a machine vision text recognition algorithm, the flexible cable image captured by the camera is subjected to cable code tilt correction, text detection, and recognition to obtain the cable code content in the image.

[0055] In the embodiment, for the flexible cable assembly inspection, first pass Figure 2 The camera in the flexible cable captures the image of the coding area. The captured image needs to include the entire coding area. Figure 3 As shown in , the collected cable images are preprocessed, the images with large text inclinations are corrected, and the cable coding content in the preprocessed images is identified by a text recognition algorithm; S1 includes:

[0056] S1.1: Image preprocessing involves performing tilt correction on the code by detecting straight lines in the image. Hough transform is used to extract straight line information from the captured image. The tilt angle of the detected straight line is obtained by averaging. A rotation matrix is ​​obtained by solving the formula. Affine transformation is performed with the image center as the rotation center to complete image rotation correction. The rotation matrix solution formula is as follows:

[0057]

[0058] Where α = cos(angle), β = sin(angle), center.x and center.y are the center coordinates of the image;

[0059] After image preprocessing, we can get Figure 3The coded text shown in the image is in a horizontal state, and then enters the next step of coding area recognition.

[0060] S1.2: Cable coding text detection is performed based on the CTPN text detection algorithm. The convolutional network is used to extract cable image features after rotation correction of the preprocessed image. The recurrent neural network is used to detect text boxes of variable length. The entire coding area is divided into small text candidate boxes for detection. Finally, the detected small boxes are spliced ​​together to obtain the complete text box. Figure 3 The image after the encoding region detection.

[0061] S1.3: The detected cable coding area is cropped and input into the CRNN network for text recognition. The CRNN network is used to perform recognition based on the image sequence to obtain the content of the coded text in the text box, thereby realizing content recognition in the variable-length coding area.

[0062] S2: Match the code content identified in S1 with the parts in the assembly process database to obtain the flexible cable assembly inspection result;

[0063] In the embodiment, the coded content identified in S1 is compared and matched with the corresponding task number in the assembly process database based on text similarity, completing the automatic mapping between the coded image and the process task, providing a reference object for checking the cable assembly status;

[0064] The Edit Distance algorithm is used as the similarity measurement method to identify the cable coding results (such as Figure 3 The similarity between the cable name in the software system database is calculated. Between the two strings, one of the strings is converted into the second string after the least number of operations by inserting characters, deleting characters, and replacing characters. The number of operations at this time is the edit distance. a represents the encoding recognition result string (i.e. CR20815), b represents the part name string in the database (such as CR20816), i and j represent the subscripts of the characters in the two strings a and b, and the edit distance between the first i characters of string a and the first j characters of string b is lev a,b (i,j) is:

[0065]

[0066] Among them 1(a i ≠b j ) is an indicator function, a i ≠b j It is 1 when , otherwise it is 0. The three formulas in the min operation represent deleting, inserting and replacing characters in a to achieve b.

[0067] After calculating the edit distance between the encoding recognition result string and all the part name strings in the database, the part name character with the smallest edit distance value is selected as the matching result. The software system can call up the assembly process file of the corresponding cable based on the matching result, and display the assembly information in the process file on the human-computer interaction interface of the software to guide the operator to check the key information and assist in completing the assembly result detection;

[0068] S3: Inspection of the assembly status of rigid structural parts of aerospace products: Based on the images of rigid structural parts captured by the camera, the position and posture of the rigid structural parts in the camera coordinate system are obtained using the visual recognition pose estimation algorithm based on RGB data and the least squares regression algorithm;

[0069] In the embodiment, for the inspection of the assembly status of rigid structural parts, the camera first collects the assembly status picture of the rigid parts, and uses the visual recognition pose estimation algorithm based on RGB data to calculate the pose of the structural parts relative to the camera coordinate system, that is, Figure 4 The pose of the rigid part in the camera coordinate system is then restored using the least squares regression algorithm to restore the coordinate scale that is distorted due to the lack of depth data;

[0070] In S3, the visual recognition pose estimation algorithm based on RGB data is a pose estimation method based on RGB image input. By collecting some pictures of rigid parts at different angles in advance and inputting them into the algorithm, the recognition of rigid parts can be achieved. Due to the lack of input of depth data and CAD model, the pose data output by the algorithm has scale distortion. To solve the problem of pose scale distortion, the least squares regression algorithm is used to perform linear fitting on the experimentally measured data to obtain the fitting straight line formula:

[0071] f(x i )=wx i +b

[0072] where f(x i ) is the coordinate data after conversion, x i The coordinate data output by the prediction algorithm before conversion, w is the conversion weight, and b is the recognition deviation. w and b are obtained by minimizing the sum of the Euclidean distances from the experimental sample to the straight line. The formula is as follows:

[0073]

[0074] Among them E (w,b) That is the sum of the Euclidean distances from the sample to the fitting line, y i is the real coordinate data of the structure, m is the number of experimental data samples, and E (w,b) We can obtain the partial derivatives of w and b respectively and make them equal to 0:

[0075]

[0076]

[0077] in

[0078] Perform least square fitting on the X, Y, and Z directions respectively, and after the calculation is completed, you can get the following Figure 5 The three-direction fitting straight line shown in the figure, the horizontal axis is the output data of the pose recognition algorithm, and the vertical axis is the actual coordinate data of the rigid part relative to the camera coordinate system. Using the fitted straight line formula, the coordinate data of the rigid part in the image identified by the pose recognition algorithm is input, and the coordinate data of the rigid part in the real scale is output.

[0079] S4: Convert the pose of the rigid structural component in the camera coordinate system to the pose in the world coordinate system, and compare the converted pose data with the actual pose data of the structural component to obtain the assembly inspection result of the rigid structural component;

[0080] In this embodiment, the pose of the structural component obtained in S3 in the camera coordinate system is converted to a pose in the world coordinate system through the coordinate conversion matrix, and the pose data is compared with the actual pose data of the structural component to determine whether the deviation between the two exceeds the error threshold; the pose conversion formula is as follows:

[0081]

[0082] Where T1 is the position of the rigid structural part after the conversion is completed. Figure 4 The pose matrix in the world coordinate system; T2 is the pose matrix of the rigid structure before transformation. Figure 4 The pose matrix in the camera coordinate system is obtained by rescaling the output pose of the visual recognition pose estimation algorithm based on RGB data; is the transformation matrix from the camera coordinate system to the world coordinate system. This matrix is ​​obtained using the camera calibration method based on the Pnp algorithm. The camera is identified by Figure 4 The QR code on the assembly panel is used to calibrate its own coordinates. The center of the QR code is used as the origin of the world coordinate system, the panel plane is the XOY plane of the world coordinate system, and the outward direction of the panel normal is the positive direction of the Z axis of the world coordinate system. The rotation matrix R and translation T of the camera relative to the world coordinate are obtained by marking. The two are combined to obtain the transformation matrix

[0083]

[0084] After the conversion is completed, the estimated pose of the rigid part in the world coordinate system is compared with the pose data marked on the rigid part in the process file. If the data deviation is within the allowable range, the assembly result is correct.

[0085] S5: The inspection results of the assembly status of flexible cables and rigid structural parts are transmitted to the human-computer interaction interface of the software system. The workpieces that have completed the inspection and are assembled accurately are marked in the process library, and the operator is guided to gradually complete the assembly status inspection of each workpiece of the product.

[0086] In the embodiment, the detection results of the flexible cable and the rigid structure are transmitted to Figure 6 The human-computer interaction interface of the software system marks the workpieces that have completed inspection and are assembled accurately in the process library, and guides the operator to gradually complete the assembly status inspection of each workpiece of the product.

[0087] The software system is an assembly inspection software established based on the assembly process files of aerospace products. It can be run on PCs, tablets, and mobile phones. The human-computer interaction interface of the software automatically obtains assembly process information based on the detection and recognition results, marks the assembly processes that have completed the inspection, and guides the operator to inspect the remaining assembly process steps.

[0088] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

[0089] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A method for detecting aerospace product assembly based on visual recognition, characterized in that: The method includes: S1: Inspection of flexible cable assembly for aerospace products: Using a machine vision text recognition algorithm, the flexible cable image captured by the camera is subjected to cable code tilt correction, text detection, and recognition to obtain the cable code content in the image. S2: Match the code content identified in S1 with the parts in the assembly process database to obtain the flexible cable assembly inspection result; S3: Inspection of the assembly status of rigid structural parts of aerospace products: Based on the images of rigid structural parts captured by the camera, the position and posture of the rigid structural parts in the camera coordinate system are obtained using the visual recognition pose estimation algorithm based on RGB data and the least squares regression algorithm; S4: Convert the pose of the rigid structural component in the camera coordinate system to the pose in the world coordinate system, and compare the converted pose data with the actual pose data of the structural component to obtain the assembly inspection result of the rigid structural component; S5: The inspection results of the assembly status of flexible cables and rigid structural parts are transmitted to the human-computer interaction interface of the software system. The workpieces that have completed the inspection and are assembled accurately are marked in the process library, and the operator is guided to gradually complete the assembly status inspection of all workpieces of the product.

2. The aerospace product assembly detection method based on visual recognition according to claim 1, characterized in that: The S1 performs cable code tilt correction, text detection and recognition on the flexible cable image captured by the camera based on a machine vision text recognition algorithm to obtain the cable code content in the image, including: S1.1: Use the Hough transform to extract the straight line information in the cable image. Obtain the tilt angle of the extracted straight line by averaging. Then solve the rotation matrix M. Perform an affine transformation with the image center as the rotation center to complete the tilt correction of the cable code in the image. S1.2: Use a convolutional network to extract features from the tilt-corrected cable image. Use a recurrent neural network to detect text boxes of variable length. Divide the entire coding area in the image into small text candidate boxes for cable coding text detection. Finally, the detected small boxes are spliced ​​together to obtain the complete cable coding text box, which is the cable coding area. S1.3: The detected cable coding area is cropped and input into the CRNN network, and recognition is performed based on the image sequence to obtain the cable coding content.

3. The aerospace product assembly detection method based on visual recognition according to claim 2, characterized in that: The solution formula of the rotation matrix M is as follows: Where α=cos(angle), β=sin(angle), center.x and center.y are the center coordinates of the image.

4. The aerospace product assembly detection method based on visual recognition according to claim 1, characterized in that: In S2, the edit distance between the cable code content character string and all the part name character strings in the assembly process database is calculated; and the part name character string with the smallest edit distance value is selected as the matching result.

5. The aerospace product assembly detection method based on visual recognition according to claim 4, characterized in that: The formula for calculating the edit distance is: where lev a,b (i, j) is the edit distance between the first i characters of the cable code content string a and the first j characters of b; 1(a i ≠b j ) is the indicator function, a i ≠b j It is 1 when it is, otherwise it is 0; the three formulas in the min operation represent deleting, inserting and replacing characters in a to achieve b.

6. The aerospace product assembly detection method based on visual recognition according to claim 1, characterized in that: In S3, the pose estimation algorithm based on RGB data is used to estimate the pose of the rigid structural component in the camera coordinate system for the image of the rigid structural component. The least squares regression algorithm is used to perform linear fitting on the experimentally measured data to obtain the pose of the rigid structural component in the camera coordinate system at a real scale: f(x i )=wx i +b where f(x i ) is the converted coordinate data, that is, the position and orientation of the rigid structure of the real scale in the camera coordinate system; x i is the pose coordinate data of the rigid structure in the camera coordinate system output by the recognition algorithm before conversion, w is the conversion weight, and b is the recognition deviation.

7. The aerospace product assembly detection method based on visual recognition according to claim 6, characterized in that: w and b are obtained by minimizing the sum of the Euclidean distances from the experimental sample to the straight line. The formula for the sum of the Euclidean distances from the experimental sample to the straight line is as follows: Among them E (w,b) That is the sum of the Euclidean distances from the sample to the fitting line, y i is the real coordinate data of the structure, m is the number of experimental data samples; E (w,b) Calculate the partial derivatives of w and b respectively, and make the partial derivatives equal to 0, and we get: The mean 8. The aerospace product assembly detection method based on visual recognition according to claim 1, characterized in that: The formula for posture conversion of S4 is as follows: T1=T1 2 T2 Where T1 is the pose matrix of the rigid structure in the world coordinate system after the transformation is completed; T2 is the pose matrix of the rigid structure in the camera coordinate system before the transformation; T1 2 The transformation matrix from the camera coordinate system to the world coordinate system.

9. The aerospace product assembly detection method based on visual recognition according to claim 1, characterized in that: In S4, if the deviation between the converted posture data and the actual posture data of the structural component does not exceed the error threshold, the assembly result of the rigid structural component is correct.

Images