A machine vision self-perception clear imaging method for an aero curved plate structure

Abstract

The application belongs to the field of aviation equipment testing, and particularly relates to a machine vision self-perception clear imaging method for an aviation curved plate structure. The method comprises the following steps: assuming that a structural unit in the curved plate structure is a plane, acquiring position information of a teaching unit by a machine vision system, and determining a detection position and posture of the machine vision system at the teaching unit by manual teaching; scanning the curved plate structure by the machine vision system, and acquiring position information of a new structural unit when a new structural unit is detected; calculating a position change of the new structural unit relative to the teaching unit according to the position information of the teaching unit and the new structural unit; calculating a detection position and posture of the new structural unit according to the detection position and posture of the teaching unit and the position change; driving the machine vision system to move by the detection position and posture of the new structural unit, collecting an image of the new structural unit, and traversing all the structural units in the curved plate structure to obtain a curved plate structure image.

Description

Technical Field

[0001] This application belongs to the field of aviation equipment testing, and specifically relates to a machine vision self-sensing clear imaging method for aviation curved plate structures. Background Technology

[0002] The purpose of aircraft structural strength testing is to discover damage, expose weaknesses in the structural design, and support structural design improvements. Therefore, timely and reliable damage detection capability is the core requirement of aircraft structural strength testing.

[0003] Manual visual inspection is the primary means of detecting structural damage. However, since minor damage is only visible under high load and pressurization conditions, and personnel cannot enter the aircraft for inspection under these conditions for safety reasons, manual visual inspection suffers from problems such as being time-consuming and prone to missing defects.

[0004] Machine vision technology can effectively address the above difficulties. Utilizing robots and imaging devices, it can inspect aircraft structures at any time, unaffected by load conditions, and represents the future direction of aircraft structural damage detection. However, due to the complexity of aircraft structures, diverse configurations, long duration of strength fatigue tests (around 10 years), and the presence of random vibrations during testing, the application of machine vision technology faces the following challenges: 1) Difficulty in automatically identifying damage under strong interference; 2) Image blurring caused by relative displacement due to structural vibrations in macro imaging; 3) In small field-of-view situations, the number of target positions for imaging is in the tens of thousands, resulting in a large workload for manual robot teaching. Furthermore, considering the accumulation of relative displacements during long-term operation, periodic robot position checks and repeated manual teaching are required.

[0005] Therefore, it is desirable to have a technical solution to overcome or at least mitigate one of the aforementioned defects of the prior art. Summary of the Invention

[0006] The purpose of this application is to provide a machine vision self-perceived clear imaging method for aerospace curved plate structures to solve at least one problem existing in the prior art.

[0007] The technical solution of this application is:

[0008] A machine vision self-perceived clear imaging method for aerospace curved plate structures includes:

[0009] Step 1: Assuming that the structural unit in the curved wall panel structure is a plane, select one structural unit in the curved wall panel structure as a teaching unit, obtain the position information of the teaching unit through the machine vision system, and determine the detection position and posture of the machine vision system in the teaching unit through manual teaching.

[0010] Step 2: Scan the curved wall panel structure using the machine vision system. When a new structural unit is detected, obtain the position information of the new structural unit.

[0011] Step 3: Calculate the positional changes of the new structural unit and the teaching unit based on the positional information of the teaching unit and the new structural unit;

[0012] Step 4: Calculate the detection position and orientation of the new structural unit based on the detection position and orientation of the teaching unit and the position change;

[0013] Step 5: Using the detected position and pose of the new structural unit as the target pose, drive the machine vision system to move and acquire images of the new structural unit. Repeat steps 2 to 5 to traverse all structural units in the curved wall panel structure to obtain an image of the curved wall panel structure.

[0014] In at least one embodiment of this application, the curved wall panel structure includes:

[0015] Curved surface skin;

[0016] Multiple stringers are arranged in parallel, and the stringers are connected to the curved skin by rivets;

[0017] Multiple partition frames are arranged in parallel, and the partition frames are arranged to intersect with the stringers. The partition frames are connected to the curved skin by rivets.

[0018] The curved wall panel structure is divided into multiple structural units by the stringers and the partitions.

[0019] In at least one embodiment of this application, the machine vision system includes:

[0020] The columns are arranged in parallel, two in total;

[0021] An X-axis sliding shaft is installed between the two columns;

[0022] A Z-axis sliding shaft is slidably mounted on the X-axis sliding shaft;

[0023] A three-axis robotic arm, wherein the three-axis robotic arm is slidably mounted on the Z-axis sliding axis;

[0024] An industrial camera is mounted on the three-axis robotic arm;

[0025] A distance sensor, which is mounted on the industrial camera;

[0026] The controller is used to acquire signals from the distance sensor and to control the Z-axis sliding axis, the three-axis robotic arm, and the industrial camera.

[0027] In at least one embodiment of this application, when in use, the XZ plane of the machine vision system faces the curved wall panel structure.

[0028] In at least one embodiment of this application, step two, which involves scanning the curved wall panel structure using the machine vision system, further includes:

[0029] Determine the scanning area and scanning trajectory;

[0030] The machine vision system scans the curved wall panel structure according to the scanning range and the scanning trajectory.

[0031] In at least one embodiment of this application, the scanning trajectory is:

[0032] For each vertical position, iterate through all horizontal detection points; then move down the vertical direction and begin scanning the horizontal detection points for the next vertical position.

[0033] In at least one embodiment of this application, step three, calculating the positional change between the new structural unit and the teaching unit based on the positional information of the teaching unit and the new structural unit, includes:

[0034] The positions of the new structural unit and the teaching unit change as follows:

[0035] Δ=((x1,y1,z1),(θx1,θy1,θz1))-((x0,y0,z0),(θx,θy,θz))

[0036] Where Δ represents the position change, (x0, y0, z0) are the coordinates of the center point of the teaching unit in the machine vision system's spatial coordinate system, (θx, θy, θz) are the rotation angles of the teaching unit relative to the X, Y, and Z axes of the machine vision system's spatial coordinate system, (x1, y1, z1) are the coordinates of the center point of the new structural unit in the machine vision system's spatial coordinate system, and (θx1, θy1, θz1) are the rotation angles of the new structural unit relative to the X, Y, and Z axes of the machine vision system's spatial coordinate system.

[0037] In at least one embodiment of this application, step four, calculating the detection position and pose of the new structural unit based on the detection position and pose of the teaching unit and the position change, includes:

[0038] The poses of the N detection positions of the new structural unit are:

[0039] f'i=fi+Δ

[0040] i = 1, 2, 3, ..., N

[0041] Where f'i is the i-th detected position and pose of the new structural unit, fi is the i-th detected position and pose of the teaching unit, and Δ is the position change.

[0042] The invention has at least the following beneficial technical effects:

[0043] The machine vision self-perception clear imaging method for aerospace curved plate structures proposed in this application can realize automatic pose conversion and clear imaging of machine vision, and solves the problems of huge workload of manual teaching and the need to update the detection position regularly. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of a curved wall panel structure according to one embodiment of this application;

[0045] Figure 2 This is a schematic diagram of a machine vision system according to one embodiment of this application;

[0046] Figure 3 This is a schematic diagram showing the relative positional relationship between a machine vision system and a curved wall panel structure according to one embodiment of this application;

[0047] Figure 4 This is a flowchart of an automatic pose correction process for a machine vision system according to one embodiment of this application.

[0048] in:

[0049] 1-Curved skin; 2-Stringer; 3-Frame; 4-Rivet; 5-Structural unit; 6-Column; 7-X-axis sliding axis; 8-Z-axis sliding axis; 9-Three-axis robotic arm; 10-Industrial camera; 11-Distance sensor. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0051] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting the scope of protection of this application.

[0052] The following is in conjunction with the appendix Figures 1 to 4 This application will be described in further detail.

[0053] This application provides a machine vision self-sensing clear imaging method for aerospace curved plate structures, including the following steps:

[0054] Step 1: Assuming the structural units in the curved panel structure are planar, select one structural unit in the curved panel structure as the teaching unit, obtain the position information of the teaching unit through the machine vision system, and determine the detection position and posture of the machine vision system in the teaching unit through manual teaching.

[0055] Step 2: Scan the curved wall panel structure using a machine vision system. When a new structural unit is detected, obtain the position information of the new structural unit.

[0056] Step 3: Based on the positional information of the teaching unit and the new structural unit, calculate the positional changes between the new structural unit and the teaching unit;

[0057] Step 4: Calculate the detection position and attitude of the new structural unit based on the detection position and attitude of the teaching unit and its position changes;

[0058] Step 5: Using the detected position and pose of the new structural unit as the target pose, drive the machine vision system to move and acquire images of the new structural unit. Repeat steps 2 to 5 to traverse all structural units in the curved wall panel structure and obtain the image of the curved wall panel structure.

[0059] Curved panel structures are typical components of aircraft and also typical inspection objects of machine vision systems. In one embodiment of this application, the curved panel structure is as follows: Figure 1As shown, the structure includes a curved skin 1, stringers 2, partitions 3, rivets 4, and structural units 5. Multiple stringers 2 are arranged in parallel and connected to the curved skin 1 via rivets 4. Multiple partitions 3 are also arranged in parallel, intersecting with the stringers 2, and connected to the curved skin 1 via rivets 4. The stringers 2 and partitions 3 divide the curved panel structure into multiple structural units 5. Each stringer 2 and each partition 3 has the same configuration and is arranged in parallel. Each structural unit 5 is formed by the intersection of stringers 2 and partitions 3, and each structural unit 5 has a similar configuration. Each structural unit 5 contains 10 to 20 rivets 4 along the direction of the stringers 2 and 5 to 10 rivets 4 along the direction of the partitions 3. To clearly photograph minute cracks, typically there are 3 rivets 4 along the direction of the stringers 2 and 1 to 2 rivets 4 along the direction of the partitions 3 within the camera's field of view.

[0060] In this embodiment, the machine vision system for curved wall panel structures is as follows: Figure 2 As shown, the system includes columns 6, an X-axis sliding axis 7, a Z-axis sliding axis 8, a three-axis robotic arm 9, an industrial camera 10, a distance sensor 11, and a controller. In the machine vision system, two columns 6 are arranged in parallel. The X-axis sliding axis 7 is installed between the two columns 6. The Z-axis sliding axis 8 is slidably installed on the X-axis sliding axis 7. The three-axis robotic arm 9 is slidably installed on the Z-axis sliding axis 8. The industrial camera 10 is installed on the three-axis robotic arm 9. The distance sensor 11 is installed on the industrial camera 10. The controller is used to collect signals from the distance sensor 11 and to control the Z-axis sliding axis 8, the three-axis robotic arm 9, and the industrial camera 10. In this embodiment, the relative positional relationship between the curved wall panel structure and the machine vision system is as follows: Figure 3 As shown, during use, the XZ plane of the machine vision system faces the curved wall panel structure.

[0061] Because the wall panel is curved, achieving clear imaging of every structural detail (such as whether there are tiny cracks next to rivet holes) cannot be accomplished simply by linear movement along the X, Y, and Z directions. A common method is manual teaching for each field of view, which is not only labor-intensive but also requires re-inspection and positioning after a period of operation, resulting in low efficiency. Combining target recognition technology to achieve automatic pose correction and clear imaging in a machine vision system is an effective way to solve this problem. However, for 3D positioning, a binocular camera is usually required, but the size, flexibility, and computing speed of a binocular camera system cannot meet the requirements of strength testing.

[0062] Therefore, this application presents a machine vision self-perceived clear imaging method for aerospace curved plate structures, based on two-dimensional target recognition and depth information scanning, to achieve automatic clear imaging by the machine vision system. The overall scheme is as follows:

[0063] First, in step one, although the curved wall panel structure is curved, the curvature is small, and the structural unit occupies a small proportion of the entire wall panel. Therefore, it is assumed that the interior of the structural unit is a plane, and the influence of the curved surface is not considered when calculating the rotation angle of the structural unit.

[0064] Based on the above assumptions, a structural unit is selected as the teaching unit. Then, through manual teaching, the N detection points and corresponding detection positions and poses of the machine vision system on the teaching unit are determined. The position information of the teaching unit is denoted as ((x0,y0,z0),(θx,θy,θz)), and its vertices are denoted as A. j The depths at the apex from the laser sensor are h and h, respectively. j Where (x0, y0, z0) are the coordinates of the center point of the teaching unit in the spatial coordinate system of the machine vision system, and (θx, θy, θz) are the rotation angles of the teaching unit relative to the X, Y, and Z axes of the spatial coordinate system of the machine vision system; the N detection positions and attitudes of the machine vision system are denoted as f1, f2, f3, ..., fn.

[0065] In step two, after the teaching is completed, the curved wall panel structure is scanned by a machine vision system. The camera's field of view is larger than one structural unit and smaller than two structural units. During the detection process, the controller performs real-time detection of the image. If a complete structural unit is detected, the scanning is paused, and the position information of the new structural unit is obtained by combining the distance sensor scan, denoted as ((x1,y1,z1),(θx1,θy1,θz1)), and its vertices are denoted as B. j The depths at the apex from the laser sensor are l j Where (x01, y01, z01) are the coordinates of the center point of the new structural unit in the spatial coordinate system of the machine vision system, and (θx1, θy1, θz1) are the rotation angles of the new structural unit relative to the X, Y, and Z axes of the spatial coordinate system of the machine vision system.

[0066] In a preferred embodiment of this application, the machine vision system scans the curved wall panel structure according to a preset scanning range and scanning trajectory. The scanning trajectory is as follows: for each longitudinal position, all transverse detection points are traversed; then, the system moves downward along the longitudinal direction and begins traversing the transverse detection points for the next longitudinal position.

[0067] In step three, based on the positional information of the teaching unit and the new structural unit, the positional changes between the new structural unit and the teaching unit are calculated, including:

[0068] The positions of the new structural unit and the teaching unit change as follows:

[0069] Δ=((x1,y1,z1),(θx1,θy1,θz1))-((x0,y0,z0),(θx,θy,θz))

[0070] Where Δ represents the position change, (x0, y0, z0) are the coordinates of the center point of the teaching unit in the machine vision system's spatial coordinate system, (θx, θy, θz) are the rotation angles of the teaching unit relative to the X, Y, and Z axes of the machine vision system's spatial coordinate system, (x1, y1, z1) are the coordinates of the center point of the new structural unit in the machine vision system's spatial coordinate system, and (θx1, θy1, θz1) are the rotation angles of the new structural unit relative to the X, Y, and Z axes of the machine vision system's spatial coordinate system.

[0071] In step four, based on the detected position and orientation of the teaching unit and its position changes, the detected position and orientation of the new structural unit are calculated, including:

[0072] The poses of the N detection positions of the new structural unit are:

[0073] f'i=fi+Δ

[0074] i = 1, 2, 3, ..., N

[0075] Where f'i is the i-th detected position and pose of the new structural unit, fi is the i-th detected position and pose of the teaching unit, and Δ is the position change.

[0076] Finally, in step five, with f'i as the target posture, the machine vision system is driven to move to achieve image acquisition, and finally a complete image of the curved panel structure is obtained.

[0077] This application presents a machine vision self-perceived clear imaging method for aerospace curved plate structures, which provides a pose correction method for a machine vision system based on two-dimensional target recognition and depth information scanning, wherein, as... Figure 4 As shown, the automatic pose correction process of the machine vision system consists of four parts: pre-process, system trajectory planning, target recognition based on video stream, and automatic pose correction during shooting.

[0078] In the pre-process stage, the two endpoints of the diagonal of the panel to be inspected are set in the machine vision system to clarify the inspection range; manual teaching is carried out for the teaching unit, and the machine vision system is manually determined to capture the pose of each vertex and rivet in the structural unit; the horizontal and vertical dimensions of the object to be inspected are set.

[0079] Coarse planning of the detection trajectory mainly involves determining the approximate scanning path of the machine vision system. For example, for each vertical position, all horizontal detection points are traversed; then, the system moves down along the vertical direction to start traversing and scanning the horizontal detection points for the next vertical position.

[0080] Based on target recognition in video streams, the machine vision system moves along a coarse trajectory while simultaneously detecting the content of the scene in real time. When the system identifies a complete structural unit in the scene, it pauses the coarse trajectory movement, automatically assigns a number to the structural unit, and simultaneously activates a distance sensor to measure the depth information of the four vertices of the structural unit. The number, size, and vertex depth information are then sent to the machine vision system controller.

[0081] Automatic pose correction is used to acquire clear, detailed images. The pose change Δ of the new structural unit relative to the taught structural unit is calculated by measuring its dimensions, vertex depth, and coordinates, resulting in a new target pose for the machine vision system. This new target pose then drives the machine vision system to move, achieving clear imaging. After acquiring a clear image of a new structural unit, the machine vision system continues scanning along the coarse detection path.

[0082] The machine vision self-perception clear imaging method for aerospace curved plate structures proposed in this application works by the following points:

[0083] a) If the relative pose between the end effector of the machine vision system and the structural unit to be detected remains unchanged, the imaging effect of the system can be considered to be consistent with the teaching effect.

[0084] b) During the scanning process of the machine vision system, the system needs to perform real-time detection of images to promptly identify structural units. It is preferable to use fast object detection algorithms based on deep learning, such as YOLO, to detect structural units.

[0085] c) Calculation method for target rotation angle and position of structural unit:

[0086] 1) The teaching unit in this application is the reference unit, and new structural units are obtained in real time, that is, the attitude and position transformation matrix between the two units needs to be obtained. Taking the test piece as an example, it only has attitude selection in a single direction, so the attitude transformation matrix is ​​simplified to a rotation matrix around the Y-axis.

[0087] Its attitude transformation matrix is ​​expressed as:

[0088]

[0089] Its position transformation matrix is ​​expressed as:

[0090] A T B =[X B1 -X A1 Y B1 -Y A1 Z B1 -Z A1 ]

[0091] Here, b is the rotation angle, which is the only unknown quantity. Next, the rotation angle b will be obtained through the known variables in the reference element and the new structural element.

[0092] 2) Calculate the normal vectors of the reference element and the new structural element:

[0093]

[0094] Therefore, by b can be solved, where || is a unit vector.

[0095] 3) According to A R B (Y) A T B The spatial coordinates after the attitude transformation can be obtained.

[0096] The machine vision self-perception clear imaging method for aircraft curved plate structures proposed in this application takes into account engineering constraints such as the detection space limitations of aircraft structures and the efficiency of algorithm operation. It can realize automatic pose conversion and clear imaging of machine vision, improve the engineering applicability and ease of use of machine vision technology in aircraft strength testing, and solve problems such as huge workload of manual teaching and the need for regular updates of detection positions.

[0097] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A machine vision self-perceived clear imaging method for aerospace curved plate structures, characterized in that, include: Step 1: Assuming that the structural unit in the curved wall panel structure is a plane, select one structural unit in the curved wall panel structure as a teaching unit, obtain the position information of the teaching unit through the machine vision system, and determine the detection position and posture of the machine vision system in the teaching unit through manual teaching. Step 2: Scan the curved wall panel structure using the machine vision system. When a new structural unit is detected, obtain the position information of the new structural unit. Step 3: Calculate the positional changes of the new structural unit and the teaching unit based on the positional information of the teaching unit and the new structural unit; Step 4: Calculate the detection position and orientation of the new structural unit based on the detection position and orientation of the teaching unit and the position change; Step 5: Using the detected position and pose of the new structural unit as the target pose, drive the machine vision system to move and acquire images of the new structural unit. Repeat steps 2 to 5 to traverse all structural units in the curved wall panel structure to obtain an image of the curved wall panel structure.

2. The machine vision self-perceived clear imaging method for aerospace curved plate structures according to claim 1, characterized in that, The curved wall panel structure includes: Curved surface skin (1); A long stringer (2) is arranged in parallel with multiple strings, and the long stringer (2) is connected to the curved skin (1) by rivets (4); Multiple partition frames (3) are arranged in parallel, and the partition frames (3) are arranged to cross the stringers (2). The partition frames (3) are connected to the curved skin (1) by rivets (4). The curved wall panel structure is divided into multiple structural units (5) by the stringers (2) and the partitions (3).

3. The machine vision self-perceived clear imaging method for aerospace curved plate structures according to claim 2, characterized in that, The machine vision system includes: Column (6), two columns (6) are arranged in parallel; X-axis sliding shaft (7), which is installed between the two columns (6); Z-axis sliding shaft (8), which is slidably mounted on X-axis sliding shaft (7); A three-axis robotic arm (9) is slidably mounted on the Z-axis sliding shaft (8); An industrial camera (10) is mounted on the three-axis robotic arm (9); Distance sensor (11), the distance sensor (11) is mounted on the industrial camera (10); The controller is used to acquire the signal from the distance sensor (11) and to control the Z-axis sliding axis (8), the three-axis robotic arm (9), and the industrial camera (10).

4. The machine vision self-perceived clear imaging method for aerospace curved plate structures according to claim 3, characterized in that, When in use, the XZ plane of the machine vision system faces the curved wall panel structure.

5. The machine vision self-perceived clear imaging method for aerospace curved plate structures according to claim 4, characterized in that, Step two, which involves scanning the curved wall panel structure using the machine vision system, also includes: Determine the scanning area and scanning trajectory; The machine vision system scans the curved wall panel structure according to the scanning range and the scanning trajectory.

6. The machine vision self-perceived clear imaging method for aerospace curved plate structures according to claim 5, characterized in that, The scanning trajectory is as follows: For each vertical position, iterate through all horizontal detection points; then move down the vertical direction and begin scanning the horizontal detection points for the next vertical position.

7. The machine vision self-perceived clear imaging method for aerospace curved plate structures according to claim 6, characterized in that, In step three, based on the position information of the teaching unit and the new structural unit, the positional changes of the new structural unit and the teaching unit are calculated, including: The positions of the new structural unit and the teaching unit change as follows: Δ=((x1,y1,z1),(θx1,θy1,θz1))-((x0,y0,z0),(θx,θy,θz)) Where Δ represents the position change, (x0, y0, z0) are the coordinates of the center point of the teaching unit in the machine vision system's spatial coordinate system, (θx, θy, θz) are the rotation angles of the teaching unit relative to the X, Y, and Z axes of the machine vision system's spatial coordinate system, (x1, y1, z1) are the coordinates of the center point of the new structural unit in the machine vision system's spatial coordinate system, and (θx1, θy1, θz1) are the rotation angles of the new structural unit relative to the X, Y, and Z axes of the machine vision system's spatial coordinate system.

8. The machine vision self-perceived clear imaging method for aerospace curved plate structures according to claim 7, characterized in that, In step four, the detection position and orientation of the new structural unit are calculated based on the detection position and orientation of the teaching unit and the position change, including: The poses of the N detection positions of the new structural unit are: f'i=fi+Δ i = 1, 2, 3, ..., N Where f'i is the i-th detected position and pose of the new structural unit, fi is the i-th detected position and pose of the teaching unit, and Δ is the position change.

Images