Machine vision self-sensing clear imaging method for aviation curved plate structure

Abstract

The invention belongs to the field of aviation equipment testing, and particularly relates to a machine vision self-sensing clear imaging method for an aviation curved plate structure. The method comprises the following steps: assuming that a structure unit in the curved surface wallboard structure is a plane, acquiring position information of a teaching unit through a machine vision system, and determining a detection position posture of the machine vision system in the teaching unit through manual teaching; scanning the curved surface wallboard structure through a machine vision system, and when a new structure unit is detected, obtaining position information of the new structure unit; calculating the position change of the new structure unit and the teaching unit according to the position information of the teaching unit and the new structure unit; calculating the detection position posture of the new structure unit according to the detection position posture and the position change of the teaching unit; and driving a machine vision system to move according to the detection position attitude of the new structural unit, performing image acquisition on the new structural unit, and traversing all structural units in the curved-surface wallboard structure to obtain a curved-surface wallboard structure image.

Description

Technical Field

[0001] The present application belongs to the field of aviation equipment testing, and in particular relates to a machine vision self-perception clear imaging method for aviation curved plate structures. Background Art

[0002] The purpose of aircraft structural strength testing is to discover damage, expose structural design weaknesses, and support structural design improvements. Therefore, timely and reliable damage detection capabilities are the core requirement of aircraft structural strength testing.

[0003] Manual visual inspection is the main means of detecting structural damage. However, since minor damage is only visible under high load and pressurization conditions, and staff cannot enter the aircraft for inspection at this time for safety reasons, manual visual inspection has the problems of being time-consuming and prone to omissions.

[0004] Machine vision technology can solve the above difficulties well. It uses robots and imaging devices to detect aircraft structures at any time without being affected by the load state. It is the development direction of aircraft structure damage detection. However, due to the complex structure of the aircraft, diverse configurations, long duration of strength fatigue tests (about 10 years), and random vibration during the test, the application of machine vision technology has the following difficulties: 1) It is difficult to automatically identify damage under strong interference; 2) In the state of macro imaging, the relative displacement caused by structural vibration causes image blur; 3) In the case of a small field of view, the imaging target position is tens of thousands, and the workload of robot manual teaching is large. In addition, considering the accumulation of relative displacement during long-term operation, it is necessary to regularly detect the position of the robot and repeat the manual teaching work.

[0005] Therefore, it is desired to have a technical solution to overcome or at least alleviate at least one of the above-mentioned defects of the prior art. Summary of the invention

[0006] The purpose of this application is to provide a machine vision self-perception clear imaging method for aviation curved plate structures to solve at least one problem existing in the prior art.

[0007] The technical solution of this application is:

[0008] A machine vision self-perception clear imaging method for aviation curved plate structures, comprising:

[0009] Step 1: Assuming that the structural unit in the curved wall panel structure is a plane, a structural unit in the curved wall panel structure is selected as a teaching unit, the position information of the teaching unit is obtained through a machine vision system, and the detection position and posture of the machine vision system on the teaching unit is determined through manual teaching;

[0010] Step 2: Scan the curved wall panel structure through the machine vision system, and when a new structural unit is detected, obtain the position information of the new structural unit;

[0011] Step 3: Calculate the position change of the new structure unit and the teaching unit according to the position information of the teaching unit and the new structure unit;

[0012] Step 4: Calculate the detection position and posture of the new structure unit according to the detection position and posture of the teaching unit and the position change;

[0013] Step 5: Taking the detected position posture of the new structural unit as the target posture, drive the machine vision system to move, collect images of the new structural unit, and loop steps 2 to 5 to traverse all structural units in the curved wall panel structure to obtain a curved wall panel structure image.

[0014] In at least one embodiment of the present application, the curved wall panel structure comprises:

[0015] Surface skinning;

[0016] A plurality of long stringers are arranged in parallel, and the long stringers are connected to the curved surface skin through rivets;

[0017] Bulkheads, a plurality of which are arranged in parallel, and the bulkheads are arranged crosswise with the long stringers, and the bulkheads are connected to the curved skin through rivets;

[0018] The curved wall panel structure is divided into a plurality of structural units by the long stringers and the bulkheads.

[0019] In at least one embodiment of the present application, the machine vision system includes:

[0020] A column, two of which are arranged in parallel;

[0021] An X-axis sliding shaft, the X-axis sliding shaft is installed between the two columns;

[0022] A Z-direction sliding shaft, the Z-direction sliding shaft being slidably mounted on the X-direction sliding shaft;

[0023] A three-axis mechanical arm, the three-axis mechanical arm is slidably mounted on the Z-direction sliding axis;

[0024] An industrial camera, wherein the industrial camera is mounted on the three-axis robotic arm;

[0025] A distance sensor, wherein the distance sensor is mounted on the industrial camera;

[0026] The controller is used to collect the signal of the distance sensor and realize the control of the Z-axis sliding axis, the three-axis mechanical arm and the industrial camera.

[0027] In at least one embodiment of the present application, when in use, the XZ plane of the machine vision system faces the curved wall panel structure.

[0028] In at least one embodiment of the present application, in step 2, scanning the curved wall panel structure by the machine vision system further includes:

[0029] Determine the scanning range and scanning trajectory;

[0030] The machine vision system scans the curved wall panel structure according to the scanning range and the scanning trajectory.

[0031] In at least one embodiment of the present application, the scanning trajectory is:

[0032] For each longitudinal position, all transverse detection points are traversed; then the scanning is started by moving down longitudinally and traversing the transverse detection points for the next longitudinal position.

[0033] In at least one embodiment of the present application, in step three, calculating the position change of the new structure unit and the teaching unit according to the position information of the teaching unit and the new structure unit includes:

[0034] The positions of the new structural unit and the teaching unit change as follows:

[0035] Δ=((x1,y1,z1),(θx1,θy1,θz1))-((x0,y0,z0),(θx,θy,θz))

[0036] Among them, Δ is the position change, (x0, y0, z0) is the coordinate of the center point of the teaching unit in the space coordinate system of the machine vision system, (θx, θy, θz) is the rotation angle of the teaching unit relative to the X, Y, and Z axes of the space coordinate system of the machine vision system, (x1, y1, z1) is the coordinate of the center point of the new structure unit in the space coordinate system of the machine vision system, and (θx1, θy1, θz1) is the rotation angle of the new structure unit relative to the X, Y, and Z axes of the space coordinate system of the machine vision system.

[0037] In at least one embodiment of the present application, in step 4, calculating the detected position and posture of the new structure unit according to the detected position and posture of the teaching unit and the position change includes:

[0038] The N detected positions and postures of the new structural unit are:

[0039] f'i=fi+Δ

[0040] i=1、2、3、…、N

[0041] Among them, f'i is the i-th detected position and posture of the new structure unit, fi is the i-th detected position and posture of the teaching unit, and Δ is the position change.

[0042] The invention has at least the following beneficial technical effects:

[0043] The machine vision self-perception clear imaging method for aviation curved plate structures of the present application can realize automatic posture conversion and clear imaging of machine vision, and solves the problems of huge workload of manual teaching and the need to regularly update the detection position. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a schematic diagram of a curved wall panel structure according to an embodiment of the present application;

[0045] Figure 2 is a schematic diagram of a machine vision system according to one embodiment of the present application;

[0046] Figure 3 is a schematic diagram of the relative position relationship between a machine vision system and a curved wall panel structure according to one embodiment of the present application;

[0047] Figure 4 This is a flowchart of automatic posture correction of a machine vision system according to one embodiment of the present application.

[0048] in:

[0049] 1-curved skin; 2-long stringer; 3-partition frame; 4-rivet; 5-structural unit; 6-column; 7-X-axis sliding axis; 8-Z-axis sliding axis; 9-three-axis robotic arm; 10-industrial camera; 11-distance sensor. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical scheme and advantages of the implementation of this application clearer, the technical scheme in the embodiment of this application will be described in more detail below in conjunction with the drawings in the embodiment of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of this application, not all of them. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain this application, and should not be construed as limitations on this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The embodiments of this application are described in detail below in conjunction with the drawings.

[0051] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the scope of protection of the present application.

[0052] The following is combined with Figures 1 to 4 This application is described in further detail.

[0053] The present application provides a machine vision self-perception clear imaging method for an aviation curved plate structure, comprising the following steps:

[0054] Step 1: Assuming that the structural unit in the curved wall panel structure is a plane, a structural unit in the curved wall panel structure is selected as a teaching unit, the position information of the teaching unit is obtained through the machine vision system, and the detection position and posture of the machine vision system in the teaching unit is determined through manual teaching;

[0055] Step 2: Scan the curved wall panel structure through a machine vision system, and when a new structural unit is detected, obtain the position information of the new structural unit;

[0056] Step 3: Calculate the position change of the new structure unit and the teaching unit according to the position information of the teaching unit and the new structure unit;

[0057] Step 4: Calculate the detection position and posture of the new structure unit according to the detection position and posture and position change of the teaching unit;

[0058] Step 5: Use the detected position and posture of the new structural unit as the target posture, drive the machine vision system to move, collect images of the new structural unit, and loop steps 2 to 5 to traverse all structural units in the curved wall panel structure to obtain the curved wall panel structure image.

[0059] The curved panel structure is a typical component of an aircraft and is also a typical detection object of a machine vision system. In one embodiment of the present application, the curved panel structure is as follows: Figure 1As shown, it includes a curved skin 1, a long stringer 2, a bulkhead 3, a rivet 4, and a structural unit 5. A plurality of long stringers 2 are arranged in parallel, and the long stringers 2 are connected to the curved skin 1 through rivets 4; a plurality of bulkheads 3 are arranged in parallel, and the bulkhead 3 is arranged crosswise with the long stringers 2, and the bulkhead 3 is connected to the curved skin 1 through rivets 4; the curved wall panel structure is divided into a plurality of structural units 5 by the long stringers 2 and the bulkhead 3. Among them, each long stringer 2 and each bulkhead 3 have the same configuration and are arranged in parallel, and the structural unit 5 is formed by the cross of the long stringer 2 and the bulkhead 3, and each structural unit 5 has a similar configuration, and each structural unit 5 has 10 to 20 rivets 4 along the direction of the long stringer 2 and 5 to 10 rivets 4 along the direction of the bulkhead 3. In order to clearly shoot the tiny cracks, there are usually 3 rivets 4 along the direction of the long stringer 2 and 1 to 2 rivets 4 along the direction of the bulkhead 3 in the camera field of view.

[0060] In this embodiment, the machine vision system for the curved wall panel structure is as follows: Figure 2 As shown, it includes a column 6, an X-axis sliding shaft 7, a Z-axis sliding shaft 8, a three-axis robot arm 9, an industrial camera 10, a distance sensor 11 and a controller. In the machine vision system, two columns 6 are arranged in parallel, the X-axis sliding shaft 7 is installed between the two columns 6, the Z-axis sliding shaft 8 is slidably installed on the X-axis sliding shaft 7, the three-axis robot arm 9 is slidably installed on the Z-axis sliding shaft 8, the industrial camera 10 is installed on the three-axis robot arm 9, and the distance sensor 11 is installed on the industrial camera 10. The controller is used to collect the signal of the distance sensor 11 and realize the control of the Z-axis sliding shaft 8, the three-axis robot arm 9 and the industrial camera 10. In this embodiment, the relative position relationship between the curved wall panel structure and the machine vision system is shown as follows: Figure 3 As shown, when in use, the XZ plane of the machine vision system faces the curved wall panel structure.

[0061] Since the wall panel is a curved surface, it is impossible to capture each structural detail (such as whether there are tiny cracks next to the rivet hole) by simply relying on linear motion along the X, Y, and Z directions. The commonly used method is to manually teach each field of view, which is not only labor-intensive but also requires re-checking the positioning after running for a period of time, which is inefficient. Combining target recognition technology to achieve automatic posture correction of the machine vision system and achieve clear imaging is an effective way to solve this problem. However, for three-dimensional positioning, a binocular camera is usually required, and the size, flexibility, and calculation speed of the binocular camera system cannot meet the requirements of strength testing.

[0062] To this end, the machine vision self-perception clear imaging method for aviation curved plate structures in this application is based on two-dimensional target recognition and depth information scanning to achieve the purpose of automatic clear imaging by the machine vision system. The overall plan is:

[0063] First, in step 1, although the curved wall panel structure is a curved surface, the curvature is small and the structural unit occupies a small proportion of the entire wall panel. Therefore, it is assumed that the interior of the structural unit is a plane and the influence of the curved surface is not considered when calculating the rotation angle of the structural unit.

[0064] Based on the above assumptions, a structural unit is selected as the teaching unit. Then, through manual teaching, the N detection points of the machine vision system on the teaching unit and the corresponding detection position and posture are determined. The position information of the teaching unit is recorded as ((x0, y0, z0), (θx, θy, θz)), and its vertices are recorded as A j , the depths of the vertices from the laser sensor are h j , where (x0, y0, z0) are the coordinates of the center point of the teaching unit in the spatial coordinate system of the machine vision system, and (θx, θy, θz) are the rotation angles of the teaching unit relative to the X, Y, and Z axes of the spatial coordinate system of the machine vision system; the N detection positions and postures of the machine vision system are denoted as f1, f2, f3,...fn.

[0065] In step 2, after the teaching is completed, the curved wall panel structure is scanned by the machine vision system, and the camera field of view is larger than one structural unit and smaller than two structural units. During the detection process, the controller detects the image in real time. If a complete structural unit is detected, the scanning is paused, and the position information of the new structural unit is obtained by combining the distance sensor scanning, which is recorded as ((x1, y1, z1), (θx1, θy1, θz1)), and its vertices are recorded as B j , the depths of the vertices from the laser sensor are l j , where (x01, y01, z01) are the coordinates of the center point of the new structure unit in the space coordinate system of the machine vision system, and (θx1, θy1, θz1) are the rotation angles of the new structure unit relative to the X, Y, and Z axes of the space coordinate system of the machine vision system.

[0066] In the preferred embodiment of the present application, the machine vision system scans the curved wall panel structure according to a preset scanning range and scanning trajectory. The scanning trajectory is: for each longitudinal position, traverse all transverse detection points; then move down along the longitudinal direction and start traversal scanning of transverse detection points for the next longitudinal position.

[0067] In step 3, according to the position information of the teaching unit and the new structure unit, the position change of the new structure unit and the teaching unit is calculated, including:

[0068] The positions of the new structural unit and the teaching unit change as follows:

[0069] Δ=((x1,y1,z1),(θx1,θy1,θz1))-((x0,y0,z0),(θx,θy,θz))

[0070] Among them, Δ is the position change, (x0, y0, z0) is the coordinate of the center point of the teaching unit in the space coordinate system of the machine vision system, (θx, θy, θz) is the rotation angle of the teaching unit relative to the X, Y, and Z axes of the space coordinate system of the machine vision system, (x1, y1, z1) is the coordinate of the center point of the new structure unit in the space coordinate system of the machine vision system, and (θx1, θy1, θz1) is the rotation angle of the new structure unit relative to the X, Y, and Z axes of the space coordinate system of the machine vision system.

[0071] In step 4, the detection position and posture of the new structure unit are calculated according to the detection position and posture and position change of the teaching unit, including:

[0072] The N detected positions and postures of the new structural unit are:

[0073] f'i=fi+Δ

[0074] i=1、2、3、…、N

[0075] Among them, f'i is the i-th detected position and posture of the new structure unit, fi is the i-th detected position and posture of the teaching unit, and Δ is the position change.

[0076] Finally, in step five, with f'i as the target posture, the machine vision system is driven to move to achieve image acquisition, and finally a complete curved wall panel structure image is obtained.

[0077] The machine vision self-perception clear imaging method for aviation curved plate structure of the present application provides a machine vision system posture correction method based on two-dimensional target recognition and depth information scanning, wherein, Figure 4 As shown in the figure, the automatic posture correction process of the machine vision system includes four parts: pre-process, system trajectory planning, target recognition based on video stream, and automatic correction of shooting posture.

[0078] In the pre-process stage, the two endpoints of the diagonal line of the wall panel to be inspected are set in the machine vision system to clarify the inspection range; manual teaching is performed on the teaching unit to manually determine the shooting posture of the machine vision system for each vertex and rivet in the structural unit; and the horizontal and vertical dimensions of the inspection object are set;

[0079] The rough planning of the detection trajectory is mainly to determine the approximate scanning path of the machine vision system. For example, for each longitudinal position, all the horizontal detection points are traversed; then, the horizontal detection points for the next longitudinal position are traversed and scanned downward along the longitudinal direction;

[0080] Based on the target recognition of the video stream, the machine vision system moves along the rough trajectory and detects the content of the picture in real time. When the system recognizes that a complete structural unit appears in the picture, the rough trajectory movement is paused, the structural unit is automatically numbered, and the distance sensor is enabled to measure the depth information of the four vertices of the structural unit, and the number, size and vertex depth information are sent to the machine vision system controller.

[0081] The shooting posture is automatically corrected to obtain clear and detailed images. The posture change Δ of the new structural unit relative to the teaching structural unit and the new target posture of the machine vision system are calculated by measuring the size, vertex depth, coordinates, etc. of the new structural unit; then the machine vision system is driven to move with the new target posture to achieve the purpose of clear imaging. After completing the clear image acquisition of a new structural unit, the machine vision system continues to scan along the rough detection path.

[0082] The working principle of the machine vision self-perception clear imaging method for aviation curved plate structure in this application includes the following points:

[0083] a) If the relative position between the end of the machine vision system and the structural unit to be detected remains unchanged, it can be considered that the system imaging effect is consistent with the teaching effect.

[0084] b) During the scanning process of the machine vision system, the system needs to detect the image in real time to find the structural units in time. It is preferred to use YOLO and other fast target detection algorithms based on deep learning to detect structural units.

[0085] c) Calculation method of target angle and position of structural unit:

[0086] 1) The teaching unit of this application is the reference unit, and the new structural unit is obtained in real time, that is, the posture and position conversion matrix between the two units is obtained. Taking the measured object as an example, it has only a single direction with posture selection, so the posture conversion matrix is ​​simplified to a rotation matrix around the Y axis.

[0087] Its attitude transformation matrix is ​​expressed as:

[0088]

[0089] Its position transformation matrix is ​​expressed as:

[0090] A T B =[X B1 -X A1 Y B1 -Y A1 Z B1 -Z A1 ]

[0091] Among them, b is the rotation angle, which is the only unknown quantity. Next, the rotation angle b will be obtained through the known variables in the benchmark unit and the new structure unit.

[0092] 2) Calculate the normal vectors of the base unit and the new structural unit:

[0093]

[0094] Therefore, by We can solve for b, where || is a unit vector.

[0095] 3) According to A R B (Y), A T B The spatial coordinates after posture transformation can be obtained.

[0096] The machine vision self-perception clear imaging method for aviation curved plate structures in this application focuses on engineering constraints such as the detection space limitation of the aircraft structure and the algorithm operation efficiency. It can realize automatic posture conversion and clear imaging of machine vision, improve the engineering applicability and ease of use of machine vision technology in aircraft strength testing, and solve the problems of huge workload of manual teaching and the need to regularly update the detection position.

[0097] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A machine vision self-perception clear imaging method for aviation curved plate structures, characterized in that: include: Step 1: Assuming that the structural unit in the curved wall panel structure is a plane, a structural unit in the curved wall panel structure is selected as a teaching unit, the position information of the teaching unit is obtained through a machine vision system, and the detection position and posture of the machine vision system on the teaching unit is determined through manual teaching; Step 2: Scan the curved wall panel structure through the machine vision system, and when a new structural unit is detected, obtain the position information of the new structural unit; Step 3: Calculate the position change of the new structure unit and the teaching unit according to the position information of the teaching unit and the new structure unit; Step 4: Calculate the detection position and posture of the new structure unit according to the detection position and posture of the teaching unit and the position change; Step 5: Taking the detected position posture of the new structural unit as the target posture, drive the machine vision system to move, collect images of the new structural unit, loop steps 2 to 5, traverse all structural units in the curved wall panel structure, and obtain the curved wall panel structure image.

2. The machine vision self-perception clear imaging method for aviation curved plate structure according to claim 1, characterized in that: The curved wall panel structure comprises: Curved skin (1); A plurality of long stringers (2) are arranged in parallel, and the long stringers (2) are connected to the curved surface skin (1) via rivets (4); A plurality of bulkheads (3) are arranged in parallel, and the bulkheads (3) and the long stringers (2) are arranged crosswise, and the bulkheads (3) are connected to the curved skin (1) via rivets (4); The curved wall panel structure is divided into a plurality of structural units (5) by means of the long stringer (2) and the bulkhead (3).

3. The machine vision self-perception clear imaging method for aviation curved plate structure according to claim 2, characterized in that: The machine vision system comprises: A column (6), two of the columns (6) are arranged in parallel; An X-direction sliding shaft (7), wherein the X-direction sliding shaft (7) is installed between the two upright posts (6); A Z-direction sliding shaft (8), wherein the Z-direction sliding shaft (8) is slidably mounted on the X-direction sliding shaft (7); A three-axis mechanical arm (9), wherein the three-axis mechanical arm (9) is slidably mounted on the Z-direction sliding shaft (8); An industrial camera (10), wherein the industrial camera (10) is mounted on the three-axis mechanical arm (9); A distance sensor (11), wherein the distance sensor (11) is mounted on the industrial camera (10); A controller is used to collect signals from the distance sensor (11) and to control the Z-axis sliding axis (8), the three-axis mechanical arm (9) and the industrial camera (10).

4. The machine vision self-perception clear imaging method for aviation curved plate structure according to claim 3, characterized in that: When in use, the XZ plane of the machine vision system faces the curved wall panel structure.

5. The machine vision self-perception clear imaging method for aviation curved plate structure according to claim 4, characterized in that: In step 2, scanning the curved wall panel structure by the machine vision system also includes: Determine the scanning range and scanning trajectory; The machine vision system scans the curved wall panel structure according to the scanning range and the scanning trajectory.

6. The machine vision self-perception clear imaging method for aviation curved plate structure according to claim 5, characterized in that: The scanning trajectory is: For each longitudinal position, all transverse detection points are traversed; then the scanning is started by moving down longitudinally and traversing the transverse detection points for the next longitudinal position.

7. The machine vision self-perception clear imaging method for aviation curved plate structure according to claim 6, characterized in that: In step three, according to the position information of the teaching unit and the new structure unit, the position change of the new structure unit and the teaching unit is calculated, including: The positions of the new structural unit and the teaching unit change as follows: Δ=((x1,y1,z1),(θx1,θy1,θz1))-((x0,y0,z0),(θx,θy,θz)) Among them, Δ is the position change, (x0, y0, z0) is the coordinate of the center point of the teaching unit in the space coordinate system of the machine vision system, (θx, θy, θz) is the rotation angle of the teaching unit relative to the X, Y, and Z axes of the space coordinate system of the machine vision system, (x1, y1, z1) is the coordinate of the center point of the new structure unit in the space coordinate system of the machine vision system, and (θx1, θy1, θz1) is the rotation angle of the new structure unit relative to the X, Y, and Z axes of the space coordinate system of the machine vision system.

8. The machine vision self-perception clear imaging method for aviation curved plate structure according to claim 7, characterized in that: In step 4, the detection position and posture of the new structure unit are calculated according to the detection position and posture of the teaching unit and the position change, including: The N detected positions and postures of the new structural unit are: f'i=fi+Δ i=1、2、3、…、N Among them, f'i is the i-th detected position and posture of the new structure unit, fi is the i-th detected position and posture of the teaching unit, and Δ is the position change.

Images