An airport pavement surface image detection system based on linear array scanning imaging

Abstract

The present invention provides an airport pavement surface image detection system based on a linear array camera, comprising: an imaging control device fixed on a moving chassis, a visual imaging device, an industrial computer and a positioning device; the moving chassis travels on the airport pavement according to a preset driving path; the imaging control device is fixed at the front wheel of the moving chassis to record the rotation angle of the front wheel; the visual imaging device performs linear array scanning imaging on the airport pavement based on the received trigger signal, obtains the corresponding airport pavement surface image information and sends it to the industrial computer; the positioning device is used to obtain the position information of the moving chassis and send it to the industrial computer; the industrial computer detects abnormal targets in the received pavement surface image, and when the abnormal target is detected, determines the area and position of the abnormal target based on the position information obtained by the positioning device and generates the corresponding detection result. The present invention can reduce the workload of the staff, and has the advantages of high precision, high efficiency, good timeliness, etc.

Description

Technical Field

[0001] The present application relates to the field of airport pavement surface image detection, and in particular to an airport pavement image detection system based on a linear array camera. Background Art

[0002] The airport pavement is an important facility of the airport. The abnormal information contained in the airport pavement surface image includes the integrity of the marking lines, surface cracks, surface diseases, surface foreign matter, etc. The airport needs to pay attention to the dynamic changes of the above information at all times to eliminate safety hazards and ensure the safe operation of the airport.

[0003] At present, the airport pavement surface image detection adopts manual visual inspection to obtain the pavement surface image information. The inspection process is that one inspector drives the car while another staff member observes the pavement surface condition. If any abnormality is encountered, the car will stop to observe and record the information manually. This manual inspection method has the disadvantages of large workload, low efficiency, poor accuracy, easy to miss detection, and strong subjectivity.

[0004] With the rapid growth of air traffic and the commissioning of new smart airports, some airports have introduced inspection robots. The operation mode of the inspection robot is to mount the array camera on the motion platform, collect the surface image information of the road surface and store it on the onboard hard disk. After the inspection task is completed, the inspection data is copied manually and analyzed based on the recognition algorithm. However, due to the limitation of the imaging principle, the exposure time of the array camera is relatively long, and it is easy to produce smear under high-speed motion conditions, which affects the accuracy of the analysis results. Therefore, the inspection speed is greatly limited. If a high-power light source is used, its high cost and high heat generation bring new problems. At the same time, the offline data processing method reduces the real-time acquisition of information, and it is impossible to check abnormalities in the first time, which poses certain safety hazards. In addition, in order to expand the field of view, the imaging plane of the array camera usually needs to be tilted to the ground plane during installation, which will cause tangential distortion of the collected pictures, thus bringing difficulties to image stitching. Summary of the invention

[0005] In view of the above technical problems, the technical solution adopted by this application is:

[0006] The embodiment of the present application provides an airport pavement surface image detection system based on a linear array camera, comprising: a moving chassis and an imaging control device, a visual imaging device, an industrial computer and a positioning device fixed on the moving chassis; the industrial computer is respectively connected to the imaging control device, the visual imaging device and the positioning device in communication; wherein the moving chassis is used to travel on the airport pavement to be detected according to a preset driving path based on the instructions of the industrial computer; the imaging control device is fixed to the front wheel axle of the moving chassis, and is used to record the rotation angle of the front wheel, and generate a corresponding trigger signal based on each recorded rotation angle and send it to the visual imaging device; the visual imaging device includes a linear array camera, which is used to perform linear array scanning imaging on the airport pavement to be detected based on the received trigger signal, obtain a corresponding pavement surface image and send it to the industrial computer; the positioning device is used to obtain the position information of the moving chassis and send it to the industrial computer; the industrial computer is used to detect abnormal targets in the received pavement surface image, and when the abnormal target is detected, the area and position of the abnormal target are determined based on the position information obtained by the positioning device and a corresponding detection result is generated.

[0007] The airport pavement surface image detection system based on a linear array camera provided by the present application can process the images collected by the visual imaging device in a timely manner because the visual imaging device and the industrial computer are mounted on a moving chassis, and the information acquisition is highly real-time, and abnormal targets on the pavement can be checked in the first place. In addition, because the visual imaging device uses a linear array camera for image acquisition, the acquired images are highly accurate, making image stitching easy. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0009] Figure 1 A schematic diagram of the structure of an airport pavement surface image detection system based on a linear array camera provided in an embodiment of the present application;

[0010] Figure 2 A schematic diagram of the structure of a single mounting bracket of an airport pavement surface image detection system based on a linear array camera provided in an embodiment of the present application;

[0011] Figure 3 A top view of a visual imaging device of an airport pavement surface image detection system based on a linear array camera provided in an embodiment of the present application;

[0012] Figure 4A front view of a visual imaging device of an airport pavement surface image detection system based on a linear array camera provided in an embodiment of the present application;

[0013] Figure 5 A schematic diagram of the structure of a photoelectric rotary encoder of an airport pavement surface image detection system based on a linear array camera provided in an embodiment of the present application;

[0014] Figure 6 A schematic diagram of the inspection route of an airport pavement surface image detection system based on a linear array camera provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0016] Figure 1 A schematic diagram of the structure of an airport pavement surface image detection system based on a linear array camera provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a single mounting bracket of an airport pavement surface image detection system based on a linear array camera provided in an embodiment of the present application; Figure 3 A top view of a visual imaging device of an airport pavement surface image detection system based on a linear array camera provided in an embodiment of the present application; Figure 4 A front view of a visual imaging device of an airport pavement surface image detection system based on a linear array camera provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of a photoelectric rotary encoder of an airport pavement surface image detection system based on a linear array camera provided in an embodiment of the present application; Figure 6 A schematic diagram of the inspection route of an airport pavement surface image detection system based on a linear array camera provided in an embodiment of the present application.

[0017] like Figure 1 and Figure 2 As shown, the embodiment of the present application provides an airport pavement surface image detection system based on line scanning imaging, including a moving chassis 1, a visual imaging device 2, a positioning device 3, an imaging control device 5 and an industrial computer 6. Among them, the visual imaging device 2, the positioning device 3, the imaging control device 5 and the industrial computer 6 are all fixed on the moving chassis 1 and move with the movement of the moving chassis. The industrial computer 6 is respectively connected to the visual imaging device 2, the positioning device 3, and the imaging control device 5 for communication.

[0018] In the embodiment of the present application, the industrial computer 6 may include a motion control module, a data acquisition module and a data processing module. The motion control module is connected to the motion chassis for controlling the motion of the motion chassis 1, and the data acquisition module is connected to the visual imaging device 2, the positioning device 3 and the data processing module.

[0019] In the embodiment of the present application, the motion chassis 1 may be a square structure, including four wheels 4, which are used to travel on the airport pavement to be inspected according to a preset driving path under the control of the motion control module of the industrial computer. Before the inspection work begins, the user inputs the landmark GPS points in the inspection process and draws the trajectory, and uses the remote control to control the motion chassis 1 to move to the starting point of the inspection task. The motion control module controls the motion chassis to move along the trajectory from the starting point until the inspection task ends at the end point. Taking the airport runway as an example, the preset inspection route can be as follows: Figure 6 In addition, to ensure safe operation, a millimeter wave radar + ultrasonic radar can be installed on the motion chassis 1 for obstacle avoidance, and the principle of "stopping when encountering a dynamic obstacle and bypassing when encountering a static obstacle" is followed during the inspection process.

[0020] Furthermore, if Figure 1 and Figure 5 As shown, in the embodiment of the present application, the imaging control device 5 is fixed to the front wheel axle 11 of the moving chassis 1, and is used to record the rotation angle of the front wheel, and generate a corresponding trigger signal based on each recorded rotation angle and send it to the visual imaging device 2. For each rotation of the wheel, the imaging control device 5 generates N trigger signals, and the trigger signal is a pulse signal. Specifically, in the embodiment of the present application, the imaging control device 5 can be a photoelectric rotary encoder, such as Figure 5 As shown, the photoelectric rotary encoder is connected to the front wheel axle 11 of the moving chassis through a coupling, rotates synchronously with the front wheel, encodes the wheel rotation angle, and generates an imaging control pulse signal N times / revolution, which is sent as a trigger signal to the visual imaging device to control it to perform image scanning. The number of pulses N output per revolution of the photoelectric rotary encoder is related to the circumference Cs of the moving chassis wheel and the accuracy r of the visual imaging module, and the number of pulses N should be greater than Cs / r. In an illustrative embodiment, the number of pulses output per revolution of the photoelectric rotary encoder N=3000, the circumference of the moving chassis wheel Cs=600mm, and the accuracy of the visual imaging device r=0.5mm. Furthermore, as Figure 5 As shown, a protective cover 12 may be provided on the photoelectric rotary encoder to protect the photoelectric rotary encoder.

[0021] In an embodiment of the present application, the visual imaging device 2 includes a linear array camera, which is used to perform linear array scanning imaging on the airport pavement to be inspected based on the received trigger signal, obtain the corresponding pavement surface image and send it to the industrial computer 6, specifically to the data acquisition module of the industrial computer 6.

[0022] Further, in the embodiments of the present application, Figure 1 , Figure 2 , Figure 3 and Figure 4 As shown, the visual imaging device 2 may include: a first linear array scanning imaging group 2-1 and a second linear array scanning imaging group 2-2, each linear array scanning imaging group includes a linear array camera and a linear light source, that is, the first linear array scanning imaging group 2-1 includes a first linear array camera and a first linear light source, and the second linear array scanning imaging group 2-2 includes a second linear array camera and a second linear light source. The pulse signal generated by the imaging control device 5 can be converted into two pulse signals through a splitter, and the two linear array scanning imaging groups are controlled to perform image scanning at the same time.

[0023] like Figure 1 As shown, each linear array scanning imaging group can be fixed on the top of the moving chassis by means of a mounting bracket 7 to prevent the moving chassis from blocking the field of view of the linear array camera. Figure 2 As shown, the mounting bracket 7 may include a fixing bracket 701, a connecting bracket 702 and a mounting bracket 703. The fixing bracket 701 may be fixed to the moving chassis 1 by, for example, bolts. The connecting bracket 702 and the fixing bracket 701 are vertically connected and may be connected by a reinforcing rod 704 to improve the connection strength between the two. The mounting bracket 703 is arranged above the connecting bracket 702 and is used to fix the scanning imaging group. In order to reduce the shaking of the visual imaging device 2 during the movement of the moving chassis 1 and avoid affecting the imaging effect, a rubber pad is inserted between the visual imaging device 2 and the mounting bracket 7.

[0024] Furthermore, in the embodiment of the present application, a narrow band filter with a wavelength of a1 is installed at the front end of each linear array camera, and its optical axis is vertically downward, and the imaging plane is perpendicular to the extended line of the driving direction of the moving chassis 1, so as to perform two-dimensional imaging scanning on the road surface. Each linear light source can be a laser sheet light, and the wavelength can range from 800 to 920 nm. The plane illuminated by the linear light source is coplanar with the plane of the linear array camera to provide illumination for the linear array camera.

[0025] Further, in the embodiments of the present application, Figure 4 As shown, the imaging fields of the first linear array scanning imaging group 2-1 and the second linear array scanning imaging group 2-2 have a preset overlapping area 10, that is, an overlapping length. The imaging planes of the two linear array cameras are coplanar, and the wavelength a1-1 of the first linear light source is different from the wavelength a1-2 of the second linear light source, that is, a1-1≠a1-2, so as to reduce mutual interference during operation, and the two linear light sources illuminate the overlapping area, that is, the preset overlapping area does not exceed b, and the value range of b is 400-800mm, so that the two linear array scanning imaging groups can achieve field of view splicing in the road surface scanning area. Furthermore, a1-1=808nm, a1-2=915nm, to ensure safety.

[0026] Furthermore, in the embodiment of the present application, the field of view width of the visual imaging device 2 can be approximately 3-4m. In order to cover the entire road surface image, a cluster inspection of multiple units can be carried out simultaneously or a single unit can be operated reciprocatingly, while ensuring that the fields of view of two adjacent paths have an overlapping area of ​​20-30% to ensure subsequent image stitching.

[0027] Furthermore, the installation height of the first linear array scanning imaging group 2-1 and the second linear array scanning imaging group 2-2, that is, the height from the airport pavement is h and the spacing is d. The setting of the installation height h can be determined in combination with the depth of field of the linear array camera (determining the clarity of the imaging), the acquisition efficiency, and the overall stability of the moving chassis. The installation height determines the field of view. In an illustrative embodiment, the value range of h is 1500-2000mm, and the field of view is 2000mm-2400mm. The camera spacing d can be 1400-1600mm to ensure a 20-30% overlap rate of the field of view.

[0028] Further, in the embodiment of the present application, the visual imaging device needs to perform camera calibration after determining the installation height. The calibration process can adopt the existing technology. In an exemplary embodiment, the calibration process can be: placing the checkerboard image at different positions on the road surface and tilting it at different angles, using the visual imaging device to collect 15-20 different checkerboard images, using Zhang Zhengyou's camera calibration method to solve the internal parameter matrix A and the external parameter matrix [Rt], and using the internal parameter matrix A to correct the image distortion, and using the internal parameter matrix A and the external parameter matrix [Rt] to calculate the transformation relationship of a single point from the image pixel coordinate system to the world coordinate system, so that the data processing module reads the position information of the moving chassis output by the positioning device, calculates the size and actual position coordinates of the apparent abnormal area of ​​the road surface, and realizes the accurate positioning of the abnormal area. Further, in the embodiment of the present application, the positioning device 3 is used to obtain the position information of the moving chassis and send it to the data acquisition module of the industrial computer 6. In an exemplary embodiment, the hardware part of the positioning device 3 can be a GPS receiver and an IMU inertial navigation combined navigation form.

[0029] Furthermore, it also includes a counter C, the initial value of C is 0; the counter is connected to the imaging control device and the industrial computer, and each time the imaging control device generates a trigger signal, C=C+1; wherein, when C is equal to the preset number threshold N, the visual imaging device generates an image, and C=0, that is, the counter restarts counting; the imaging control device 5 is also used to control the positioning device 3 to obtain the current position information of the moving chassis when C=1. The current position information obtained by the positioning device 3 will be sent to the data processing module, and the position information is used as the reference information of each image.

[0030] In the embodiment of the present application, a preset number threshold N=4096 can be set, which is consistent with the resolution of the visual imaging device, so that the imaging size is a square of 4096×4096. During the detection process, the imaging control device 5 sends a trigger signal to the visual imaging device 2. Each time the visual imaging device 2 receives a trigger signal, it collects a line of image. At the same time, the imaging control device 5 obtains the number of counters and determines whether the number reaches the preset number threshold N. If the number of trigger signals reaches N, the visual imaging device is controlled to output an image and the trigger counter is cleared.

[0031] Furthermore, in an embodiment of the present application, the data acquisition module is used to obtain the data collected by the visual imaging device 2 and the positioning device 3 and store and send it to the data processing module, and can be hardware-connected to the visual imaging device via a gigabit network cable.

[0032] In the embodiment of the present application, the image collected by the visual imaging device is bitmap data, which is encapsulated in bmp format in the data acquisition module and then compressed into jpg format in the data processing module. The bmp format image can be effectively reduced in file size after jpg compression, which is conducive to the rapid transmission of the image to the background server through the 5G network.

[0033] Furthermore, in an embodiment of the present application, the data processing module is used to collect data on the surface image of the road collected by the visual imaging device, store it, and receive the position information of the moving chassis sent by the positioning device. Set the file name and storage location of the image, and set the properties of the linear array camera, including exposure time, output frame length and width, etc. For example, the image name is set to "longitude_latitude_altitude.jpg", the exposure time is set to 50us, and the output frame length and width are 4096×4096. The data processing module decodes the GPS data received from the positioning device, and calculates the position information of the moving chassis in combination with the IMU inertial navigation data.

[0034] Furthermore, the data processing module is used to detect abnormal targets in the received road surface image, and when abnormal targets are detected, the area and position of the abnormal targets are determined based on the position information obtained by the positioning device and the corresponding detection results are generated. The abnormal targets may include incomplete marking lines, surface cracks, surface diseases, and surface foreign objects.

[0035] Specifically, the data processing module is used to perform the following operations:

[0036] S100, based on the semantic segmentation algorithm of deep learning, performs pixel-level classification and extraction on abnormal targets in each received road surface image to obtain the shape information of the abnormal targets.

[0037] S200, using the area calculation function and the center of gravity calculation function of OpenCV according to the shape information of the abnormal target to obtain the pixel value area Ap of the abnormal target and the pixel value coordinates (xp, yp) in the image.

[0038] S300, based on the position information obtained by the positioning device, obtain the position information (xL, yL) of the moving chassis corresponding to the abnormal target, and calculate the actual area A=λ of the abnormal target in combination with the pixel value coordinates (xp, yp) of the abnormal target in the image 2 Ap and the actual position (x, y) = λ(xp, yp) + (xL, yL) + (x△, y△), where λ is the conversion coefficient between each pixel in the image and the actual size, and (x△, y△) is the position offset relationship between the pixel reference origin in the image and the reference point of the moving chassis. In the embodiment of the present application, the acquisition accuracy of the visual imaging device 2 is 0.5mm, so the actual size corresponding to each pixel is 0.5mm, that is, λ = 0.5, and the actual area corresponding to each pixel is 0.25mm2, that is, λ 2 =0.25.

[0039] S400, generating corresponding detection results based on the calculated actual area and actual position of the abnormal target. The detection result can be a file m.txt, which is stored in the same folder as the original image. The content of the detection result file m.txt includes: the actual position (x, y) of the abnormal target, the area size, and the abnormal type determination. The detection result file name can be set to: abnormal type determination + sequential number (starting number is 0000).

[0040] Furthermore, the data processing module is also used to perform the following operations:

[0041] S500, stitching each image captured by each linear array camera through a RANSAC algorithm to generate a continuous road surface image;

[0042] S600: stitching the continuous road surface images of the two stitched line array cameras to obtain a road surface image covered by the current path.

[0043] Furthermore, in the embodiment of the present application, in order to avoid the positioning device in some special environments (such as indoor environments) where the accuracy is reduced due to a weak GPS signal, a photoelectric rotary encoder is used to correct the positioning data. When the positioning device receives a weak GPS signal at time t1, the position information of the moving chassis at time t0 when the GPS signal strength is in the normal range and the pulse signal accumulated by the photoelectric rotary encoder in the time period t0-t1 are called, and the corrected moving chassis position information is calculated in the data processing module. In order to reduce the cumulative error, the times t0 and t1 should be as close as possible.

[0044] Furthermore, in the embodiment of the present application, a data transmission device 8 is also included. The data transmission device 8 is arranged on the moving chassis and is connected to the industrial computer 6 for communication, and is used to send the image with abnormal targets and the corresponding detection results and the entire image of the current path obtained by the industrial computer 6 to the background server. The data transmission device 8 can be a router, and a 5G traffic card is inserted into the SIM card slot of the router. The detection result file is packaged and sent to the background server in real time through the 5G local area network, and the entire spliced ​​surface image of the road surface is transmitted to the background server after the inspection is completed.

[0045] Furthermore, in the embodiment of the present application, a power supply device (not shown) is also included to supply power to the imaging control device, the visual imaging device, and the industrial computer.

[0046] In summary, the airport pavement surface image detection system based on linear array cameras provided in the embodiment of the present application uses a linear array module for large-format, high-resolution acquisition. It can complete the high-definition acquisition of the entire pavement surface image within ten minutes when mounted on a high-speed motion chassis, with high inspection efficiency and high acquisition accuracy. It uses a high-brightness, low-energy linear laser light source, so the inspection work is not restricted by night lighting, and 24-hour all-weather inspection can be achieved. The linear array module has a low cost and a high degree of promotion. The front-end industrial computer runs the pavement surface image recognition algorithm, which can avoid missed inspections, improve accuracy, and eliminate the subjectivity of manual inspections. It can realize real-time algorithm processing, detect abnormal pavement surface conditions in the first time and quickly locate them, which can effectively improve the level of airport operation safety management.

[0047] Although some specific embodiments of the present application have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are only for illustration and not for limiting the scope of the present application. It will also be appreciated by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. An airport pavement surface image detection system based on a linear array camera, characterized in that: include: A moving chassis and an imaging control device, a visual imaging device, an industrial computer and a positioning device fixed on the moving chassis; the industrial computer is respectively connected to the imaging control device, the visual imaging device and the positioning device for communication; Wherein, the motion chassis is used to travel on the airport pavement to be inspected according to a preset driving path based on the instruction of the industrial computer; The imaging control device is fixed to the front wheel axle of the moving chassis, and is used to record the rotation angle of the front wheel, and generate a corresponding trigger signal based on each recorded rotation angle and send it to the visual imaging device; The visual imaging device includes a linear array camera, which is used to perform linear array scanning imaging on the airport pavement to be detected based on the received trigger signal, obtain the corresponding pavement surface image and send it to the industrial computer; The positioning device is used to obtain the position information of the moving chassis and send it to the industrial computer; The industrial computer is used to detect abnormal targets in the received road surface image, and when an abnormal target is detected, the area and position of the abnormal target are determined based on the position information obtained by the positioning device and a corresponding detection result is generated; The industrial computer is specifically used to perform the following operations: Based on the semantic segmentation algorithm of deep learning, the abnormal targets in each received road surface image are classified and extracted at the pixel level to obtain the shape information of the abnormal targets; According to the shape information of the abnormal target, use OpenCV's area calculation function and center of gravity calculation function to obtain the pixel value area Ap of the abnormal target and the pixel value coordinates (xp, yp) in the image; Based on the position information obtained by the positioning device, the position information (xL, yL) of the moving chassis corresponding to the abnormal target is obtained, and the actual area A=λ of the abnormal target is calculated by combining the pixel value coordinates (xp, yp) of the abnormal target in the image. 2 Ap and actual position (x, y) = λ(xp, yp) + (xL, yL) + (x△, y△), where λ is the conversion factor between each pixel in the image and the actual size, and (x△, y△) is the position offset relationship between the pixel reference origin in the image and the reference point of the moving chassis; The corresponding detection result is generated based on the calculated actual area and actual position of the abnormal target.

2. The system according to claim 1, characterized in that It also includes a counter C, the initial value of C is 0; the counter is connected to the imaging control device and the industrial computer for communication, and each time the imaging control device generates a trigger signal, C=C+1; wherein, when C is equal to a preset number threshold N, the visual imaging device generates an image, and C=0; The imaging control device is also used to control the positioning device to obtain the current position information of the moving chassis when C=1.

3. The system according to claim 1, characterized in that The visual imaging device comprises: a first linear array scanning imaging group and a second linear array scanning imaging group, each linear array scanning imaging group comprises a linear array camera and a linear light source; and the imaging fields of the two linear array scanning imaging groups have overlapping areas.

4. The system according to claim 3, characterized in that The industrial computer is also used to perform the following operations: Each image captured by each linear array camera is stitched together using the RANSAC algorithm to generate a continuous road surface image; and The continuous road surface images of the two stitched linear array cameras are stitched together to obtain the road surface image covered by the current path.

5. The system according to claim 3, characterized in that The length of the overlapping area is 400-800 mm.

6. The system according to claim 3, characterized in that A narrow-band filter is installed at the front end of the line array camera, the optical axis of the line array camera is vertically downward, and the imaging plane is perpendicular to the extended line of the driving direction of the moving chassis.

7. The system according to claim 3, characterized in that The wavelengths of the line light sources of the first line scanning imaging group and the second line scanning imaging group are different.

8. The system according to claim 3, characterized in that The height of the line array camera from the airport pavement is 1500-2000 mm, and the distance between the two line array cameras is 1400-1600 mm.

9. The system according to any one of claims 1 to 8, characterized in that: The imaging control device is a photoelectric rotary encoder.

Images