Visual detection method and device for bridge diseases, computer equipment, computer readable storage medium and computer program product

Abstract

The invention relates to a bridge disease visual detection method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a bridge time sequence image shot by an image acquisition device on the bridge inspection vehicle in the running process of the bridge inspection vehicle, movement position information of the bridge inspection vehicle when the bridge time sequence image is shot, and laser point cloud data acquired in a preset range of the bridge inspection vehicle; outputting a disease detection result corresponding to each image frame in the bridge time sequence image through a disease recognition model; projecting the laser point cloud data into a bridge time sequence image according to the motion position information and calibration parameters of the image acquisition device to obtain a matching relationship between the laser point cloud data and a disease detection result; and according to the matching relationship, performing de-overlapping and merging processing on the disease detection result, and according to the laser point cloud data matched with the processed target detection result, determining the bridge disease position. By adopting the method, the detection accuracy of bridge diseases can be improved.

Description

Technical Field

[0001] The present application relates to the field of visual inspection technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium and computer program product for visual inspection of bridge defects. Background Art

[0002] Bridge defects refer to defects in the design and construction of bridges, as well as various damages and destructions caused by factors such as environment and load during use. Bridge defects are one of the main causes of bridge accidents, so regular inspections of bridges are required to repair and maintain the defects.

[0003] In the prior art, maintenance personnel can use a bridge inspection vehicle installed at the bottom of a large bridge to move under the large bridge to conduct regular inspections to see if there are any diseases at the bottom of the bridge. However, since the bridge disease detection in the prior art is mainly based on the maintenance personnel's naked eye observation of the bridge disease, it is easy to have the problem of non-objective and incomplete disease detection. Summary of the invention

[0004] Based on this, it is necessary to provide a visual inspection method, device, computer equipment, computer-readable storage medium and computer program product for bridge defects with higher detection accuracy in order to solve the above technical problems.

[0005] In a first aspect, the present application provides a method for visually detecting bridge defects, comprising:

[0006] Acquire the time-series images of the bridge taken by the image acquisition device on the bridge inspection vehicle during the operation of the bridge inspection vehicle, the movement position information of the bridge inspection vehicle when the time-series images of the bridge are taken, and the laser point cloud data collected within the preset range of the bridge inspection vehicle;

[0007] Inputting the bridge time-series image into a defect recognition model for target detection, and obtaining a defect detection result corresponding to each image frame in the bridge time-series image output by the defect recognition model;

[0008] According to the motion position information and the calibration parameters of the image acquisition device, the laser point cloud data is projected into the time-series image of the bridge to obtain a matching relationship between the laser point cloud data and the disease detection result;

[0009] Performing de-duplication and merging processing on the disease detection results according to the matching relationship to obtain processed target detection results;

[0010] The bridge defect location is determined based on the laser point cloud data matching the target detection result.

[0011] In one embodiment, there are multiple image acquisition devices, and image acquisition areas of the multiple image acquisition devices have overlapping areas;

[0012] The step of projecting the laser point cloud data onto the time-series image of the bridge according to the motion position information and the calibration parameters of the image acquisition device to obtain a matching relationship between the laser point cloud data and the disease detection result includes:

[0013] According to the motion position information and the calibration parameters of each of the image acquisition devices, the laser point cloud data is projected onto the time-series images of the bridge acquired by each of the image acquisition devices to obtain a matching relationship between the laser point cloud data and the corresponding disease detection results of each of the image acquisition devices;

[0014] The de-duplication and merging processing is performed on the disease detection results according to the matching relationship to obtain the processed target detection results, including:

[0015] When the disease detection results collected by multiple image acquisition devices match the same laser point cloud data, the disease detection results collected by the multiple image acquisition devices are de-overlapped and merged to obtain the target detection result.

[0016] In one embodiment, the step of inputting the bridge time-series images into a defect recognition model for target detection, and obtaining a defect detection result corresponding to each image frame in the bridge time-series images output by the defect recognition model, includes:

[0017] Perform feature extraction processing on each of the image frames in the bridge time-series image by using the disease recognition model to obtain image feature data corresponding to each of the image frames;

[0018] Determine, by the disease recognition model, the disease area and disease category of the bridge disease in the bridge time series image according to the image feature data corresponding to each image frame;

[0019] The disease area and the disease category are output as the disease detection result through the disease recognition model.

[0020] In one embodiment, determining the location of the bridge defect based on the laser point cloud data matching the target detection result includes:

[0021] Converting the damaged area in the target detection result into laser point cloud data matching the target detection result to obtain three-dimensional spatial coordinates corresponding to the damaged area;

[0022] The damage category in the target detection result is stored in the three-dimensional space coordinates and output to obtain the bridge damage location.

[0023] In one of the embodiments, the calibration parameters include calibration internal parameters and calibration external parameters of the image acquisition device;

[0024] The step of projecting the laser point cloud data onto the time-series image of the bridge according to the motion position information and the calibration parameters of the image acquisition device to obtain a matching relationship between the laser point cloud data and the disease detection result includes:

[0025] According to the motion position information and the calibrated external parameters of the image acquisition device, the laser point cloud data is projected into a camera coordinate system where the image acquisition device is located, to obtain camera projection data corresponding to the laser point cloud data;

[0026] According to the calibrated internal parameters of the image acquisition device, the camera projection data is converted into two-dimensional projection data in the pixel coordinate system of the image acquisition device;

[0027] When the pixel coordinates of the two-dimensional projection data and the disease detection result are consistent, a matching relationship between the laser point cloud data and the disease detection result is generated.

[0028] In one embodiment, the method further comprises:

[0029] The bridge inspection vehicle is periodically triggered to perform an inspection task, and the inspection task is used to instruct the bridge inspection vehicle to run along the direction of the bridge.

[0030] In a second aspect, the present application also provides a visual detection device for bridge defects, comprising:

[0031] A data acquisition module is used to acquire the time-series images of bridges taken by the image acquisition device on the bridge inspection vehicle during the operation of the bridge inspection vehicle, the movement position information of the bridge inspection vehicle when the time-series images of bridges are taken, and the laser point cloud data collected within a preset range of the bridge inspection vehicle;

[0032] A defect recognition module, used for inputting the bridge time-series image into a defect recognition model for target detection, and obtaining a defect detection result corresponding to each image frame in the bridge time-series image output by the defect recognition model;

[0033] A point cloud matching module, used to project the laser point cloud data into the bridge time-series image according to the motion position information and the calibration parameters of the image acquisition device, and obtain a matching relationship between the laser point cloud data and the disease detection result;

[0034] A result deduplication module is used to perform deduplication and merging processing on the disease detection results according to the matching relationship to obtain a processed target detection result;

[0035] The defect positioning module is used to determine the location of the bridge defect based on the laser point cloud data matching the target detection result.

[0036] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the visual detection method for bridge defects described in any one of the embodiments of the first aspect is implemented.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the visual detection method for bridge defects described in any one of the embodiments of the first aspect above is implemented.

[0038] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the visual detection method for bridge defects described in any one of the embodiments of the first aspect above is implemented.

[0039] The above-mentioned visual inspection method, device, computer equipment, computer-readable storage medium and computer program product for bridge defects are obtained by obtaining the time-series images of the bridge taken by the image acquisition device on the bridge inspection vehicle during the operation of the bridge inspection vehicle, the motion position information of the bridge inspection vehicle when the bridge time-series images are taken, and the laser point cloud data collected within the preset range of the bridge inspection vehicle; the time-series images of the bridge are input into the defect recognition model for target detection, and the defect detection results corresponding to each image frame in the time-series images of the bridge are obtained by the output of the defect recognition model; according to the motion position information and the calibration parameters of the image acquisition device, The laser point cloud data is projected onto the bridge time-series image to obtain the matching relationship between the laser point cloud data and the defect detection results; the defect detection results are de-duplicated and merged according to the matching relationship to obtain the processed target detection results; the location of the bridge defect is determined according to the laser point cloud data that matches the target detection results, which can not only eliminate the detection interference caused by the same bridge defect appearing simultaneously in multiple image frames of the bridge time-series image, thereby improving the detection accuracy of the bridge defect, but also because it does not require on-site operation by maintenance personnel, it can automatically realize bridge defect inspections, which is also beneficial to reducing personnel safety risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A diagram showing an application environment of a visual detection method for bridge defects in one embodiment;

[0042] Figure 2 A schematic diagram of a flow chart of a visual inspection method for bridge defects in one embodiment;

[0043] Figure 3A Schematic diagram of image frames in a time-series image of a bridge in one embodiment Figure 1 ;

[0044] Figure 3B Schematic diagram of image frames in a time-series image of a bridge in one embodiment Figure 2 ;

[0045] Figure 3C Schematic diagram 3 of an image frame in a time-series image of a bridge in one embodiment;

[0046] Figure 4 A schematic diagram of a process for generating disease detection results in one embodiment;

[0047] Figure 5A A schematic diagram of a disease detection result under the coating shedding category in one embodiment;

[0048] Figure 5B It is a schematic diagram of the defect detection results under the categories of paint peeling and bolt missing in one embodiment;

[0049] Figure 5C A schematic diagram of a disease detection result under the crack category in one embodiment;

[0050] Figure 5D is a schematic diagram of disease detection results in one embodiment;

[0051] Figure 6 A schematic diagram of a flow chart of a matching relationship generation step in one embodiment;

[0052] Figure 7 A schematic diagram of a flow chart of a visual inspection method for bridge defects in another embodiment;

[0053] Figure 8 is a structural block diagram of a visual inspection device 800 for bridge defects in one embodiment;

[0054] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0057] The visual detection method of bridge defects provided in the embodiments of the present application can be applied to Figure 1 In the application environment shown, the bridge inspection vehicle control terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers.

[0058] Exemplarily, the server 104 can obtain the bridge time-series images taken by the image acquisition device installed on the bridge inspection vehicle during the operation of the bridge inspection vehicle, the motion position information of the bridge inspection vehicle when taking the bridge time-series images, and the laser point cloud data collected within the preset range of the bridge inspection vehicle through the bridge inspection vehicle control terminal 102. The server 104 can be deployed with a pre-trained disease recognition model, and the bridge time-series images are input into the disease recognition model for target detection, and the disease detection results corresponding to each image frame in the bridge time-series images output by the disease recognition model are obtained. The server 104 projects the laser point cloud data into the bridge time-series images according to the motion position information of the bridge inspection vehicle and the calibration parameters of the image acquisition device, and obtains the matching relationship between the laser point cloud data and the disease detection results. The server 104 performs de-overlapping and merging processing on the disease detection results according to the matching relationship to obtain the processed target detection results. The server 104 can determine the location of the bridge disease according to the laser point cloud data matching the target detection results.

[0059] The bridge inspection vehicle control terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart TVs, smart vehicle-mounted devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0060] In an exemplary embodiment, Figure 2 As shown in FIG. 1 , a visual detection method for bridge defects is provided, and the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps S202 to S210. Among them:

[0061] Step S202, obtaining the time-series images of the bridge taken by the image acquisition device on the bridge inspection vehicle during the operation of the bridge inspection vehicle, the movement position information of the bridge inspection vehicle when the time-series images of the bridge are taken, and the laser point cloud data collected within a preset range of the bridge inspection vehicle.

[0062] Among them, the bridge inspection vehicle can be used to characterize a special vehicle designed specifically for bridge inspection. It can provide a working platform for bridge inspectors during the inspection process and is mainly used for mobile inspection and maintenance operations.

[0063] One or more image acquisition devices may be deployed. Optionally, in some embodiments, the image acquisition device may include any one or more of a variety of devices such as a camera, a webcam, an image sensor, etc.

[0064] The bridge time sequence image can be used to represent a data structure formed by a group of image frames arranged in time sequence. In this embodiment, the bridge time sequence image can include a plurality of bridge bottom images taken in time sequence by the image acquisition device during the operation of the bridge inspection vehicle.

[0065] The motion position information may include the motion direction and three-dimensional spatial position of the bridge inspection vehicle.

[0066] The preset range can be determined based on the distance between the bridge inspection vehicle and the bottom of the bridge. The laser point cloud data can be used to characterize the point cloud data between the bridge inspection vehicle and the bottom of the bridge, that is, the point cloud data near the bridge inspection vehicle.

[0067] For example, the server can receive in real time each image frame captured by the image acquisition device installed on the bridge inspection vehicle during the operation of the bridge inspection vehicle and sent by the bridge inspection vehicle control terminal, and then compose each image frame into a bridge time sequence image according to the acquisition time sequence of the image frame. Alternatively, the server can also periodically receive the bridge time sequence image directly sent by the bridge inspection vehicle control terminal.

[0068] The server can communicate with the programmable logic controller (PLC) installed in the bridge inspection vehicle through the bridge inspection vehicle control terminal, and obtain the movement position information of the bridge inspection vehicle when the bridge time-series image is taken according to the PLC. Among them, the PLC can be used to control the bridge inspection vehicle to run along the direction of the bridge.

[0069] The server can communicate with point cloud acquisition devices such as lidar, depth camera, structured light scanner, time-of-flight (ToF) sensor, stereo vision camera, ultrasonic sensor, etc. in or near the bridge inspection vehicle through the bridge inspection vehicle control terminal to obtain laser point cloud data collected within the preset range of the bridge inspection vehicle.

[0070] Step S204: input the bridge time-series image into the defect recognition model for target detection, and obtain the defect detection result corresponding to each image frame in the bridge time-series image output by the defect recognition model.

[0071] Among them, the disease recognition model can be used to characterize a target recognition model obtained based on deep neural network training.

[0072] The disease detection result may include, but is not limited to, any one or more of a variety of data results such as whether there is a bridge disease, the disease category of the bridge disease, or the location area where the bridge disease is located.

[0073] Exemplarily, a damage recognition model pre-trained using bridge image samples labeled with damage categories may be pre-deployed in the server. The bridge time-series images are input into the damage recognition model, and each image frame in the bridge time-series images is respectively subjected to target detection by the damage recognition model. Based on the features of the bridge image samples under different damage categories learned in the pre-training stage, it is determined whether there is a bridge damage in each image frame, the damage category to which the bridge damage belongs, or the regional location of the bridge damage. The damage detection model outputs the damage detection result corresponding to each image frame.

[0074] Step S206, projecting the laser point cloud data onto the time-series image of the bridge according to the motion position information and the calibration parameters of the image acquisition device, and obtaining a matching relationship between the laser point cloud data and the disease detection result.

[0075] The calibration parameters may include but are not limited to any one or more of the position of the image acquisition device in the bridge inspection vehicle, the installation angle, the focal length of the image acquisition device, the principal point, the distortion parameters, and other data.

[0076] Exemplarily, the server can obtain the calibration parameters of the image acquisition device through the bridge inspection vehicle control terminal. According to the movement position of the bridge inspection vehicle during the bridge time-series image acquisition and the calibration parameters of the image acquisition device, the coordinate system transformation matrix for projecting the laser point cloud data onto the bridge time-series image is determined, and the laser point cloud data is processed using the coordinate system transformation matrix to obtain the two-dimensional projection data of the laser point cloud data in the bridge time-series image. The two-dimensional projection data of the laser point cloud data is compared with the pixel coordinates of the defect detection results in the bridge time-series image. When it is determined that the two-dimensional projection data is consistent with the pixel coordinates, a matching relationship between the laser point cloud data and the defect detection results is established.

[0077] Step S208, performing de-duplication and merging processing on the disease detection results according to the matching relationship to obtain processed target detection results.

[0078] Exemplarily, in general, the disease detection results of the same bridge disease in different image frames of the bridge time series image should match the same laser point cloud data in the world coordinate system. Therefore, according to the matching relationship in step S206, several (which may be one or more) disease detection results that match the same laser point cloud data can be determined. A deduplication operation can be performed to retain only the disease detection result obtained at the earliest acquisition time. Alternatively, the information of all disease detection results is merged. Alternatively, the disease category information of the disease detection result can also be deduplicated, and the disease area location information of the disease detection result can be merged. The disease detection result after deduplication and merging is used as the target detection result. It can be understood by those skilled in the art that the above-mentioned deduplication and merging method does not constitute a limitation on the technical solution of the present application. Based on the inventive concept of the present application, the technical means of deduplication and merging can also be flexibly adjusted according to the actual use requirements in different application scenarios.

[0079] Step S210, determining the location of the bridge defect based on the laser point cloud data that matches the target detection result.

[0080] For example, the server can determine the laser point cloud data matching the target detection result based on the matching relationship between the target detection result and the laser point cloud data, and locate the bridge defect position corresponding to the target detection result at the bottom of the bridge using the laser point cloud data.

[0081] In the above-mentioned visual inspection method for bridge defects, the time-series images of the bridge taken by the image acquisition device on the bridge inspection vehicle during the operation of the bridge inspection vehicle, the motion position information of the bridge inspection vehicle when the bridge time-series images are taken, and the laser point cloud data collected within the preset range of the bridge inspection vehicle are obtained; the time-series images of the bridge are input into the defect recognition model for target detection, and the defect detection results corresponding to each image frame in the time-series images of the bridge are obtained by the output of the defect recognition model; according to the motion position information and the calibration parameters of the image acquisition device, the laser point cloud data are projected into the time-series images of the bridge, and the matching relationship between the laser point cloud data and the defect detection results is obtained; according to the matching relationship, the defect detection results are de-duplicated and merged to obtain the processed target detection results; the location of the bridge defect is determined according to the laser point cloud data matching the target detection results, which can not only eliminate the detection interference caused by the same bridge defect appearing in multiple image frames of the time-series images of the bridge at the same time, thereby improving the detection accuracy of the bridge defect, but also because it does not require on-site operation by maintenance personnel, it can automatically realize bridge defect inspection, which is also conducive to reducing personnel safety risks.

[0082] In an exemplary embodiment, there are multiple image acquisition devices. The image acquisition areas of the multiple image acquisition devices have overlapping areas. The image acquisition areas of each image acquisition device between adjacent image acquisition moments also have overlapping areas. Figure 3A , Figure 3B as well as Figure 3C As shown (the boxed area in the above figure is the detection box where the bridge disease is located, Figure 3A , Figure 3B as well as Figure 3C It can be used to represent image frames acquired by different image acquisition devices at different times for the same area at the bottom of the bridge). In the present application, the same bridge defect may be acquired multiple times at different acquisition times by different image acquisition devices.

[0083] In order to solve the above technical problem, in this embodiment, another implementation method of the above steps S206 to S208 is proposed:

[0084] According to the motion position information and the calibration parameters of each image acquisition device, the laser point cloud data is projected onto the time-series images of the bridge acquired by each image acquisition device, and the matching relationship between the laser point cloud data and the corresponding disease detection results of each image acquisition device is obtained;

[0085] When the disease detection results collected by multiple image acquisition devices match the same laser point cloud data, the disease detection results collected by the multiple image acquisition devices are de-overlapped and merged to obtain the target detection result.

[0086] Exemplarily, the server can obtain the motion position information of the bridge inspection vehicle when each image acquisition device performs image acquisition, as well as the calibration parameters of each image acquisition device relative to the bridge inspection vehicle through the bridge inspection vehicle control terminal. The following operations are performed for each image frame of the bridge time-series image acquired by each image acquisition device: the laser point cloud data within the preset range of the bridge inspection vehicle is processed according to the motion position information at the time of image frame acquisition and the calibration parameters of the image acquisition device, and the laser point cloud data is projected into the corresponding image frame in the bridge time-series image to obtain the two-dimensional projection data of the laser point cloud data in the image frame. The two-dimensional projection data of the laser point cloud data is matched with the defect area where the defect detection result in the image frame is located, so as to obtain the matching relationship between the laser point cloud data and the defect detection result in each image frame. Furthermore, a matching relationship between the laser point cloud data and the defect detection result corresponding to the bridge time-series image acquired by each image acquisition device is constructed.

[0087] The established matching relationships are compared. When the disease detection results collected by multiple image acquisition devices match the same laser point cloud data, only one of the disease detection results needs to be retained. The duplicate disease detection results can be processed with reference to the deduplication and merging operations provided in the above step S208 to obtain the target detection results after deduplication and merging.

[0088] In this embodiment, by simultaneously acquiring time-series images of the bridge by multiple image acquisition devices, de-duplication and merging are performed on the defect detection results obtained by each image acquisition device at each image acquisition moment. This can avoid interference with bridge defect detection when the same bridge defect is acquired multiple times, and reduce the amount of data calculation required for subsequent positioning of bridge defects.

[0089] In an exemplary embodiment, Figure 4 As shown, step S204 includes steps S402 to S406. Among them:

[0090] Step S402: Perform feature extraction processing on each image frame in the bridge time series image through the defect recognition model to obtain image feature data corresponding to each image frame.

[0091] Exemplarily, in the model pre-training stage, the server can pre-acquire sample images of defects in the bridge defect scene. Receive the user's annotation data on the outline of the bridge defect area in the bridge image and its bridge defect category. Use the defect sample images as training data and the annotation data as training labels, and input them into the initial deep neural network, such as the location-based object segmentation network (Segmenting Objects by Locations, referred to as SOLO network), the monocular target detection network (You Only Look Once, referred to as YOLO network), etc. According to the defect classification results output by the initial deep neural network and the loss value of the annotation data, the initial deep neural network is iteratively trained until the iteration stop condition is met, and a pre-trained defect recognition model is obtained.

[0092] The server can input the bridge time-series images into the defect recognition model, and use the defect recognition model to perform feature extraction processing on each image frame in the bridge time-series images, such as extracting the color, texture, shape and other features of the image frame, so as to obtain the image feature data corresponding to each image frame.

[0093] Step S404: Determine the defect area and defect category of the bridge defect in the bridge time series image according to the image feature data corresponding to each image frame through the defect recognition model.

[0094] Among them, the defect area can be used to characterize the detection frame area where the bridge defect is located.

[0095] Defect categories may include but are not limited to crack categories, paint peeling categories, bolt missing categories, etc.

[0096] Exemplarily, the server can use the defect recognition model to classify and identify the image feature data corresponding to each image frame in the currently received bridge time-series image based on the image features corresponding to different defect categories learned from the defect sample images in the pre-training phase, determine whether there is a bridge defect in the bridge time-series image, and if there is a bridge defect, further determine the defect area and defect category in the image frame where the bridge defect exists.

[0097] Step S406: output the disease area and disease category as the disease detection result through the disease recognition model.

[0098] For example, the server can output the disease area and disease category detected in the image frame as the disease detection result of the image frame through the disease recognition model. Figure 5A As shown in FIG. 1 , a defect detection result of a defect recognition model outputting a defect category of paint peeling is shown. Figure 5BAs shown in FIG. 1 , a defect detection result of a defect recognition model outputting a defect category including a paint peeling category and a bolt missing category is shown. Figure 5C As shown, a defect detection result of a defect identification model outputting a defect category of crack category is shown. Among them, Figure 5A and Figure 5B The C in the code is used to indicate the coating peeling category. Figure 5B The B in the bolt is used to indicate that the bolt lacks a category. Figure 5C The word "crock" in the formula is used to indicate the type of crack. Figure 5A , Figure 5B as well as Figure 5C The decimal in represents the probability that the bridge disease in the image frame belongs to the disease category calculated by the disease recognition model.

[0099] For example, when the bridge inspection vehicle runs from one end of the bridge to the other end, the defect recognition model performs target detection on the image frames collected at different times in the bridge time series image, and can identify the following defects: Figure 5D The disease detection results of the bridge at different locations shown in the figure are helpful to improve the comprehensiveness of the inspection during the inspection process of the bridge inspection vehicle.

[0100] In this embodiment, by using a deep neural network-based disease recognition model to determine the disease category and disease area of ​​bridge diseases in the image frame, the accuracy of the disease detection results can be improved.

[0101] In an exemplary embodiment, the above step S210 may also be implemented by the following method:

[0102] The defect area in the target detection result is converted into laser point cloud data matching the target detection result to obtain the three-dimensional space coordinates corresponding to the defect area; the defect category in the target detection result is stored in the three-dimensional space coordinates and output to obtain the bridge defect location.

[0103] For example, the server can convert the two-dimensional defect area in the target detection result to the corresponding three-dimensional space area in the laser point cloud data according to the matching relationship between the target detection result and the laser point cloud data, thereby obtaining the three-dimensional space coordinates corresponding to the defect area in the image frame. The defect category in the target detection result is stored in the corresponding three-dimensional space coordinate output. The bridge defect position where the bridge defect exists in the bridge is determined according to the three-dimensional space coordinates, and the defect category of the bridge defect position is recorded.

[0104] In this embodiment, by matching the image pixels with the laser point cloud, the two-dimensional defect area and the defect category detected in the two-dimensional image are converted to the bridge defect position in the three-dimensional space and the corresponding defect category is stored at the bridge defect position, so that the bridge defect condition can be displayed to the user more intuitively.

[0105] In an exemplary embodiment, the calibration reference may include a calibration internal parameter and a calibration external parameter of the image acquisition device.

[0106] like Figure 6 As shown, the above step S206 may include steps S602 to S606. Among them:

[0107] Step S602 , projecting the laser point cloud data into the camera coordinate system where the image acquisition device is located according to the motion position information and the calibrated external parameters of the image acquisition device, to obtain camera projection data corresponding to the laser point cloud data.

[0108] Among them, the calibration external parameters may include but are not limited to parameter information such as the installation position and installation posture of the image acquisition device in the bridge inspection vehicle.

[0109] Exemplarily, the server can construct a first coordinate system transformation matrix based on the motion position information of the bridge inspection vehicle when the image acquisition device acquires images and the calibration external parameters of the image acquisition device. The laser point cloud data is processed using the first coordinate system transformation matrix, and the laser point cloud data is projected from the three-dimensional coordinate system corresponding to the bridge inspection vehicle to the camera coordinate system where the image acquisition device is located (both coordinate systems are coordinate systems in three-dimensional space, and the difference lies in the different origin reference objects of the coordinate systems), and the camera projection data corresponding to the laser point cloud data is obtained.

[0110] Step S604: converting the camera projection data into two-dimensional projection data in a pixel coordinate system of the image acquisition device according to the calibrated internal parameters of the image acquisition device.

[0111] The calibration internal parameters may include but are not limited to parameter information such as focal length, principal point, and distortion parameters of the image acquisition device.

[0112] Exemplarily, the server may obtain calibration parameters of the image acquisition device. A second coordinate system transformation matrix for converting from a camera coordinate system of the image acquisition device to a pixel coordinate system of the bridge time-series image acquired by the image acquisition device is calculated based on the calibration internal parameters. The camera projection data of the laser point cloud data is processed using the second coordinate system transformation matrix to convert the camera projection data into two-dimensional projection data in the pixel coordinate system of the image acquisition device.

[0113] Step S606: When the pixel coordinates of the two-dimensional projection data and the defect detection result are consistent, a matching relationship between the laser point cloud data and the defect detection result is generated.

[0114] For example, the server may compare the two-dimensional projection data of the laser point cloud data in the bridge time-series image with the pixel coordinates of the defect detection result in the bridge time-series image. When it is determined that the pixel coordinates of the two-dimensional projection data and the defect detection result are consistent, a matching relationship between the laser point cloud data and the defect detection result is generated.

[0115] In this embodiment, by using the motion position information of the bridge inspection vehicle at the time of image acquisition and the calibrated external parameters and calibrated internal parameters of the image acquisition device, the three-dimensional laser point cloud data is projected into the two-dimensional bridge time-series image, and a matching relationship between the laser point cloud data and the two-dimensional defect detection results is established, which can help identify repeatedly detected bridge defects.

[0116] In an exemplary embodiment, the visual detection method for bridge defects provided in the present application may also include: regularly triggering a bridge inspection vehicle to perform an inspection task, where the inspection task is used to instruct the bridge inspection vehicle to run along the direction of the bridge.

[0117] For example, the server may be provided with periodic inspection triggering logic, such as logic for triggering an inspection task at preset intervals, or logic for triggering an inspection task when the time interval from the last inspection task execution reaches a preset time length, etc. By periodically triggering the bridge inspection vehicle to execute the inspection task, the bridge inspection vehicle may be instructed to run along the direction of the bridge.

[0118] In this embodiment, by regularly triggering the bridge inspection vehicle to perform inspection tasks, the frequency of bridge disease detection can be increased, and short-term rapidly developing bridge diseases can be discovered in time.

[0119] In an exemplary embodiment, Figure 7 As shown, a bridge disease detection method is also provided, including the following steps S702 to S714. Among them:

[0120] Step S702, periodically triggering the bridge inspection vehicle to perform inspection tasks, where the inspection tasks are used to instruct the bridge inspection vehicle to run along the direction of the bridge.

[0121] Exemplarily, the server can periodically send control instructions to the bridge inspection vehicle control terminal to trigger the bridge inspection vehicle to regularly perform inspection tasks and run along the direction of the bridge. During the operation, the server controls multiple image acquisition devices installed in the bridge inspection vehicle to start capturing images of the bottom of the bridge at preset acquisition intervals.

[0122] Step S704, obtaining the time-series images of the bridge taken by the image acquisition device on the bridge inspection vehicle during the operation of the bridge inspection vehicle, the movement position information of the bridge inspection vehicle when the time-series images of the bridge are taken, and the laser point cloud data collected within the preset range of the bridge inspection vehicle.

[0123] For example, the server can obtain a time-series image of a bridge composed of multiple image frames taken in chronological order by multiple image acquisition devices installed on the bridge inspection vehicle during its operation. The programmable logic controller in the bridge inspection vehicle obtains the motion position information of the bridge inspection vehicle when each image acquisition device performs image acquisition, as well as the laser point cloud data collected within a preset range of the bridge inspection vehicle.

[0124] Step S706: input the bridge time-series images into the defect recognition model, and output the defect areas and defect categories identified in the bridge time-series images as defect detection results through the defect recognition model.

[0125] Exemplarily, the server may store sample images of defects that are labeled with defect categories and defect area contours. A corresponding defect recognition model is trained based on the sample images of defects. The currently received time-series bridge images are input into the defect recognition model for target detection and classification, and the defect area and defect category corresponding to each image frame in the time-series bridge images are determined, so that the defect area in the image frame and its corresponding defect category are output as the defect detection result of the image frame.

[0126] Step S708, projecting the laser point cloud data onto the time-series image of the bridge according to the motion position information and the calibration parameters and calibration external parameters of the image acquisition device, to obtain two-dimensional projection data corresponding to the laser point cloud data.

[0127] Exemplarily, the server can use the motion position information of the bridge inspection vehicle at the time of image acquisition and the calibrated external parameters of the image acquisition device to perform calculations and generate a first coordinate system transformation matrix. Perform calculations and processing based on the calibrated external parameters and calibration parameters of the image acquisition device to generate a second coordinate system transformation matrix. Use the first coordinate system transformation matrix to perform calculations and processing on the laser point cloud data, project the laser point cloud data to the camera coordinate system where the image acquisition device is located, and obtain the camera projection data corresponding to the laser point cloud data. Use the second coordinate system transformation matrix to perform calculations and processing on the camera projection data, project the camera projection data to the pixel coordinate system where the bridge time series image is located, and obtain the corresponding two-dimensional projection data.

[0128] Step S710: When the pixel coordinates of the two-dimensional projection data and the defect detection result are consistent, a matching relationship between the laser point cloud data and the defect detection result is generated.

[0129] Step S712, de-duplication and merging processing is performed on the disease detection results according to the matching relationship to obtain processed target detection results.

[0130] Exemplarily, when the server determines that the two-dimensional projection data of the laser point cloud data in the image frame is consistent with the pixel coordinates of the defect detection result in the image frame, a matching relationship between the laser point cloud data and the defect detection result can be generated. According to the matching relationship between the defect detection result in each image frame collected by each image acquisition device and the laser point cloud data, the defect detection results are de-duplicated and merged to obtain the target detection result. That is, the target detection result includes a number (including one or more) of unique defect detection results corresponding to the laser point cloud data.

[0131] Step S714, based on the laser point cloud data matching the target detection result, determine the three-dimensional space coordinates corresponding to the defect area of ​​the target detection result, store the defect category of the target detection result in the three-dimensional space coordinates and output it to obtain the bridge defect location.

[0132] For example, the server can use the three-dimensional spatial coordinates of the laser point cloud data matching the target detection result as the three-dimensional spatial coordinates corresponding to the defect area of ​​the target detection result, store the defect category of the target detection result in the three-dimensional spatial coordinates and output it to obtain the bridge defect location.

[0133] In this embodiment, the frequency of bridge defect detection can be increased by periodically triggering the bridge inspection vehicle to perform inspection tasks. The accuracy of bridge defect detection can be improved by using a defect recognition model based on a deep neural network to determine the defect detection result in each image frame. By performing de-overlapping and merging processing based on the matching relationship between laser point cloud data and defect detection results, the same bridge defect can be avoided from appearing in multiple defect detection results, reducing data redundancy of target detection results, thereby helping to improve the efficiency of subsequent processing of locating the bridge defect position.

[0134] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0135] Based on the same inventive concept, the embodiment of the present application also provides a visual detection device for bridge defects for implementing the visual detection method for bridge defects involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more visual detection devices for bridge defects provided below can refer to the limitations of the visual detection method for bridge defects above, and will not be repeated here.

[0136] In an exemplary embodiment, Figure 8 As shown, a visual inspection device 800 for bridge defects is provided, comprising: a data acquisition module 802, a defect recognition module 804, a point cloud matching module 806, a result deduplication module 808 and a defect positioning module 810, wherein:

[0137] The data acquisition module 802 is used to obtain the time-series images of the bridge taken by the image acquisition device on the bridge inspection vehicle during its operation, the movement position information of the bridge inspection vehicle when the time-series images of the bridge are taken, and the laser point cloud data collected within the preset range of the bridge inspection vehicle.

[0138] The defect recognition module 804 is used to input the bridge time-series image into the defect recognition model for target detection, and obtain the defect detection result corresponding to each image frame in the bridge time-series image output by the defect recognition model.

[0139] The point cloud matching module 806 is used to project the laser point cloud data into the time-series image of the bridge according to the motion position information and the calibration parameters of the image acquisition device, so as to obtain the matching relationship between the laser point cloud data and the disease detection results.

[0140] The result deduplication module 808 is used to perform deduplication and merging processing on the disease detection results according to the matching relationship to obtain the processed target detection results.

[0141] The defect location module 810 is used to determine the location of the bridge defect based on the laser point cloud data that matches the target detection result.

[0142] In an exemplary embodiment, there are multiple image acquisition devices, and there are overlapping areas in the image acquisition areas of the multiple image acquisition devices. The point cloud matching module 806 is also used to project the laser point cloud data to the time-series image of the bridge acquired by each image acquisition device according to the motion position information and the calibration parameters of each image acquisition device, and obtain the matching relationship between the laser point cloud data and the disease detection results corresponding to each image acquisition device. The result deduplication module 808 is also used to perform deduplication and merging processing on the disease detection results acquired by multiple image acquisition devices when the disease detection results acquired by multiple image acquisition devices match the same laser point cloud data, so as to obtain the target detection result.

[0143] In an exemplary embodiment, the defect recognition module 804 includes: a feature extraction unit, which is used to perform feature extraction processing on each image frame in the bridge time series image through a defect recognition model to obtain image feature data corresponding to each image frame; a feature recognition unit, which is used to determine the defect area and defect category of the bridge defect in the bridge time series image according to the image feature data corresponding to each image frame through the defect recognition model; and a result output unit, which is used to output the defect area and defect category as a defect detection result through the defect recognition model.

[0144] In an exemplary embodiment, the defect location module 810 is also used to convert the defect area in the target detection result into laser point cloud data that matches the target detection result to obtain three-dimensional spatial coordinates corresponding to the defect area; store the defect category in the target detection result in the three-dimensional spatial coordinates and output it to obtain the bridge defect location.

[0145] In an exemplary embodiment, the calibration parameters include the calibration internal parameters and the calibration external parameters of the image acquisition device. The point cloud matching module 806 includes: a camera projection unit, which is used to project the laser point cloud data into the camera coordinate system where the image acquisition device is located according to the motion position information and the calibration external parameters of the image acquisition device, and obtain the camera projection data corresponding to the laser point cloud data; a pixel projection unit, which is used to convert the camera projection data into two-dimensional projection data in the pixel coordinate system of the image acquisition device according to the calibration internal parameters of the image acquisition device; a coordinate matching unit, which is used to generate a matching relationship between the laser point cloud data and the disease detection result when the pixel coordinates of the two-dimensional projection data and the disease detection result are consistent.

[0146] In an exemplary embodiment, the visual inspection device 800 for bridge defects further includes: a regular inspection module for regularly triggering a bridge inspection vehicle to perform an inspection task, wherein the inspection task is used to instruct the bridge inspection vehicle to run along the direction of the bridge.

[0147] Each module in the visual inspection device 800 for bridge defects can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0148] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as bridge time-series images, motion position information, laser point cloud data, model parameters of a disease recognition model, and disease detection results. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a visual detection method for bridge diseases is implemented.

[0149] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0150] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0151] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0152] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0153] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0154] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0155] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A visual inspection method for bridge defects, characterized in that: The method comprises: Acquire the time-series images of the bridge taken by the image acquisition device on the bridge inspection vehicle during the operation of the bridge inspection vehicle, the movement position information of the bridge inspection vehicle when the time-series images of the bridge are taken, and the laser point cloud data collected within the preset range of the bridge inspection vehicle; Inputting the bridge time-series image into a defect recognition model for target detection, and obtaining a defect detection result corresponding to each image frame in the bridge time-series image output by the defect recognition model; According to the motion position information and the calibration parameters of the image acquisition device, the laser point cloud data is projected into the time-series image of the bridge to obtain a matching relationship between the laser point cloud data and the disease detection result; Performing de-duplication and merging processing on the disease detection results according to the matching relationship to obtain processed target detection results; The bridge defect location is determined based on the laser point cloud data matching the target detection result.

2. The method according to claim 1, characterized in that There are multiple image acquisition devices, and image acquisition areas of the multiple image acquisition devices have overlapping areas; The step of projecting the laser point cloud data onto the time-series image of the bridge according to the motion position information and the calibration parameters of the image acquisition device to obtain a matching relationship between the laser point cloud data and the disease detection result includes: According to the motion position information and the calibration parameters of each of the image acquisition devices, the laser point cloud data is projected onto the time-series images of the bridge acquired by each of the image acquisition devices to obtain a matching relationship between the laser point cloud data and the corresponding disease detection results of each of the image acquisition devices; The de-duplication and merging processing is performed on the disease detection results according to the matching relationship to obtain the processed target detection results, including: When the disease detection results collected by multiple image acquisition devices match the same laser point cloud data, the disease detection results collected by the multiple image acquisition devices are de-overlapped and merged to obtain the target detection result.

3. The method according to claim 1, characterized in that The step of inputting the bridge time-series image into a defect recognition model for target detection, and obtaining a defect detection result corresponding to each image frame in the bridge time-series image output by the defect recognition model, comprises: Perform feature extraction processing on each of the image frames in the bridge time-series image by using the disease recognition model to obtain image feature data corresponding to each of the image frames; Determine, by the disease recognition model, the disease area and disease category of the bridge disease in the bridge time series image according to the image feature data corresponding to each image frame; The disease area and the disease category are output as the disease detection result through the disease recognition model.

4. The method according to claim 3, characterized in that The determining of the bridge defect location according to the laser point cloud data matching the target detection result includes: Converting the damaged area in the target detection result into laser point cloud data matching the target detection result to obtain three-dimensional spatial coordinates corresponding to the damaged area; The damage category in the target detection result is stored in the three-dimensional space coordinates and output to obtain the bridge damage location.

5. The method according to claim 1, characterized in that The calibration parameters include the calibration internal parameters and the calibration external parameters of the image acquisition device; The step of projecting the laser point cloud data onto the time-series image of the bridge according to the motion position information and the calibration parameters of the image acquisition device to obtain a matching relationship between the laser point cloud data and the disease detection result includes: According to the motion position information and the calibrated external parameters of the image acquisition device, the laser point cloud data is projected into a camera coordinate system where the image acquisition device is located, to obtain camera projection data corresponding to the laser point cloud data; According to the calibrated internal parameters of the image acquisition device, the camera projection data is converted into two-dimensional projection data in the pixel coordinate system of the image acquisition device; When the pixel coordinates of the two-dimensional projection data and the disease detection result are consistent, a matching relationship between the laser point cloud data and the disease detection result is generated.

6. The method according to claim 1, characterized in that The method further comprises: The bridge inspection vehicle is periodically triggered to perform an inspection task, and the inspection task is used to instruct the bridge inspection vehicle to run along the direction of the bridge.

7. A visual inspection device for bridge defects, characterized in that: The device comprises: A data acquisition module is used to acquire the time-series images of bridges taken by the image acquisition device on the bridge inspection vehicle during the operation of the bridge inspection vehicle, the movement position information of the bridge inspection vehicle when the time-series images of bridges are taken, and the laser point cloud data collected within a preset range of the bridge inspection vehicle; A defect recognition module, used for inputting the bridge time-series image into a defect recognition model for target detection, and obtaining a defect detection result corresponding to each image frame in the bridge time-series image output by the defect recognition model; A point cloud matching module, used to project the laser point cloud data into the bridge time-series image according to the motion position information and the calibration parameters of the image acquisition device, and obtain a matching relationship between the laser point cloud data and the disease detection result; A result deduplication module is used to perform deduplication and merging processing on the disease detection results according to the matching relationship to obtain a processed target detection result; The defect positioning module is used to determine the location of the bridge defect based on the laser point cloud data matching the target detection result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images