Expansion joint device defect detection method, system, device and readable storage medium

Abstract

A method, system, device and readable storage medium for detecting expansion joint device defects, which relate to the technical field of bridge health monitoring, include: when a first target detection model detects that a target expansion joint device exists in a first image captured by a front vehicle-mounted camera device, a target acceleration vibration signal is obtained based on the initial time when the target expansion joint device appears in the first image and the end time when the target expansion joint device appears in a second image captured by a rear vehicle-mounted camera device; the smoothness state of the target expansion joint device is determined according to the target acceleration vibration signal; defects are identified in the second image by a second target detection model to generate defect identification results; and the smoothness state and defect identification results are used as defect detection results of the target expansion joint device. Through this application, the defect detection efficiency of the expansion joint device is effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of bridge health monitoring, and in particular to a method, system, device and readable storage medium for detecting defects in an expansion joint device. Background Art

[0002] In order to meet the requirements of bridge deck deformation, expansion joint devices are usually installed between the two beam ends, between the beam ends and the abutments, or at the hinged position of the bridge. The expansion joint device is required to be able to freely expand and contract in both directions parallel and perpendicular to the bridge axis, be firm and reliable, and be smooth, without sudden jumps and noise when vehicles pass by. It must also be able to prevent rainwater and garbage and soil from infiltrating and blocking. In addition, it must be simple and convenient in terms of installation, inspection, maintenance, and removal of dirt. It should be understood that at the location where the expansion joint device is installed, the railings and the bridge deck pavement must be disconnected. The function of the bridge expansion joint device is to adjust the displacement and connection between the superstructure caused by vehicle loads and bridge construction materials. Once the expansion joint device of the skew bridge is damaged, it will seriously affect the speed, comfort and safety of driving, and even cause driving safety accidents.

[0003] At present, the detection of expansion joint device defects is mainly achieved through manual visual inspection, which is greatly disturbed by traffic and usually needs to be carried out at night, resulting in low detection efficiency. Therefore, how to effectively improve the detection efficiency of expansion joint device defects is an urgent problem to be solved. Summary of the invention

[0004] The present application provides a method, system, device and readable storage medium for detecting defects in an expansion joint device, which can solve the technical problem of low efficiency in detecting defects in an expansion joint device existing in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a method for detecting defects in an expansion joint device, the method comprising:

[0006] When the first target detection model detects that a target expansion joint device exists in a first image captured by a front vehicle-mounted camera device, a target acceleration vibration signal is acquired based on an initial time when the target expansion joint device appears in the first image and an end time when the target expansion joint device appears in a second image captured by a rear vehicle-mounted camera device;

[0007] determining a smoothness state of a target expansion joint device according to a target acceleration vibration signal;

[0008] Performing disease recognition on the second image by using a second object detection model to generate a disease recognition result;

[0009] The smoothness status and the defect identification results are used as the defect detection results of the target expansion joint device.

[0010] In combination with the first aspect, in one implementation, determining the smoothness state of the target expansion joint device according to the target acceleration vibration signal includes:

[0011] Determining whether the maximum acceleration vibration signal and the minimum acceleration vibration signal in the target acceleration vibration signal are both within a preset vibration signal threshold range;

[0012] If yes, the smoothness state of the target expansion joint device is determined to be smooth;

[0013] If not, it is determined that the smoothness state of the target expansion joint device is uneven.

[0014] In combination with the first aspect, in one embodiment, after the step of determining that the smoothness state of the target expansion joint device is uneven, the method further includes:

[0015] Acquire target point cloud data corresponding to the target expansion joint device;

[0016] The target height difference between the target expansion joint device and the road surface at the joint is calculated based on the target point cloud data, and the target height difference is used as the disease detection result of the target expansion joint device.

[0017] In combination with the first aspect, in one implementation, the method further includes:

[0018] Determine a first pixel positioning result of a target expansion joint device through a first target detection model and a first image;

[0019] Based on the first pixel positioning result, the horizontal width and the vertical width of the first image, the horizontal pixel width and the vertical pixel width of the first image, the distance between the lower edge of the first image and the vehicle-mounted positioning device, and the vehicle body positioning result corresponding to the area where the target expansion joint device is located, the device positioning result of the target expansion joint device is determined, and the device positioning result is used as the disease detection result of the target expansion joint device.

[0020] In combination with the first aspect, in one embodiment, the first pixel positioning result includes a first horizontal coordinate of the target expansion joint device and the first ordinate ,in, , ,(x 1 ,y 1 ) represents the pixel position of the upper left corner of the target expansion joint device in the first image, h represents the horizontal pixel width of the first image, z represents the vertical pixel width of the first image, Δx 1 Represents the relative width of the detection box in the first target detection model, Δy 1 Indicates the relative height of the detection box in the first target detection model;

[0021] The device positioning result includes a second horizontal coordinate of the target expansion joint device and the second ordinate ,in, , , (Q 1 ,W 1 ) represents the vehicle positioning result, X 2 represents the vertical width of the first image, H represents the horizontal width of the first image, X 1 Indicates the distance between the lower edge of the first image and the vehicle-mounted positioning device.

[0022] In combination with the first aspect, in one implementation, the performing disease recognition on the second image by using the second object detection model to generate a disease recognition result includes:

[0023] Performing feature extraction on the second image by using a second target detection model to determine a target expansion joint device image and a corresponding second pixel positioning result;

[0024] Based on the second pixel positioning result, the target expansion joint device image is subjected to defect recognition to generate a defect recognition result, wherein the defect recognition result includes a defect category and a defect positioning result.

[0025] In combination with the first aspect, in one embodiment, the second pixel positioning result includes a third horizontal coordinate of the target expansion joint device and the third ordinate ,in, , ,(x 2 ,y 2 ) represents the pixel position of the upper left corner of the target expansion joint device in the second image, f represents the horizontal pixel width of the second image, v represents the vertical pixel width, Δx 2 Represents the relative width of the detection box in the second target detection model, Δy 2 Represents the relative height Δy of the detection box in the second target detection model 2 ;

[0026] The defect location result includes the target horizontal coordinate of the defect in the target expansion joint device and the target ordinate ,in, , , (Q 2 ,W 2 ) represents the vehicle body positioning result corresponding to the area where the target expansion joint device is located, V represents the longitudinal width of the second image, F represents the transverse width, and X 3 Indicates the distance between the lower edge of the second image and the vehicle-mounted positioning device.

[0027] In a second aspect, an embodiment of the present application provides a system for detecting defects in an expansion joint device, the system comprising:

[0028] A first processing module, which is used to obtain a target acceleration vibration signal based on an initial time when the target expansion joint device appears in the first image taken by the front vehicle-mounted camera device and an end time when the target expansion joint device appears in the second image taken by the rear vehicle-mounted camera device when a target expansion joint device is detected in the first image taken by the front vehicle-mounted camera device through the first target detection model;

[0029] A second processing module, which is used to determine the smoothness state of the target expansion joint device according to the target acceleration vibration signal;

[0030] A third processing module, which is used to perform disease recognition on the second image through a second target detection model to generate a target disease type;

[0031] The disease output module is used to use the smoothness state and the target disease type as the disease detection result of the target expansion joint device.

[0032] In conjunction with the second aspect, in one implementation, the second processing module is specifically configured to:

[0033] Determining whether the maximum acceleration vibration signal and the minimum acceleration vibration signal in the target acceleration vibration signal are both within a preset vibration signal threshold range;

[0034] If yes, the smoothness state of the target expansion joint device is determined to be smooth;

[0035] If not, it is determined that the smoothness state of the target expansion joint device is uneven.

[0036] In conjunction with the second aspect, in one implementation, the second processing module is further configured to:

[0037] Acquire target point cloud data corresponding to the target expansion joint device;

[0038] The target height difference between the target expansion joint device and the road surface at the joint is calculated based on the target point cloud data, and the target height difference is used as the disease detection result of the target expansion joint device.

[0039] In conjunction with the second aspect, in one implementation, the first processing module is further configured to:

[0040] Determine a first pixel positioning result of a target expansion joint device through a first target detection model and a first image;

[0041] Based on the first pixel positioning result, the horizontal width and the vertical width of the first image, the horizontal pixel width and the vertical pixel width of the first image, the distance between the lower edge of the first image and the vehicle-mounted positioning device, and the vehicle body positioning result corresponding to the area where the target expansion joint device is located, the device positioning result of the target expansion joint device is determined, and the device positioning result is used as the disease detection result of the target expansion joint device.

[0042] In combination with the second aspect, in one embodiment, the first pixel positioning result includes a first horizontal coordinate of the target expansion joint device and the first ordinate ,in, , ,(x 1 ,y 1 ) represents the pixel position of the upper left corner of the target expansion joint device in the first image, h represents the horizontal pixel width of the first image, z represents the vertical pixel width of the first image, Δx 1 Represents the relative width of the detection box in the first target detection model, Δy 1 Indicates the relative height of the detection box in the first target detection model;

[0043] The device positioning result includes a second horizontal coordinate of the target expansion joint device and the second ordinate ,in, , , (Q 1 ,W 1 ) represents the vehicle positioning result, X 2 represents the vertical width of the first image, H represents the horizontal width of the first image, X 1 Indicates the distance between the lower edge of the first image and the vehicle-mounted positioning device.

[0044] In conjunction with the second aspect, in one implementation, the second processing module is specifically configured to:

[0045] Performing feature extraction on the second image by using a second target detection model to determine a target expansion joint device image and a corresponding second pixel positioning result;

[0046] Based on the second pixel positioning result, the target expansion joint device image is subjected to defect recognition to generate a defect recognition result, wherein the defect recognition result includes a defect category and a defect positioning result.

[0047] In combination with the second aspect, in one embodiment, the second pixel positioning result includes a third horizontal coordinate of the target expansion joint device and the third ordinate ,in, , ,(x2 ,y 2 ) represents the pixel position of the upper left corner of the target expansion joint device in the second image, f represents the horizontal pixel width of the second image, v represents the vertical pixel width, Δx 2 Represents the relative width of the detection box in the second target detection model, Δy 2 Represents the relative height Δy of the detection box in the second target detection model 2 ;

[0048] The defect location result includes the target horizontal coordinate of the defect in the target expansion joint device and the target ordinate ,in, , , (Q 2 ,W 2 ) represents the vehicle body positioning result corresponding to the area where the target expansion joint device is located, V represents the longitudinal width of the second image, F represents the transverse width, and X 3 Indicates the distance between the lower edge of the second image and the vehicle-mounted positioning device.

[0049] In a third aspect, an embodiment of the present application provides an expansion joint device defect detection device, the expansion joint device defect detection device comprising a processor, a memory, and an expansion joint device defect detection program stored in the memory and executable by the processor, wherein when the expansion joint device defect detection program is executed by the processor, the steps of the aforementioned expansion joint device defect detection method are implemented.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which is stored an expansion joint device defect detection program, wherein when the expansion joint device defect detection program is executed by a processor, the steps of the aforementioned expansion joint device defect detection method are implemented.

[0051] The beneficial effects brought by the technical solution provided by the embodiments of the present application include:

[0052] When the first target detection model detects that a target expansion joint device exists in a first image taken by a front vehicle-mounted camera device, a target acceleration vibration signal is obtained based on the initial time when the target expansion joint device appears in the first image and the end time when the target expansion joint device appears in a second image taken by a rear vehicle-mounted camera device, so as to determine the smoothness state of the target expansion joint device through the target acceleration vibration signal; then, the second image is subjected to defect recognition through the second target detection model to generate a defect recognition result, and the smoothness state and the defect recognition result are used as the defect detection result of the target expansion joint device, thereby realizing automatic recognition and detection of driving smoothness and apparent defects of the expansion joint device, and effectively improving the defect detection efficiency of the expansion joint device. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a schematic diagram of a flow chart of an embodiment of a method for detecting defects in expansion joint devices of the present application;

[0054] Figure 2 This is a schematic diagram of an expansion joint device involved in the embodiment of the present application;

[0055] Figure 3 A schematic diagram of data calibration involved in the embodiment of the present application;

[0056] Figure 4 This is a functional module diagram of an embodiment of the expansion joint device disease detection system of the present application;

[0057] Figure 5 This is a schematic diagram of the hardware structure of the expansion joint device defect detection equipment involved in the embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0060] In a first aspect, an embodiment of the present application provides a method for detecting defects in an expansion joint device.

[0061] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of the expansion joint device defect detection method of the present application. Figure 1 As shown, the expansion joint device disease detection method includes:

[0062] Step S10: When the first target detection model detects that a target expansion joint device exists in the first image taken by the front vehicle-mounted camera device, a target acceleration vibration signal is obtained based on the initial time when the target expansion joint device appears in the first image and the end time when the target expansion joint device appears in the second image taken by the rear vehicle-mounted camera device.

[0063] Exemplarily, in this embodiment, the expansion joint device defect detection is achieved by a front vehicle-mounted camera device arranged at the front side of the roof (i.e., the roof position close to the front direction of the vehicle), a rear vehicle-mounted camera device arranged at the rear of the vehicle, and a combination of a 3D camera, an acceleration sensor, and a GPS positioning and inertial navigation system.

[0064] Among them, the front-mounted vehicle camera device can be preferably a landscape camera for photographing natural landscapes, urban landscapes and other environmental landscapes, which is mainly used to photograph the landscape in the direction of the vehicle's advance at a certain angle, so as to determine whether there is an expansion joint device and its location through the image it photographs. It should be noted that the landscape camera can be an encoder-triggered camera, and it is necessary to ensure that the continuous frame images it collects are such that the lower edge of the P-th frame image is the upper edge of the P-1-th frame image, thereby avoiding the same part of the expansion joint device from appearing repeatedly in the continuous frame images; the rear-mounted vehicle camera device can be preferably an encoder-triggered camera, which is mainly used to photograph at a certain angle or perpendicular to the road surface, so as to obtain the apparent information of the expansion joint device image through the image it photographs; the 3D camera can be preferably a line laser 3D camera, which is mainly used to obtain the point cloud information of the expansion joint device; the acceleration sensor is used to obtain continuous acceleration vibration signals near the expansion joint device, so as to judge the smoothness of the expansion joint device; the GPS positioning and inertial navigation system are used to accurately locate the vehicle position to determine the vehicle's position on the road.

[0065] It should be understood that the first target detection model and the second target detection model in this embodiment may be the same or different. For example, both may be Yolov9 models, Yolov10 models, or other models for target detection. Therefore, which model to use and whether the two are the same can be determined according to actual needs, and are not limited here. Since the methods and principles for implementing target detection through Yolov9 models, Yolov10 models, etc. are common knowledge in the field, they will not be repeated here. However, since MobileNetV4 is a new lightweight backbone network, it has the characteristics of fast detection speed and can perform detection tasks using fewer detection resources. Therefore, in order to improve the lightweight of the model, this embodiment can preferentially replace the backbone network of the target detection model with MobileNetV4.

[0066] In addition, the second target detection model in this embodiment is mainly used for the recognition of apparent diseases, so the convolution module Covn of the RepNCSPELAN4 module passed into the second target detection model can be replaced with a lightweight iRMB inverted residual block to improve the accuracy of the model. Specifically, since the iRMB inverted residual block adds an attention mechanism, which can help the network focus on important features during the learning process, thereby improving the performance of the network, adding attention in front of the RepNCSPELAN4 module can effectively improve the module learning ability, thereby improving the accuracy of the model. Among them, during the training stage of the first target detection model and the second target detection model, the widely collected expansion joint device detection images can be data labeled, and the model training can be carried out using 500 epochs and the Mosaic data enhancement technology.

[0067] In this embodiment, the first image captured by the front vehicle-mounted camera device is transmitted in real time to the first target detection model for identification and detection of the expansion joint device; if the target expansion joint device (such as Figure 2 As shown), the target acceleration vibration signal is obtained according to the initial time when the target expansion joint device appears in the first image and the termination time when it appears in the second image taken by the rear vehicle-mounted camera device; for example, the initial time when the target expansion joint device appears in the first image is T1 and the termination time when it appears in the second image is T2, then the target acceleration vibration signal corresponding to the time range of [T1, T2] is intercepted from the continuous acceleration vibration signal collected by the acceleration sensor, that is, the time corresponding to the initial signal of the target acceleration vibration signal is T1, and the time corresponding to its termination signal is T2.

[0068] Step S20: determining the smoothness state of the target expansion joint device according to the target acceleration vibration signal.

[0069] Exemplarily, in this embodiment, whether the target expansion joint device is smooth is judged by the vibration condition of the target acceleration vibration signal, thereby determining its smoothness state.

[0070] Step S30: performing disease recognition on the second image using a second object detection model to generate a disease recognition result.

[0071] Exemplarily, in this embodiment, while determining the smoothness state of the target expansion joint device, the apparent defects of the target expansion joint device will also be identified and detected. Specifically, the second image captured by the rear-mounted vehicle camera is input into the second target detection model for defect identification to obtain a defect identification result including a defect category and a defect location result; wherein the defect types include but are not limited to cracks in the concrete anchoring area, damage to the anchoring area, damage to the sealing rubber strip, and loss of the expansion joint anchoring screw. It should be noted that the method and principle of how to implement defect identification and detection through the target detection model is common knowledge in the art, and for the sake of brevity of description, it will not be repeated here.

[0072] Step S40: Using the smoothness state and the defect identification result as the defect detection result of the target expansion joint device.

[0073] For example, in this embodiment, after the smoothness state and the defect recognition result of the target expansion joint device are determined, the smoothness state and the defect recognition result can be used together as the defect detection result of the target expansion joint device. It can be seen that this embodiment realizes the automatic recognition and detection of driving smoothness and apparent defects of the expansion joint device, which not only effectively improves the defect detection efficiency of the expansion joint device, but also improves the comprehensiveness of the defect detection of the expansion joint device.

[0074] Further, in one embodiment, determining the smoothness state of the target expansion joint device according to the target acceleration vibration signal includes:

[0075] Determining whether the maximum acceleration vibration signal and the minimum acceleration vibration signal in the target acceleration vibration signal are both within a preset vibration signal threshold range;

[0076] If yes, the smoothness state of the target expansion joint device is determined to be smooth;

[0077] If not, it is determined that the smoothness state of the target expansion joint device is uneven.

[0078] For example, it should be noted that the specific value setting of the vibration signal threshold range can be determined according to actual needs and is not limited here. In this embodiment, the driving smoothness of the target expansion joint device can be determined by whether the maximum acceleration vibration signal and the minimum acceleration vibration signal in the target acceleration vibration signal are both within the vibration signal threshold range; specifically, if the maximum acceleration vibration signal and the minimum acceleration vibration signal are both within the vibration signal threshold range, the smoothness state of the target expansion joint device is determined to be smooth; and as long as there is one of the maximum acceleration vibration signal and the minimum acceleration vibration signal that is not within the vibration signal threshold range, it indicates that problems such as bridge head jumping may occur, and the smoothness state of the target expansion joint device is determined to be uneven.

[0079] Further, in one embodiment, after the step of determining that the smoothness state of the target expansion joint device is uneven, the method further includes:

[0080] Acquire target point cloud data corresponding to the target expansion joint device;

[0081] The target height difference between the target expansion joint device and the road surface at the joint is calculated based on the target point cloud data, and the target height difference is used as the disease detection result of the target expansion joint device.

[0082] Exemplarily, in this embodiment, if the target expansion joint device is detected to be uneven, the 3D camera can be controlled to scan the target expansion joint device to obtain the surface point cloud data of the target expansion joint device, so as to further quantify the uneven state according to the surface point cloud data collected by the 3D camera, so that the operator can take effective measures to solve the unevenness problem based on the quantification result. Specifically, since there is an obvious crack between the anchoring area of ​​the target expansion joint device and the asphalt concrete pavement, the crack can be used as a segmentation feature of the target expansion joint device and the pavement, so when calculating the expansion joint height difference, the average height of the target expansion joint device at the joint and the average height of the pavement at the joint are calculated based on the target point cloud data corresponding to the target expansion joint device obtained by the 3D camera, and then the target height difference between the two average heights is calculated, and the target height difference can further quantitatively reflect the smoothness between the target expansion joint device and the pavement.

[0083] Furthermore, in one embodiment, the method further includes:

[0084] Determine a first pixel positioning result of a target expansion joint device through a first target detection model and a first image;

[0085] Determine a device positioning result of the target expansion joint device based on the first pixel positioning result, the horizontal width and the vertical width of the first image, the horizontal pixel width and the vertical pixel width of the first image, the distance between the lower edge of the first image and the vehicle positioning device, and the vehicle body positioning result corresponding to the area where the target expansion joint device is located, and use the device positioning result as the disease detection result of the target expansion joint device;

[0086] The first pixel positioning result includes the first horizontal coordinate of the target expansion joint device. and the first ordinate ,in, , ,(x 1 ,y 1 ) represents the pixel position of the upper left corner of the target expansion joint device in the first image, h represents the horizontal pixel width of the first image, z represents the vertical pixel width of the first image, Δx 1Represents the relative width of the detection box in the first target detection model, Δy 1 Indicates the relative height of the detection box in the first target detection model;

[0087] The device positioning result includes a second horizontal coordinate of the target expansion joint device and the second ordinate ,in, , , (Q 1 ,W 1 ) represents the vehicle positioning result, X 2 represents the longitudinal width of the first image, H represents the lateral width of the first image, and X represents the distance between the lower edge of the first image and the vehicle positioning device. 1 . .

[0088] Exemplarily, in this embodiment, in the process of shooting the vehicle's moving video by the front-mounted vehicle camera device, not only the first target detection model is used to identify in real time the initial time when the target expansion joint device appears in the first image, but also the position of the target expansion joint device in the first image is identified, so as to trigger the GPS positioning and inertial navigation system to locate the area where the target expansion joint device is located, thereby achieving the positioning of the target expansion joint device.

[0089] It should be understood that the recognition accuracy of the front-mounted vehicle camera relative to the expansion joint device determines the quality of the collected data and the computing power consumption. Therefore, in order to ensure the recognition accuracy of the front-mounted vehicle camera, after the front-mounted vehicle camera is installed on the top of the vehicle, the front-mounted vehicle camera is controlled to use a fixed-focus lens and use the Zhang Zhengyou camera calibration method to correct the camera image distortion. Specifically, taking the front-mounted vehicle camera as a landscape camera as an example, the landscape camera is controlled to shoot at a certain angle toward the road surface, and the image is cropped and calibrated based on the captured image; wherein, see Figure 3 As shown, assuming that the field of view length of the landscape camera is X 2 (For example, X 2 =4m), that is, the lower edge of the captured image is determined as the tangent point between the image and the front of the vehicle, the upper edge is determined by taking the lower edge of the image as the reference of the camera field of view, and the captured image is symmetrically distributed around the vehicle axis, then the actual longitudinal width of the image captured is X 2 , the horizontal width is H and the horizontal pixel width is h, the vertical pixel width is z; then the lower edge of the image is used as the measuring end, and the distance X between the lower edge of the image and the bottom of the vehicle positioning device is measured 1 to complete the calibration.

[0090] In this embodiment, after the first image captured by the landscape camera is input into the first target detection model for target detection, a detection frame is output; it can be understood that the output result of the detection frame is usually the pixel position (x 1 ,y 1 ), but the pixel position (x 1 ,y 1 ) cannot accurately represent the real pixel location of the target expansion joint device; and in order to obtain a more accurate pixel location result, this embodiment will be based on the pixel position (x 1 ,y 1 ), the horizontal pixel width h and the vertical pixel width z of the first image, and the relative width Δx of the detection box in the first target detection model 1 and relative height Δy 1 To accurately determine the first pixel positioning result of the target expansion joint device : , .

[0091] After the target expansion joint device is identified in the first image, the GPS positioning and inertial navigation system will be triggered to perform RTK (Real-time kinematic) positioning on the area where the target expansion joint is located to obtain a high-precision vehicle body positioning result (Q 1 ,W 1 ); then combined with the first pixel positioning result , the vertical width of the first image X 2 , horizontal width H, horizontal pixel width h and vertical pixel width z, distance X between the lower edge of the first image and the vehicle-mounted positioning device 1 And the vehicle positioning result (Q 1 ,W 1 ) Determine the actual positioning result of the target expansion joint device, that is, the device positioning result : , ; Finally, the device positioning result is used as one of the disease detection results of the target expansion joint device, that is, the position of the target expansion joint device can be accurately and quickly found through the device positioning result.

[0092] Furthermore, in one embodiment, the performing disease recognition on the second image by using the second object detection model to generate a disease recognition result includes:

[0093] Performing feature extraction on the second image by using a second target detection model to determine a target expansion joint device image and a corresponding second pixel positioning result;

[0094] Performing defect recognition on the target expansion joint device image based on the second pixel positioning result to generate a defect recognition result, wherein the defect recognition result includes a defect category and a defect positioning result;

[0095] The second pixel positioning result includes the third horizontal coordinate of the target expansion joint device and the third ordinate ,in, , ,(x 2 ,y 2 ) represents the pixel position of the upper left corner of the target expansion joint device in the second image, f represents the horizontal pixel width of the second image, v represents the vertical pixel width, Δx 2 Represents the relative width of the detection box in the second target detection model, Δy 2 Represents the relative height Δy of the detection box in the second target detection model 2 ;

[0096] The defect location result includes the target horizontal coordinate of the defect in the target expansion joint device and the target ordinate ,in, , , (Q 2 ,W 2 ) represents the vehicle body positioning result corresponding to the area where the target expansion joint device is located, V represents the longitudinal width of the second image, F represents the transverse width, and X 3 Indicates the distance between the lower edge of the second image and the vehicle-mounted positioning device.

[0097] For example, it should be understood that the recognition accuracy of the rear-mounted vehicle camera device relative to the location of the expansion joint device disease determines the quality of the collected data and the computing power consumption. Therefore, in order to ensure the recognition accuracy of the rear-mounted vehicle camera device, after the rear-mounted vehicle camera device is installed at the rear of the vehicle, the rear-mounted vehicle camera device is controlled to use a fixed-focus lens and use the Zhang Zhengyou camera calibration method to correct the camera image distortion. For example, if the rear-mounted vehicle camera device is an encoder-triggered camera, the encoder-triggered camera is controlled to shoot at a certain angle or perpendicular to the road surface, and the image is cropped and calibrated based on the captured image; wherein, see Figure 3 As shown, the upper edge is determined by taking the lower edge of the image as the reference of the camera field of view, and the collected images are symmetrically distributed with the vehicle axis as the center. Then, the actual collected image has a longitudinal width of V, a lateral width of F, a lateral pixel width of f, and a longitudinal pixel width of v; then the lower edge of the image is taken as the measuring end, and the distance X between the lower edge of the image and the bottom of the vehicle positioning device is measured. 3 (like Figure 3 As shown in the figure), the calibration is completed; and then the second image is acquired through the encoder-triggered camera.

[0098] In this embodiment, the detection of apparent diseases is achieved through a two-stage method to effectively improve the accuracy of disease detection. Specifically, in the first stage: the second image containing the target expansion joint device taken by the rear vehicle-mounted camera device is imported into the second target detection model for feature extraction to separate the target expansion joint device from the second image, that is, for the image in which the target expansion joint device is identified, the recognition frame information output by the second target detection model is used to separate the target expansion joint device from the original image (that is, retain the target expansion joint device), and at the same time inherit the position information of the target expansion joint device to obtain the target expansion joint device image, thereby preparing for the next stage of expansion joint disease extraction; in the second stage: according to the characteristics of expansion joint device diseases (such as cracks in the concrete anchoring area, damage to the anchoring area, damage to the sealing rubber strip, loss of the expansion joint anchoring screw, etc.), the target detection algorithm is improved, that is, the convolution module Covn passed into RepNCSPELAN4 is replaced with the spatial attention mechanism, and all the Covn convolution modules in the model are replaced with AKConv for model reconstruction, so as to perform disease recognition and pixel-level positioning on the target expansion joint device image based on the reconstructed model to determine the disease category and disease location.

[0099] The location of the defect is determined by the center point of the defect, and the lane information of the defect image is given. Specifically, the second target detection model performs defect detection on the target expansion joint device image to output a detection frame; this embodiment will be based on the pixel position (x 2 ,y 2 ), the horizontal pixel width f and the vertical pixel width v of the second image, and the relative width Δx of the detection box in the second target detection model 2 and relative height Δy 2 To accurately determine the second pixel positioning result of the target expansion joint device : , .

[0100] After the target expansion joint device is identified in the second image, the GPS positioning and inertial navigation system will be triggered to perform RTK positioning on the area where the target expansion joint is located to obtain a high-precision vehicle body positioning result (Q 2 ,W 2 ); then combined with the second pixel positioning result , the longitudinal width V, the transverse width F, the transverse pixel width f and the longitudinal pixel width v of the second image, the distance X between the lower edge of the second image and the vehicle-mounted positioning device 3 And the vehicle positioning result (Q 2 ,W 2) Determine the actual location result of the defect in the target expansion joint device, that is, the defect location result : , , through the defect positioning result, the location of the defect in the target expansion joint device can be found accurately and quickly.

[0101] In a second aspect, an embodiment of the present application also provides a system for detecting defects in an expansion joint device.

[0102] In one embodiment, referring to Figure 4 , Figure 4 This is a functional module diagram of an embodiment of the expansion joint device disease detection system of the present application. Figure 4 As shown, the expansion joint device disease detection system includes:

[0103] A first processing module, which is used to obtain a target acceleration vibration signal based on an initial time when the target expansion joint device appears in the first image taken by the front vehicle-mounted camera device and an end time when the target expansion joint device appears in the second image taken by the rear vehicle-mounted camera device when a target expansion joint device is detected in the first image taken by the front vehicle-mounted camera device through the first target detection model;

[0104] A second processing module, which is used to determine the smoothness state of the target expansion joint device according to the target acceleration vibration signal;

[0105] A third processing module, which is used to perform disease recognition on the second image through a second target detection model to generate a target disease type;

[0106] The disease output module is used to use the smoothness state and the target disease type as the disease detection result of the target expansion joint device.

[0107] Furthermore, in one embodiment, the second processing module is specifically configured to:

[0108] Determining whether the maximum acceleration vibration signal and the minimum acceleration vibration signal in the target acceleration vibration signal are both within a preset vibration signal threshold range;

[0109] If yes, the smoothness state of the target expansion joint device is determined to be smooth;

[0110] If not, it is determined that the smoothness state of the target expansion joint device is uneven.

[0111] Furthermore, in one embodiment, the second processing module is further configured to:

[0112] Acquire target point cloud data corresponding to the target expansion joint device;

[0113] The target height difference between the target expansion joint device and the road surface at the joint is calculated based on the target point cloud data, and the target height difference is used as the disease detection result of the target expansion joint device.

[0114] Furthermore, in one embodiment, the first processing module is further configured to:

[0115] Determine a first pixel positioning result of a target expansion joint device through a first target detection model and a first image;

[0116] Based on the first pixel positioning result, the horizontal width and the vertical width of the first image, the horizontal pixel width and the vertical pixel width of the first image, the distance between the lower edge of the first image and the vehicle-mounted positioning device, and the vehicle body positioning result corresponding to the area where the target expansion joint device is located, the device positioning result of the target expansion joint device is determined, and the device positioning result is used as the disease detection result of the target expansion joint device.

[0117] Furthermore, in one embodiment, the first pixel positioning result includes a first horizontal coordinate of the target expansion joint device. and the first ordinate ,in, , ,(x 1 ,y 1 ) represents the pixel position of the upper left corner of the target expansion joint device in the first image, h represents the horizontal pixel width of the first image, z represents the vertical pixel width of the first image, Δx 1 Represents the relative width of the detection box in the first target detection model, Δy 1 Indicates the relative height of the detection box in the first target detection model;

[0118] The device positioning result includes a second horizontal coordinate of the target expansion joint device and the second ordinate ,in, , , (Q 1 ,W 1 ) represents the vehicle positioning result, X 2 represents the vertical width of the first image, H represents the horizontal width of the first image, X 1 Indicates the distance between the lower edge of the first image and the vehicle-mounted positioning device.

[0119] Furthermore, in one embodiment, the second processing module is specifically configured to:

[0120] Performing feature extraction on the second image by using a second target detection model to determine a target expansion joint device image and a corresponding second pixel positioning result;

[0121] Based on the second pixel positioning result, the target expansion joint device image is subjected to defect recognition to generate a defect recognition result, wherein the defect recognition result includes a defect category and a defect positioning result.

[0122] Furthermore, in one embodiment, the second pixel positioning result includes a third horizontal coordinate of the target expansion joint device. and the third ordinate ,in, , ,(x 2 ,y 2 ) represents the pixel position of the upper left corner of the target expansion joint device in the second image, f represents the horizontal pixel width of the second image, v represents the vertical pixel width, Δx 2 Represents the relative width of the detection box in the second target detection model, Δy 2 Represents the relative height Δy of the detection box in the second target detection model 2 ;

[0123] The defect location result includes the target horizontal coordinate of the defect in the target expansion joint device and the target ordinate ,in, , , (Q 2 ,W 2 ) represents the vehicle body positioning result corresponding to the area where the target expansion joint device is located, V represents the longitudinal width of the second image, F represents the transverse width, and X 3 Indicates the distance between the lower edge of the second image and the vehicle-mounted positioning device.

[0124] Among them, the functional implementation of each module in the above-mentioned expansion joint device defect detection system corresponds to the various steps in the above-mentioned expansion joint device defect detection embodiment, and its functions and implementation processes will not be repeated here one by one.

[0125] In a third aspect, an embodiment of the present application provides an expansion joint device defect detection device, which may be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0126] Reference Figure 5 , Figure 5 Schematic diagram of the hardware structure of the expansion joint device defect detection device involved in the embodiment of the present application. In the embodiment of the present application, the expansion joint device defect detection device may include a processor, a memory, a communication interface and a communication bus.

[0127] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0128] The communication interface includes input / output (I / O) interface, physical interface and logical interface, etc., which are used to realize the interconnection of devices inside the expansion joint device defect detection device, and the interface used to realize the interconnection of the expansion joint device defect detection device with other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display (Display), a keyboard (Keyboard), etc.

[0129] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0130] The processor may be a general-purpose processor, which may call the expansion joint device defect detection program stored in the memory and execute the expansion joint device defect detection method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the expansion joint device defect detection program is called may refer to the various embodiments of the expansion joint device defect detection method of the present application, which will not be described in detail here.

[0131] Those skilled in the art will understand that Figure 5 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0132] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.

[0133] The readable storage medium of the present application stores an expansion joint device disease detection program, wherein when the expansion joint device disease detection program is executed by the processor, the steps of the expansion joint device disease detection method as described above are implemented.

[0134] Among them, the method implemented when the expansion joint device defect detection program is executed can refer to the various embodiments of the expansion joint device defect detection method of the present application, and will not be repeated here.

[0135] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0136] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0137] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0138] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0139] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0141] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for detecting defects in an expansion joint device, characterized in that: The expansion joint device disease detection method comprises: When the first target detection model detects that a target expansion joint device exists in a first image captured by a front vehicle-mounted camera device, a target acceleration vibration signal is acquired based on an initial time when the target expansion joint device appears in the first image and an end time when the target expansion joint device appears in a second image captured by a rear vehicle-mounted camera device; determining a smoothness state of a target expansion joint device according to a target acceleration vibration signal; Performing disease recognition on the second image by using a second object detection model to generate a disease recognition result; The smoothness state and the defect identification results are used as the defect detection results of the target expansion joint device; Wherein, a first pixel positioning result of a target expansion joint device is determined by a first target detection model and a first image; a device positioning result of the target expansion joint device is determined based on the first pixel positioning result, a horizontal width and a vertical width of the first image, a horizontal pixel width and a vertical pixel width of the first image, a distance between a lower edge of the first image and a vehicle-mounted positioning device, and a vehicle body positioning result corresponding to an area where the target expansion joint device is located, and the device positioning result is used as a disease detection result of the target expansion joint device; The first pixel positioning result includes a first horizontal coordinate of the target expansion joint device and the first ordinate ,in, , , (x1, y1) represents the pixel position of the upper left corner of the target expansion joint device in the first image, h represents the horizontal pixel width of the first image, z represents the vertical pixel width of the first image, Δx1 represents the relative width of the detection box in the first target detection model, and Δy1 represents the relative height of the detection box in the first target detection model; The device positioning result includes a second horizontal coordinate of the target expansion joint device and the second ordinate ,in, , , (Q1, W1) represents the vehicle positioning result, X2 represents the longitudinal width of the first image, H represents the lateral width of the first image, and X1 represents the distance between the lower edge of the first image and the vehicle-mounted positioning device.

2. The expansion joint device defect detection method according to claim 1, characterized in that: The step of determining the smoothness state of the target expansion joint device according to the target acceleration vibration signal includes: Determining whether the maximum acceleration vibration signal and the minimum acceleration vibration signal in the target acceleration vibration signal are both within a preset vibration signal threshold range; If yes, the smoothness state of the target expansion joint device is determined to be smooth; If not, it is determined that the smoothness state of the target expansion joint device is uneven.

3. The expansion joint device defect detection method according to claim 2, characterized in that: After the step of determining that the smoothness state of the target expansion joint device is uneven, the method further includes: Acquire target point cloud data corresponding to the target expansion joint device; The target height difference between the target expansion joint device and the road surface at the joint is calculated based on the target point cloud data, and the target height difference is used as the disease detection result of the target expansion joint device.

4. The expansion joint device defect detection method according to claim 1, characterized in that: The performing disease recognition on the second image by using the second object detection model to generate a disease recognition result includes: Performing feature extraction on the second image by using a second target detection model to determine a target expansion joint device image and a corresponding second pixel positioning result; Based on the second pixel positioning result, the target expansion joint device image is subjected to defect recognition to generate a defect recognition result, wherein the defect recognition result includes a defect category and a defect positioning result.

5. The expansion joint device defect detection method according to claim 4, characterized in that: The second pixel positioning result includes the third horizontal coordinate of the target expansion joint device and the third ordinate ,in, , , (x2, y2) represents the pixel position of the upper left corner of the target expansion joint device in the second image, f represents the horizontal pixel width of the second image, v represents the vertical pixel width, Δx2 represents the relative width of the detection box in the second target detection model, and Δy2 represents the relative height Δy2 of the detection box in the second target detection model; The defect location result includes the target horizontal coordinate of the defect in the target expansion joint device and the target ordinate ,in, , , (Q2, W2) represents the vehicle body positioning result corresponding to the area where the target expansion joint device is located, V represents the longitudinal width of the second image, F represents the transverse width, and X3 represents the distance between the lower edge of the second image and the vehicle-mounted positioning device.

6. A system for detecting defects in expansion joints, characterized in that: The expansion joint device defect detection system comprises: A first processing module, which is used to obtain a target acceleration vibration signal based on an initial time when the target expansion joint device appears in the first image taken by the front vehicle-mounted camera device and an end time when the target expansion joint device appears in the second image taken by the rear vehicle-mounted camera device when a target expansion joint device is detected in the first image taken by the front vehicle-mounted camera device through the first target detection model; A second processing module, which is used to determine the smoothness state of the target expansion joint device according to the target acceleration vibration signal; A third processing module, which is used to perform disease recognition on the second image through a second target detection model to generate a target disease type; A disease output module, which is used to use the smoothness state and the target disease type as the disease detection result of the target expansion joint device; Wherein, the first processing module is further used to: determine a first pixel positioning result of a target expansion joint device through a first target detection model and a first image; determine a device positioning result of the target expansion joint device based on the first pixel positioning result, a horizontal width and a vertical width of the first image, a horizontal pixel width and a vertical pixel width of the first image, a distance between a lower edge of the first image and a vehicle positioning device, and a vehicle body positioning result corresponding to an area where the target expansion joint device is located, and use the device positioning result as a disease detection result of the target expansion joint device; The first pixel positioning result includes a first horizontal coordinate of the target expansion joint device and the first ordinate ,in, , , (x1, y1) represents the pixel position of the upper left corner of the target expansion joint device in the first image, h represents the horizontal pixel width of the first image, z represents the vertical pixel width of the first image, Δx1 represents the relative width of the detection box in the first target detection model, and Δy1 represents the relative height of the detection box in the first target detection model; The device positioning result includes a second horizontal coordinate of the target expansion joint device and the second ordinate ,in, , , (Q1, W1) represents the vehicle positioning result, X2 represents the longitudinal width of the first image, H represents the lateral width of the first image, and X1 represents the distance between the lower edge of the first image and the vehicle-mounted positioning device.

7. An expansion joint device defect detection device, characterized in that: The expansion joint device defect detection device comprises a processor, a memory, and an expansion joint device defect detection program stored in the memory and executable by the processor, wherein when the expansion joint device defect detection program is executed by the processor, the steps of the expansion joint device defect detection method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an expansion joint device defect detection program, wherein when the expansion joint device defect detection program is executed by the processor, the steps of the expansion joint device defect detection method according to any one of claims 1 to 5 are implemented.

Images