Detection method and device for damaged expansion joint and storage medium

Abstract

The invention relates to the technical field of image processing, and discloses a damaged expansion joint detection method and device and a storage medium, the method is applied to computing equipment, the method comprises the following steps: determining at least one changed area pattern spot based on a real-time image and a standard image, the real-time image characterizing the real-time state of an expansion joint in a target road, and the standard image characterizing the real-time state of the expansion joint in the target road; the standard image represents that the expansion joint in the target road is in an intact state, the real-time image and the standard image are in the same coordinate system, removing vehicle information from each changed area pattern spot to obtain at least one suspicious damage area, extracting an external rectangular frame of each suspicious damage area, intercepting the real-time image by using the external rectangular frame, and determining the expansion joint in the target road according to the intercepted real-time image. According to the method for detecting the expansion joint, the intercepted area is detected, the position of the expansion joint with the detection result being in the damaged state is confirmed based on the road information and the lane line information, the method for detecting the expansion joint does not need to depend on complex external equipment, the detection process is convenient, and the detection accuracy is high.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and provides a method, a device and a storage medium for detecting a damaged expansion joint. Background Art

[0002] An expansion joint is a gap set on roads such as bridge decks and tunnel entrances and exits. The main function of an expansion joint is to allow deformation between bridge superstructures and road structures in different environments due to temperature changes, concrete shrinkage and other factors, and to ensure smooth passage of vehicles. However, expansion joints can be damaged due to improper design, construction quality, improper material selection, overloaded vehicles and other factors. Once an expansion joint is damaged, it will not only affect the safety and durability of the bridge, but also pose a serious threat to driving safety. Therefore, it is necessary to provide timely warning when an expansion joint is damaged.

[0003] However, in the current process of detecting expansion joints, a large number of external devices are needed to obtain the real-time status of the expansion joints in the road, and to perform damage detection on them again. The detection process is complicated and the detection accuracy is poor. Summary of the invention

[0004] The embodiments of the present application provide a method, device and storage medium for detecting a damaged expansion joint, so as to improve the detection efficiency and accuracy of the expansion joint.

[0005] The specific technical solutions provided by this application are as follows: In a first aspect, an embodiment of the present application provides a method for detecting a damaged expansion joint, which is applied to a computing device, and the method includes: Determine at least one area patch with changes based on the real-time image and the standard image, wherein the real-time image represents the real-time state of the expansion joint in the target road, the standard image represents that the expansion joint in the target road is in a good state, and the real-time image and the standard image are in the same coordinate system; Remove vehicle information from each area with changes, obtain at least one suspected damaged area, and extract the circumscribed rectangular frame of each suspected damaged area; The real-time image is captured using an external rectangular frame, the captured area is detected, and the position of the expansion joint that is detected to be damaged is confirmed based on the road information and lane line information, wherein the vehicle information, road information and lane line information are determined based on the real-time image.

[0006] In some possible implementations provided by the present application, before determining at least one area pattern spot with a change based on the real-time image and the standard image, the method further includes: Determine whether the real-time image and the standard image are in the same coordinate system; If not, the real-time image and the standard image are matched to the same coordinate system.

[0007] In some possible implementations provided in the present application, matching the real-time image and the standard image to the same coordinate system includes: Extracting a first feature point from the real-time image using a feature point extraction algorithm, and extracting a second feature point from the standard image using a feature point extraction algorithm; Determining a homography matrix based on the first feature point and the second feature point; The standard image is transformed into the coordinate system of the real-time image based on the homography matrix.

[0008] In some possible implementations provided in the present application, determining at least one area pattern with changes based on the real-time image and the standard image includes: Inputting the real-time image and the standard image into the first encoder and the second encoder of the pre-trained change detection network through different first input ports respectively; Using a first encoder to encode the real-time image to obtain a first encoding result, using a second encoder to encode the standard image to obtain a second encoding result, and fusing the first encoding result and the second encoding result to obtain a fused encoding result; Inputting the fused encoding result into the decoder of the change detection network through the second input port; Decoding the fusion coding result using a decoder to obtain a first decoding result; Based on the first decoding result, change information of the real-time image compared with the standard image is determined, and based on the change information, at least one area pattern with change is determined.

[0009] In some possible implementations provided in this application, vehicle information, road information, and lane line information are determined based on real-time images in the following ways: The real-time image is input into a pre-trained object segmentation network for inference. The object segmentation network is obtained by training the semantic segmentation network HRNet[2] based on a set of sample images labeled with vehicles, roads, and lane lines. Based on the inference results, the vehicle information, road information and lane line information in the real-time image are determined.

[0010] In some possible implementations provided in the present application, vehicle information is removed from each area with changes to obtain at least one suspected damaged area, and a circumscribed rectangular frame of each suspected damaged area is extracted, including: Filter out the regional spots whose areas are smaller than the preset area from the regional spots with changes, and obtain at least one valid regional spot; Searching for a target area exceeding a preset threshold in the effective area spots, wherein the preset threshold represents a threshold corresponding to the area intersection-combination ratio between the area spots with changes and the spots corresponding to the vehicle information; The found target area is eliminated from each valid area patch to obtain at least one suspected damaged area, and the circumscribed rectangular frame of each suspected damaged area is extracted.

[0011] In some possible implementations provided in the present application, a real-time image is captured using an external rectangular frame, the captured area is detected, and the location of the expansion joint with a detection result of being damaged is confirmed based on the road information and lane line information, including: In the same coordinate system, determine the overlapping area between the circumscribed rectangular frame and the real-time image, and use the circumscribed rectangular frame to intercept the real-time image in the overlapping area; The pre-trained target classification network is used to infer the captured area to determine whether the expansion joint in the real-time image is damaged. The target classification network is obtained by training the classification network MobileNetV3[1] based on a set of sample images of expansion joints detected as intact and damaged. The area of ​​each road is determined based on the mapping relationship between the road information and the lane line information, and the road where the expansion joint with the detection result being damaged is located is confirmed according to the intersection and union ratio of the area of ​​each road and the area of ​​the expansion joint with the detection result being damaged.

[0012] In a second aspect, the present application also provides a device for detecting a damaged expansion joint, including: A determination unit, configured to determine at least one area patch with changes based on a real-time image and a standard image, wherein the real-time image represents a real-time state of an expansion joint in a target road, the standard image represents that the expansion joint in the target road is in a good state, and the real-time image and the standard image are in the same coordinate system; A removal unit is used to remove vehicle information from each area with changes, obtain at least one suspected damaged area, and extract the circumscribed rectangular frame of each suspected damaged area; The detection unit is used to intercept the real-time image using an external rectangular frame, detect the intercepted area, and confirm the position of the expansion joint that is detected to be in a damaged state based on the road information and lane line information, wherein the vehicle information, road information and lane line information are determined based on the real-time image.

[0013] In some possible implementations provided in the present application, before determining at least one area pattern with changes based on the real-time image and the standard image, a matching unit is further included: Determine whether the real-time image and the standard image are in the same coordinate system; If not, the real-time image and the standard image are matched to the same coordinate system.

[0014] In some possible implementations provided in the present application, the real-time image and the standard image are matched to the same coordinate system, and the matching unit is used to: Extracting a first feature point from the real-time image using a feature point extraction algorithm, and extracting a second feature point from the standard image using a feature point extraction algorithm; Determining a homography matrix based on the first feature point and the second feature point; The standard image is transformed into the coordinate system of the real-time image based on the homography matrix.

[0015] In some possible implementations provided in the present application, at least one area pattern with a change is determined based on the real-time image and the standard image, and the determination unit is used to: Inputting the real-time image and the standard image into the first encoder and the second encoder of the pre-trained change detection network through different first input ports respectively; Using a first encoder to encode the real-time image to obtain a first encoding result, using a second encoder to encode the standard image to obtain a second encoding result, and fusing the first encoding result and the second encoding result to obtain a fused encoding result; Inputting the fused encoding result into the decoder of the change detection network through the second input port; Decoding the fusion coding result using a decoder to obtain a first decoding result; Based on the first decoding result, change information of the real-time image compared with the standard image is determined, and based on the change information, at least one area pattern with change is determined.

[0016] In some possible implementations provided in this application, vehicle information, road information, and lane line information are determined based on real-time images in the following ways: The real-time image is input into a pre-trained object segmentation network for inference. The object segmentation network is obtained by training the semantic segmentation network HRNet[2] based on a set of sample images labeled with vehicles, roads, and lane lines. Based on the inference results, the vehicle information, road information and lane line information in the real-time image are determined.

[0017] In some possible implementations provided in the present application, vehicle information is removed from each area with changes to obtain at least one suspected damaged area, and a circumscribed rectangular frame of each suspected damaged area is extracted. The removal unit is used to: Filter out the regional spots whose areas are smaller than the preset area from the regional spots with changes, and obtain at least one valid regional spot; Searching for a target area exceeding a preset threshold in the effective area spots, wherein the preset threshold represents a threshold corresponding to the area intersection-combination ratio between the area spots with changes and the spots corresponding to the vehicle information; The found target area is eliminated from each valid area patch to obtain at least one suspected damaged area, and the circumscribed rectangular frame of each suspected damaged area is extracted.

[0018] In some possible implementations provided in the present application, a real-time image is captured using an external rectangular frame, the captured area is detected, and the position of the expansion joint with a detection result of being damaged is confirmed based on the road information and lane line information. The detection unit is used to: In the same coordinate system, determine the overlapping area between the circumscribed rectangular frame and the real-time image, and use the circumscribed rectangular frame to intercept the real-time image in the overlapping area; The pre-trained target classification network is used to infer the captured area to determine whether the expansion joint in the real-time image is damaged. The target classification network is obtained by training the classification network MobileNetV3[1] based on a set of sample images of expansion joints detected as intact and damaged. The area of ​​each road is determined based on the mapping relationship between the road information and the lane line information, and the road where the expansion joint with the detection result being damaged is located is confirmed according to the intersection and union ratio of the area of ​​each road and the area of ​​the expansion joint with the detection result being damaged.

[0019] According to a third aspect, a computing device includes: A memory for storing executable instructions; A processor is used to read and execute executable instructions stored in a memory to implement any method as described in the first aspect.

[0020] In a fourth aspect, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor, the processor is enabled to execute any method described in the first aspect.

[0021] The beneficial effects of this application are as follows: In summary, in an embodiment of the present application, a method, device and storage medium for detecting a damaged expansion joint are provided. The method is applied to a computing device, including: determining at least one changed regional patch based on a real-time image and a standard image, wherein the real-time image represents the real-time state of the expansion joint in the target road, and the standard image represents that the expansion joint in the target road is in an intact state. The real-time image and the standard image are in the same coordinate system, and vehicle information is removed from each changed regional patch to obtain at least one suspected damaged area, and an circumscribed rectangular frame of each suspected damaged area is extracted, and the real-time image is intercepted using the circumscribed rectangular frame, and the intercepted area is detected, and the position of the expansion joint with a detection result of a damaged state is confirmed based on the road information and the lane line information, wherein the vehicle information, the road information and the lane line information are determined based on the real-time image. The above method for detecting expansion joints does not need to rely on complex external equipment, the detection process is convenient, and the detection accuracy is high.

[0022] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 It is a schematic diagram of the location of expansion joints in roads in the prior art; Figure 2 This is a schematic diagram of a system architecture for detecting expansion joints in an embodiment of the present application; Figure 3 This is a schematic diagram of the process of detecting the expansion joint in the embodiment of the present application; Figure 4 A schematic diagram of a process for determining whether a real-time image and a standard image are in the same coordinate system in an embodiment of the present application; Figure 5 A schematic diagram of a process for converting a real-time image and a standard image into the same coordinate system in an embodiment of the present application; Figure 6 A schematic diagram of a process of determining at least one area pattern with changes based on a real-time image and a standard image in an embodiment of the present application; Figure 7 This is a schematic diagram of the structure of a change detection network in an embodiment of the present application; Figure 8 This is a schematic diagram of the structure of a target segmentation network in an embodiment of the present application; Fig. 9 A schematic diagram of a real-time image in an embodiment of the present application; Fig.10 This is a schematic diagram of vehicle information, road information, and lane line information determined after reasoning on a real-time image using a target segmentation network in an embodiment of the present application; Fig.11 This is a schematic diagram of a process of obtaining a suspected damaged area and a circumscribed rectangular frame according to a regional pattern in an embodiment of the present application; Fig.12 This is a schematic diagram of a process for detecting and confirming the position of an expansion joint in an embodiment of the present application; Fig.13 A schematic diagram of an anti-residual in an embodiment of the present application; Fig.14 This is a schematic diagram of the first SE in the embodiment of the present application; Fig.15 This is a schematic diagram of the logical architecture of the first device for detecting a damaged expansion joint in an embodiment of the present application; Fig.16 This is a schematic diagram of the logical architecture of a second device for detecting damaged expansion joints in an embodiment of the present application; Fig.17 Schematic diagram of the physical architecture of a computing device in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0025] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented using sequences other than those illustrated or described herein.

[0026] The preferred implementation modes of the present application are described in detail below with reference to the accompanying drawings.

[0027] In the embodiment of the present application, the front end of the system includes at least one high-definition surveillance camera, see Figure 1As shown, it is assumed that there is a section of road A and a section of road B, and an expansion joint is provided at the junction of road A and road B. In order to detect the state of the expansion joint, a high-definition monitoring camera C is further provided on one side of the junction of road A and road B in the embodiment of the present application. The high-definition monitoring camera C is mainly used to obtain multiple real-time images of the expansion joint, and send the obtained real-time images to the back-end computing device, so that the computing device detects whether the expansion joint is damaged according to the real-time images.

[0028] See also Figure 2 As shown, in the embodiment of the present application, the backend of the system also includes at least one computing device. Figure 2 In the process, the computing device first receives the real-time image and the standard image, and then performs damage detection and position confirmation on the expansion joint according to the real-time image and the standard image. It should be noted that the standard image can be obtained by the high-definition monitoring camera C or other high-definition monitoring cameras installed on the road.

[0029] In the embodiment of the present application, a method for detecting a damaged expansion joint is implemented, which is mainly executed on the computing device side. In the embodiment of the present application, the results of detecting the expansion joint include two types: the first type is that the expansion joint is in a damaged state; the second type is that the expansion joint is in an intact state. The following is a detailed introduction.

[0030] See also Figure 3 As shown, in the embodiment of the present application, the specific process of a method for detecting a damaged expansion joint is as follows: Step 201: Determine at least one area patch with changes based on a real-time image and a standard image, wherein the real-time image represents the real-time status of the expansion joint in the target road, the standard image represents that the expansion joint in the target road is in an intact state, and the real-time image and the standard image are in the same coordinate system.

[0031] In the embodiment of the present application, the real-time image is an image obtained by the high-definition monitoring camera that represents the real-time status of the expansion joint in the target road. In order to detect whether the expansion joint is in a damaged state, the multiple real-time images can be taken at fixed time intervals. The standard image is an image obtained by the high-definition monitoring camera that represents the expansion joint in the target road is in good condition. For example, the standard image can be an image obtained when the road is just put into use, or an image obtained by the high-definition monitoring camera and confirmed by the back-end computing device that the expansion joint is in good condition.

[0032] It should be noted here that in order to make the expansion joint detection more accurate, before determining at least one area with changes based on the real-time image and the standard image, refer to Figure 4 As shown, it also includes: Step 101: Determine whether the real-time image and the standard image are in the same coordinate system. If so, execute step 201; if not, execute step 102.

[0033] Considering that the high-definition surveillance camera installed on the side of the road may move due to the influence of the external environment (for example, wind, rain, hail, etc.), or the camera is damaged and replaced with a new one, the above multiple real-time images may be offset. During the implementation process, before comparing the real-time image with the standard image, the coordinate system of the real-time image and the standard image needs to be calibrated first.

[0034] During the implementation process, a position point of a reference object can be selected in the real-time image, and the coordinates of the position point in the real-time image can be recorded. Similarly, the same position point of the same reference object can be selected in the standard image, and the coordinates of the position point in the standard image can be recorded. By comparing whether the two recorded coordinates are the same, it can be determined whether the real-time image and the standard image are in the same coordinate system.

[0035] If the real-time image and the standard image are in the same coordinate system, step 201 is executed, that is, at least one area with changes is determined directly based on the real-time image and the standard image; if the real-time image and the standard image are not in the same coordinate system, step 102 is executed, that is, the real-time image and the standard image are first matched to the same coordinate system.

[0036] Step 102: Match the real-time image and the standard image to the same coordinate system.

[0037] See also Figure 5 As shown, the above steps of matching the real-time image and the standard image to the same coordinate system specifically include: Step 1021: extracting first feature points from the real-time image using a feature point extraction algorithm, and extracting second feature points from the standard image using a feature point extraction algorithm.

[0038] In the process of matching the coordinate systems of the real-time image and the standard image, in order to make the matching result as accurate as possible, during the feature point extraction process, feature points will be extracted for the same position of the same reference object in the target road.

[0039] During the implementation, a feature point extraction algorithm is used to extract a first feature point from a real-time image, and a feature point extraction algorithm is used to extract a second feature point from a standard image. Exemplarily, the feature point extraction algorithm may use an Oriented FAST and Rotated BRIEF (ORB) feature point extraction algorithm, which is a computer vision algorithm for feature extraction and matching. It extracts key points and feature descriptors by improving corner detectors and descriptors, and has rotation invariance and robustness. That is, the ORB feature point extraction algorithm is used to extract a first feature point from the same position of a reference object in a real-time image, and then the ORB feature point extraction algorithm is used to extract a second feature point from the same position of a reference object in a standard image.

[0040] Step 1022: Determine a homography matrix based on the first feature point and the second feature point.

[0041] After the first feature point and the second feature point are obtained, a homography matrix may be further determined according to the first feature point and the second feature point. The homography matrix is ​​a transformation matrix.

[0042] Exemplarily, when it is necessary to transform the standard image into the coordinate system where the real-time image is located, the homography matrix is ​​obtained by dividing the first feature point by the second feature point.

[0043] Step 1023: transform the standard image into the coordinate system of the real-time image based on the homography matrix.

[0044] During implementation, after obtaining the homography matrix, the standard image is multiplied by the homography matrix to transform the standard image into the coordinate system where the real-time image is located.

[0045] After the real-time image and the standard image are in the same coordinate system, at least one area spot with changes can be determined based on the real-time image and the standard image. Figure 6 and Figure 7 As shown, including: The pre-trained change detection network used in the embodiment of the present application is different from the traditional change detection network, which only includes one encoder and one decoder. The pre-trained change detection network used in the embodiment of the present application includes two encoders and one decoder, and the two encoders are used to process the real-time image and the standard image respectively. Using the dual images to input the change detection network respectively can better compare the change information of the same position of the dual images, so that the difference between the real-time image and the standard image can be compared more conveniently. The following is a specific introduction in conjunction with the following steps.

[0046] Step 2011: The real-time image and the standard image are respectively input into the first encoder and the second encoder of the pre-trained change detection network through different first input ports.

[0047] During the implementation, in order to facilitate the comparison between the real-time image and the standard image, the real-time image is input into the first encoder of the pre-trained change detection network through one of the first input ports, so that the first encoder can encode the real-time image.

[0048] Similarly, the real-time image is input into the second encoder of the pre-trained change detection network through another first input port, so that the second encoder can encode the standard image.

[0049] Step 2012: Use the first encoder to encode the real-time image to obtain a first encoding result, use the second encoder to encode the standard image to obtain a second encoding result, and fuse the first encoding result and the second encoding result to obtain a fused encoding result.

[0050] During the implementation process, the encoder part uses an improved version of the VGG network as the main feature extraction network. Each time it passes through a pooling layer, the scale is reduced, and finally five feature layers of different scales can be obtained.

[0051] After the real-time image is input to the first encoder, the first encoder first The pooling layer is used for feature extraction, and down-sampling is performed through the sequentially connected pooling layers to gradually extract the image features of the real-time image, so that the image features can be transferred between different levels, retaining more detail features, and finally obtaining the first encoding result.

[0052] Similarly, after the standard image is input to the second encoder, the second encoder first transforms the image from the dimension The pooling layer is used for feature extraction, and down-sampling is performed through the sequentially connected pooling layers to gradually extract the image features of the real-time image, so that the image features can be transferred between different levels, retaining more detail features, and finally obtaining the second encoding result.

[0053] After obtaining the first encoding result and the second encoding result, the first encoding result and the second encoding result are fused to obtain a fused encoding result. The fusion process includes but is not limited to directly fusion, fusion of encoding results corresponding to frames in the same position, etc.

[0054] Step 2013: Input the fused encoding result into the decoder of the change detection network through the second input port.

[0055] During the implementation process, the obtained fusion coding result is input into the decoder of the change detection network through the second input port for decoding. It should be noted that the decoder part in the embodiment of the present application is composed of a series of upsampling layers, one upsampling layer corresponds to one pooling layer, and the feature layer not only comes from the upsampling of the high-dimensional feature layer, but also integrates the feature layer of the same scale from the encoder. Such a decoder integrates more high-dimensional semantic information and low-dimensional spatial information.

[0056] Step 2014: Use a decoder to decode the fusion coding result to obtain a first decoding result.

[0057] During the implementation process, the process of decoding the fusion coding result by the decoder is opposite to the encoding process. The pooling layer in the decoder is used to perform transposed convolution on the above fusion coding result, so as to gradually restore the image features of the real-time image and the standard image, and allow the image features to be transferred between different levels, thereby avoiding the loss of information during the upsampling process.

[0058] Step 2015: Determine change information of the real-time image compared with the standard image based on the first decoding result, and determine at least one changed area patch based on the change information.

[0059] After obtaining the first decoding result, if the decoding results representing the same feature point in the first decoding result are different, it is determined as change information of the real-time image compared with the standard image. During the implementation process, a regional pattern is further generated based on the above change information, thereby determining at least one regional pattern with changes based on the first decoding result.

[0060] Considering that the road will be affected by real-time vehicles, the above-mentioned changed area patches may include the change information of the moving vehicles. Based on this, in the process of detecting the expansion joint, the vehicle information needs to be eliminated. It should be noted here that the vehicle information, road information and lane line information are determined based on the real-time image in the following way: (1) The real-time image is input into a pre-trained object segmentation network for inference. The object segmentation network is obtained by training the semantic segmentation network HRNet[2] based on a set of sample images labeled with vehicles, roads, and lane lines.

[0061] Here we first introduce the target segmentation network, see Figure 8 As shown, in an embodiment of the present application, a sample image set is used in advance to train the semantic segmentation network HRNet[2] to obtain a target segmentation network. The sample images in the above sample image set are obtained by annotating multiple real-time images captured by a high-definition surveillance camera, that is, the vehicles, roads and lane lines included in each real-time image are annotated to obtain a sample image set.

[0062] The above-mentioned target segmentation network uses high-resolution and low-resolution feature layers connected in parallel, and all feature layers in the first row maintain high-resolution features, thereby avoiding the situation where the original high resolution is lost due to downsampling due to lack of memory or other reasons. When upsampling is performed in the second and third layers, it is not restored by interpolation, but directly fused with the original high-resolution features, which is better and more accurate than other networks in extracting spatial feature information. In addition, at different stages, the above-mentioned target segmentation network fuses low-resolution feature images of the same depth and similarity to improve high-resolution semantic feature information, and can fuse information of different scales for final prediction. The target segmentation network in the embodiment of the present application not only makes full use of low-resolution features, but also retains spatial features with higher precision, so that the final result is more excellent.

[0063] During the implementation process, the real-time image acquired by the high-definition surveillance camera is input into the pre-trained target segmentation network, and the high-resolution features of the real-time image are extracted through the high-resolution convolution layer of the first stage. The second stage of the above-mentioned target segmentation network includes two branches, one of which is a continuation of the convolution layer of the first stage, and the other branch is a newly generated branch based on the convolution layer of the first stage. During the implementation process, the low-resolution features of the real-time image are extracted by downsampling, and the high-resolution features of the original real-time image are directly fused by upsampling. Compared with the interpolation method, the high-resolution features of the real-time image can be better maintained. The third stage of the above-mentioned target segmentation network includes three branches, one of which is a continuation of the convolution layer of the first stage, the second branch is a continuation of the convolution layer of the second stage, and the third branch is a newly generated branch based on the convolution layer of the second stage. During the implementation process, the low-resolution features of the real-time image are extracted by downsampling, and the high-resolution features of the original real-time image are directly fused by upsampling, so as to obtain the final output feature map.

[0064] (2) Based on the inference results, determine the vehicle information, road information, and lane line information in the real-time image.

[0065] During the implementation, after obtaining the inference result, i.e., the output feature map, the vehicle information, road information, and lane line information in the real-time image can be determined. It should be noted that, usually, the threshold of the vehicle information is greater than the threshold of the road information and the lane line information.

[0066] For example, see Fig. 9 As shown in the figure, there is a real-time image of a road with six lane lines, which form five lanes, and a truck is driving in the second lane. The real-time image is input into the pre-trained object segmentation network for inference. Fig.10As shown, Fig.10 The truck outline in is the determined vehicle information. Fig.10 The lane lines in (e.g., re-determined by numbers) are the lane line information. Fig.10 A road information is determined by combining the adjacent lane line information.

[0067] Step 202: Remove vehicle information from each area patch with changes to obtain at least one suspected damaged area, and extract the circumscribed rectangular frame of each suspected damaged area.

[0068] Taking into account that vehicle information may cause occlusion and other effects on the detection of expansion joints, during the implementation process, after the vehicle information is determined based on the real-time image, the vehicle information will first be removed from the above-mentioned regional patches that have changed compared to the standard image, so as to avoid affecting the expansion joint detection.

[0069] For example, the above-mentioned determination of the suspected damaged area and extraction of the circumscribed rectangular frame of each suspected damaged area are described in detail. Fig.11 As shown, specifically including: Step 2021: Filter out the area spots with an area smaller than a preset area from the various area spots with changes, and obtain at least one valid area spot.

[0070] In addition, considering that as the road is used for a longer time, the road itself and the expansion joint will have small damage within the allowable range, that is, the small damage within the allowable range is normal wear and tear of the road. Based on this, in the implementation process, after determining each area with changes, first filter out the area with an area smaller than the preset area, that is, directly determine the area with an area smaller than the preset area as a road in normal use or an expansion joint in good condition. The area with the preset area represents the small damage within the allowable range. After the above filtering process, each area with changes is converted into at least one valid area.

[0071] Step 2022: Searching for a target area exceeding a preset threshold in the effective area patches, wherein the preset threshold represents a threshold corresponding to the area intersection-over-combination ratio between the area patches with changes and the patches corresponding to the vehicle information.

[0072] After determining the effective area patches, the vehicle information determined above is further removed from the effective area patches. Since there are differences between the overlapping areas between the vehicle information and the road information and the lane line information, during the implementation process, the target area exceeding the preset threshold is searched in the effective area patches determined above. It should be supplemented that the preset threshold represents the threshold corresponding to the area intersection and union ratio between the area patches with changes and the areas corresponding to the vehicle information, that is, the area of ​​each area patch with changes is respectively used to calculate the intersection and union ratio of the area of ​​the area patch corresponding to the vehicle information, and the intersection and union ratio representing the degree of overlap between the vehicle information and the area patches with changes is greater than a certain threshold is used as the preset threshold.

[0073] Step 2023: Eliminate the found target area from each valid area patch to obtain at least one suspected damaged area, and extract the circumscribed rectangular frame of each suspected damaged area.

[0074] After finding the target area representing the vehicle information, the found target area is removed from the above-mentioned effective area patches. The removal method includes but is not limited to setting the target area to the same area as the road information or lane line information.

[0075] During the implementation process, the effective area patches that exclude the target area are called suspicious damaged areas. Considering that the road information and lane line information are relatively stable and unchanged, the suspicious damaged area represents the state change of the expansion joint in the real-time image compared with the standard image. In order to more intuitively and vividly represent the state change of the above expansion joint, during the implementation process, the circumscribed rectangular frame of each suspicious damaged area will be extracted based on the determined suspicious damaged areas, that is, based on the outline of each suspicious damaged area, an circumscribed rectangular frame is determined for each suspicious damaged area. The circumscribed rectangular frame represents the state change range of the expansion joint in the real-time image compared with the standard image.

[0076] Step 203: Use an external rectangular frame to intercept the real-time image, detect the intercepted area, and confirm the position of the expansion joint that is detected to be damaged based on the road information and lane line information, wherein the vehicle information, road information and lane line information are determined based on the real-time image.

[0077] During the implementation process, after the circumscribed rectangular frame is confirmed, that is, after the approximate range of changes of the real-time image compared with the standard image is confirmed, the real-time image will be combined with the circumscribed rectangular frame to detect and confirm the position of the expansion joint.

[0078] For example, the above process of detecting and confirming the position of the expansion joint is as follows: Fig.12 As shown, specifically including: Step 2031: In the same coordinate system, determine the overlapping area between the circumscribed rectangular frame and the real-time image, and use the circumscribed rectangular frame to intercept the real-time image in the overlapping area.

[0079] During the implementation process, in order to improve the accuracy of detection and position confirmation of expansion joints, it is necessary to first place the circumscribed rectangular frame and the real-time image in the same coordinate system. Preferably, the circumscribed rectangular frame is placed in the same coordinate system matched to the above-mentioned real-time image and the standard image, and the specific placement position refers to the position of the circumscribed rectangular frame in the corresponding real-time image and / or standard image.

[0080] After placing the circumscribed rectangular frame and the real-time image in the same coordinate system, the overlapping area of ​​the circumscribed rectangular frame in the corresponding real-time image is determined. The overlapping area with the circumscribed rectangular frame in the real-time image represents the area range of the expansion joint included in the real-time image that may be in a damaged state. That is, by introducing the circumscribed rectangular frame, not only the influence of vehicle information on the state of the expansion joint in the real-time image is effectively eliminated, but also the range of confirmation of the damaged state of the expansion joint is further narrowed.

[0081] After further narrowing the determination range, the real-time image in the overlapping area is intercepted using the circumscribed rectangular frame, that is, only the overlapping area of ​​the real-time image and the circumscribed rectangular frame is intercepted as the determination area of ​​the expansion joint state.

[0082] Step 2032: Use a pre-trained target classification network to infer the captured area to determine whether the state of the expansion joint in the real-time image is a damaged state, wherein the target classification network is obtained by training the classification network MobileNetV3[1] based on a set of sample images in which the expansion joint is detected as being in an intact state and the expansion joint is detected as being in a damaged state.

[0083] The classification network MobileNetV3[1] adopts lightweight structures such as deep separable convolution and residual blocks, including multiple modules such as bottleneck structure, SE module and NL module. The classification network MobileNetV3[1] has higher classification accuracy and less latency. In the embodiment of the present application, the classification network MobileNetV3[1] is firstly trained by a sample image set, that is, the classification network MobileNetV3[1] is trained using a plurality of sample images of expansion joints detected as intact and expansion joints detected as damaged, thereby obtaining the above-mentioned pre-trained target classification network.

[0084] During the implementation process, after the intercepted area is obtained, the intercepted area is input into the above-mentioned target classification network for reasoning, and whether the state of the expansion joint in the real-time image is a damaged state is determined according to the result of the reasoning.

[0085] See also Fig.13As shown, the target classification network uses a deep separable convolution and an anti-residual module with a linear bottleneck, which greatly improves the detection speed and effect. The anti-residual module in the embodiment of the present application can use a regularized Relu6 function with a step size of 1 or a step size of 2. When the regularized Relu6 function with a step size of 1 is used, the number of channels is first changed through a 1*1 convolution layer, and the linear function is used to extract the initial image features, and then the features of each channel are extracted through a 3*3 depth convolution and a regularized Relu6 function with a step size of 1, and then the fused features are obtained through a 1*1 convolution layer and a regularized Relu6 function with a step size of 1. The fused features can be used as input again to repeat the convolution operation in the above steps to obtain the final output features; when the regularized Relu6 function with a step size of 2 is used, the number of channels is first changed through a 1*1 convolution layer, and the linear function is used to extract the initial image features, and then the features of each channel are extracted through a 3*3 depth convolution and a regularized Relu6 function with a step size of 2, and then the fused features are obtained through a 1*1 convolution layer and a regularized Relu6 function with a step size of 1, and the fused features can be used as the final output features.

[0086] See also Fig.14 As shown, a squeeze and excitation (SE) module is also introduced in the target classification network in the embodiment of the present application. The SE module includes a global pooling layer, a fully connected layer FC, a regularized ReLU function, a fully connected layer FC, and an activation function Sigmoid layer. The feature map is obtained by compressing the features of the truncated area of ​​the input, and the weight of each channel is generated through the fully connected layer. Therefore, the weights of different channels can be dynamically adjusted while retaining the original features, thereby improving the performance of the model.

[0087] Step 2033: Determine the area of ​​each road based on the mapping relationship between the road information and the lane line information, and confirm the road where the expansion joint with the detection result of being damaged is located according to the intersection ratio of the area of ​​each road and the area of ​​the expansion joint with the detection result of being damaged.

[0088] After determining that a certain expansion joint in the road is in a damaged state, the position of the expansion joint is further confirmed, that is, it is determined in which road the expansion joint is located. During implementation, after the road information and lane line information are determined according to the real-time image, the arrangement order of the road information is determined according to the specific lane line information, for example, the road information between the first lane line information and the second lane line information is the first road, etc., thereby establishing a mapping relationship between the road information and the lane line information.

[0089] After determining the information of each road, the area of ​​each road is further calculated. For example, the area of ​​the road information can be obtained by multiplying the length of the road by the width. At the same time, the detection result is calculated to be the state of the expansion joint is damaged. Here, the area of ​​the expansion joint needs to be calculated according to the specific shape of the expansion joint, which will not be described in detail. After determining the area of ​​each road and the area of ​​the expansion joint in the damaged state, the area of ​​each road is respectively compared with the area of ​​the expansion joint. The road with the largest intersection-and-combination ratio is confirmed as the road where the expansion joint with the detection result of the damaged state is located, thereby confirming the position of the expansion joint.

[0090] Based on the same inventive concept, refer to Fig.15 and Fig.16 As shown, in an embodiment of the present application, a detection device for a damaged expansion joint is provided, comprising: A determining unit 1501 is used to determine at least one area patch with changes based on a real-time image and a standard image, wherein the real-time image represents a real-time state of an expansion joint in a target road, the standard image represents that the expansion joint in the target road is in a good state, and the real-time image and the standard image are in the same coordinate system; The elimination unit 1502 is used to eliminate vehicle information from each changed area patch to obtain at least one suspected damaged area and extract the circumscribed rectangular frame of each suspected damaged area; The detection unit 1503 is used to use an external rectangular frame to intercept the real-time image, detect the intercepted area, and confirm the position of the expansion joint that is detected to be in a damaged state based on the road information and lane line information, wherein the vehicle information, road information and lane line information are determined based on the real-time image.

[0091] In some possible implementations provided by the present application, before determining at least one area pattern with changes based on the real-time image and the standard image, a matching unit 1500 is further included: Determine whether the real-time image and the standard image are in the same coordinate system; If not, the real-time image and the standard image are matched to the same coordinate system.

[0092] In some possible implementations provided in the present application, the real-time image and the standard image are matched to the same coordinate system, and the matching unit 1500 is used to: Extracting a first feature point from the real-time image using a feature point extraction algorithm, and extracting a second feature point from the standard image using a feature point extraction algorithm; Determining a homography matrix based on the first feature point and the second feature point; The standard image is transformed into the coordinate system of the real-time image based on the homography matrix.

[0093] In some possible implementations provided in the present application, at least one area pattern with a change is determined based on the real-time image and the standard image, and the determining unit 1501 is used to: Inputting the real-time image and the standard image into the first encoder and the second encoder of the pre-trained change detection network through different first input ports respectively; Using a first encoder to encode the real-time image to obtain a first encoding result, using a second encoder to encode the standard image to obtain a second encoding result, and fusing the first encoding result and the second encoding result to obtain a fused encoding result; Inputting the fused encoding result into the decoder of the change detection network through the second input port; Decoding the fusion coding result using a decoder to obtain a first decoding result; Based on the first decoding result, change information of the real-time image compared with the standard image is determined, and based on the change information, at least one area pattern with change is determined.

[0094] In some possible implementations provided in this application, vehicle information, road information, and lane line information are determined based on real-time images in the following ways: The real-time image is input into a pre-trained object segmentation network for inference. The object segmentation network is obtained by training the semantic segmentation network HRNet[2] based on a set of sample images labeled with vehicles, roads, and lane lines. Based on the inference results, the vehicle information, road information and lane line information in the real-time image are determined.

[0095] In some possible implementations provided in the present application, vehicle information is removed from each area with changes to obtain at least one suspected damaged area, and a circumscribed rectangular frame of each suspected damaged area is extracted. The removal unit 1502 is used to: Filter out the regional spots whose areas are smaller than the preset area from the regional spots with changes, and obtain at least one valid regional spot; Searching for a target area exceeding a preset threshold in the effective area spots, wherein the preset threshold represents a threshold corresponding to the area intersection-combination ratio between the area spots with changes and the spots corresponding to the vehicle information; The found target area is eliminated from each valid area patch to obtain at least one suspected damaged area, and the circumscribed rectangular frame of each suspected damaged area is extracted.

[0096] In some possible implementations provided in the present application, a real-time image is captured using an external rectangular frame, the captured area is detected, and the position of the expansion joint with a detection result of being damaged is confirmed based on the road information and lane line information. The detection unit 1503 is used to: In the same coordinate system, determine the overlapping area between the circumscribed rectangular frame and the real-time image, and use the circumscribed rectangular frame to intercept the real-time image in the overlapping area; The pre-trained target classification network is used to infer the captured area to determine whether the expansion joint in the real-time image is damaged. The target classification network is obtained by training the classification network MobileNetV3[1] based on a set of sample images of expansion joints detected as intact and damaged. The area of ​​each road is determined based on the mapping relationship between the road information and the lane line information, and the road where the expansion joint with the detection result being damaged is located is confirmed according to the intersection and union ratio of the area of ​​each road and the area of ​​the expansion joint with the detection result being damaged.

[0097] Based on the same inventive concept, refer to Fig.17 As shown, an embodiment of the present application provides a computing device, including: a memory 1701, used to store executable instructions; a processor 1702, used to read and execute the executable instructions stored in the memory, and execute any one of the methods of the first aspect above.

[0098] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor, the processor is enabled to execute the method described in any one of the first aspects above.

[0099] In summary, a method, device and storage medium for detecting damaged expansion joints are provided in an embodiment of the present application. The method is applied to a computing device, including: determining at least one changed regional patch based on a real-time image and a standard image, wherein the real-time image represents the real-time state of the expansion joint in the target road, and the standard image represents that the expansion joint in the target road is in an intact state. The real-time image and the standard image are in the same coordinate system, and vehicle information is removed from each changed regional patch to obtain at least one suspected damaged area, and the circumscribed rectangular frame of each suspected damaged area is extracted, and the real-time image is intercepted using the circumscribed rectangular frame, and the intercepted area is detected, and the position of the expansion joint with a detection result of a damaged state is confirmed based on the road information and the lane line information, wherein the vehicle information, the road information and the lane line information are determined based on the real-time image. The above method for detecting expansion joints does not need to rely on complex external equipment, the detection process is convenient, and the detection accuracy is high.

[0100] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program product systems. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product system implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0101] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program product systems according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0102] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0103] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0104] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for detecting damaged expansion joints, It is characterized in that Applied to a computing device, the method comprises: Determine at least one area patch with changes based on a real-time image and a standard image, wherein the real-time image represents a real-time state of an expansion joint in a target road, the standard image represents that the expansion joint in the target road is in a good state, and the real-time image and the standard image are in the same coordinate system; Remove vehicle information from each of the changed area patches to obtain at least one suspected damaged area, and extract the circumscribed rectangular frame of each of the suspected damaged areas; The real-time image is captured using the circumscribed rectangular frame, the captured area is detected, and the position of the expansion joint that is detected to be in a damaged state is confirmed based on the road information and the lane line information, wherein the vehicle information, the road information and the lane line information are determined based on the real-time image.

2. The method according to claim 1, It is characterized in that Before determining at least one changed regional pattern spot based on the real-time image and the standard image, the method further includes: Determining whether the real-time image and the standard image are in the same coordinate system; If not, the real-time image and the standard image are matched to the same coordinate system.

3. The method according to claim 2, It is characterized in that The matching of the real-time image and the standard image to the same coordinate system includes: Extracting first feature points from the real-time image using a feature point extraction algorithm, and extracting second feature points from the standard image using the feature point extraction algorithm; Determine a homography matrix based on the first feature point and the second feature point; The standard image is transformed into a coordinate system where the real-time image is located based on the homography matrix.

4. The method according to any one of claims 1 to 3, It is characterized in that The step of determining at least one area pattern spot with a change based on the real-time image and the standard image comprises: Inputting the real-time image and the standard image into a first encoder and a second encoder of a pre-trained change detection network through different first input ports respectively; Using the first encoder to encode the real-time image to obtain a first encoding result, using the second encoder to encode the standard image to obtain a second encoding result, and fusing the first encoding result and the second encoding result to obtain a fused encoding result; Inputting the fused encoding result into a decoder of a change detection network through a second input port; Decoding the fusion coding result using the decoder to obtain a first decoding result; Change information of the real-time image compared with the standard image is determined based on the first decoding result, and at least one area patch with change is determined based on the change information.

5. The method according to claim 1, It is characterized in that The vehicle information, the road information and the lane line information are determined based on the real-time image in the following manner: Inputting the real-time image into a pre-trained object segmentation network for inference, wherein the object segmentation network is obtained by training a semantic segmentation network HRNet[2] based on a sample image set labeled with vehicles, roads, and lane lines; Based on the result of the reasoning, the vehicle information, the road information and the lane line information in the real-time image are determined.

6. The method according to claim 1, It is characterized in that The step of removing vehicle information from each of the changed area patches to obtain at least one suspected damaged area and extracting a circumscribed rectangular frame of each of the suspected damaged areas includes: Filter out the regional spots with an area smaller than a preset area from the regional spots with changes, and obtain at least one valid regional spot; Searching for a target area exceeding a preset threshold in the effective area spots, wherein the preset threshold represents a threshold corresponding to the area intersection-and-combination ratio between the area spots with changes and the spots corresponding to the vehicle information; The target area found is eliminated from each of the effective area patches to obtain at least one of the suspected damaged areas, and the circumscribed rectangular frames of each of the suspected damaged areas are extracted.

7. The method according to claim 1, It is characterized in that The using of the circumscribed rectangular frame to intercept the real-time image, detecting the intercepted area, and confirming the position of the expansion joint with a detection result of being damaged based on the road information and the lane line information, includes: In the same coordinate system, determining an overlapping area between the circumscribed rectangular frame and the real-time image, and using the circumscribed rectangular frame to intercept the real-time image in the overlapping area; Use a pre-trained target classification network to infer the captured area to determine whether the expansion joint in the real-time image is in a damaged state, wherein the target classification network is obtained by training a classification network MobileNetV3[1] based on a set of sample images of expansion joints detected as intact and expansion joints detected as damaged; The area of ​​each road is determined based on the mapping relationship between the road information and the lane line information, and the road where the expansion joint with the detection result being in a damaged state is located is confirmed according to the intersection-and-union ratio of the area of ​​each road and the area of ​​the expansion joint with the detection result being in a damaged state.

8. A detection device for damaged expansion joints, It is characterized in that include: A determination unit, configured to determine at least one area patch with changes based on a real-time image and a standard image, wherein the real-time image represents a real-time state of an expansion joint in a target road, the standard image represents that the expansion joint in the target road is in a good state, and the real-time image and the standard image are in the same coordinate system; A removal unit, used to remove vehicle information from each of the changed area patches, obtain at least one suspected damaged area, and extract the circumscribed rectangular frame of each of the suspected damaged areas; A detection unit is used to use the circumscribed rectangular frame to intercept the real-time image, detect the intercepted area, and confirm the position of the expansion joint that is detected to be in a damaged state based on the road information and lane line information, wherein the vehicle information, the road information and the lane line information are determined based on the real-time image.

9. A computing device, It is characterized in that include: A memory for storing executable instructions; A processor, configured to read and execute the executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that When the instructions in the storage medium are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 7.

Images