Construction method and device of road crack depth prediction model with double-branch structure
Through the combination of the dual-branch structure and the global-local attention fusion module, the problems of inefficiency and poor identification of traditional road crack detection methods are solved, and high accuracy prediction of road crack depth is achieved in complex environments.
Patent Information
- Application Number
- CN202411380661.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-09-30
AI Technical Summary
Traditional road crack detection methods rely on manual visual inspection, are inefficient and susceptible to human factors, and are difficult to meet the needs of large-scale and high-precision detection. Especially under complex backgrounds and poor lighting conditions, crack image recognition effect is poor.
The road crack depth prediction model with a two-branch structure is adopted. By extracting the RGB features and mask features of the road image respectively, and introducing a global-local attention fusion module, combining channel attention calculation, ensuring that the model maintains the computational efficiency. At the same time, it improves the accuracy of crack depth.
It improves the accuracy and model expression ability of road crack depth prediction, can accurately identify the crack depth on the road in the case of poor lighting, and enhances the reliability and efficiency of detection.
Smart Images

Figure CN118887552B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence, and in particular to a method and device for constructing a road crack depth prediction model with a dual-branch structure. Background Art
[0002] With the aging of infrastructure and the increase in traffic volume, the demand for road maintenance and inspection is growing, especially for road pavements, which are more prone to cracking and aging than asphalt pavements. Road crack detection, as a key link in preventive maintenance, is of great significance for ensuring traffic safety and reducing maintenance costs. Traditional crack detection methods rely on manual visual inspection, which is not only inefficient but also easily affected by human factors, making it difficult to meet large-scale, high-precision detection needs. In recent years, with the development of computer vision technology, automatic road crack detection methods based on image processing have gradually become a research hotspot.
[0003] However, in practical applications, road crack images often face many challenges, such as complex background environment, poor lighting conditions, and camera shaking during shooting. These factors greatly affect the recognition effect of crack images. Specifically, complex backgrounds may contain textures similar to cracks, leading to false detections; poor lighting conditions may result in low image contrast and unclear crack features; and shaking during shooting will cause image blur, further increasing the difficulty of crack detection.
[0004] Therefore, how to use simple shooting equipment to accurately identify the depth of cracks on the road under poor lighting conditions is a problem that the existing technology urgently needs to solve. Summary of the invention
[0005] The embodiment of the present application provides a method and device for constructing a road crack depth prediction model with a dual-branch structure. The dual-branch architecture is used to extract the RGB features and mask features of the road image respectively, and a global-local attention fusion module is introduced to ensure that important local details are not lost and the global structure is taken into account, thereby improving the expression ability of the model while maintaining the calculation efficiency of the model, and improving the accuracy of crack depth prediction.
[0006] In a first aspect, an embodiment of the present application provides a method for constructing a road crack depth prediction model with a dual-branch structure, the method comprising:
[0007] Acquire a road image with annotated crack depths and a crack mask image corresponding to the road image, wherein the road image is processed using a crack segmentation module to obtain a corresponding crack mask image, wherein the crack mask image only contains crack information in the road image;
[0008] Using a feature extraction module to extract features from the road image and the crack mask image to obtain a first road image feature and a first crack mask image feature;
[0009] In the global-local attention fusion module, the first road image feature and the first crack mask image feature are channel-connected to obtain a first fusion result, and global attention calculation and local attention calculation are performed on the first fusion result to obtain a global attention result and a local attention result, and then the global attention result and the local attention result are added to obtain a second fusion result, and the second fusion result is split into a second road image feature and a second crack mask image feature;
[0010] An output module is used to output the second predicted road image feature and the second predicted crack mask feature to obtain the crack depth of each crack in the road image, a loss function is constructed based on the output result of the output module and the annotation information, and the parameters of the road crack depth prediction model are corrected according to the loss function, wherein the road crack depth prediction model includes a crack segmentation module, a feature extraction module, a global-local attention fusion module and an output module.
[0011] In a second aspect, an embodiment of the present application provides a device for constructing a road crack depth prediction model with a dual-branch structure, comprising:
[0012] An acquisition module, used to acquire a road image with annotated crack depths and a crack mask image corresponding to the road image, wherein the road image is processed using the crack segmentation module to obtain a corresponding crack mask image, and the crack mask image only contains crack information in the road image;
[0013] A feature extraction module, used for extracting features from the road image and the crack mask image to obtain a first road image feature and a first crack mask image feature;
[0014] A global-local attention fusion module, used for performing channel connection on the first road image feature and the first crack mask image feature to obtain a first fusion result, performing global attention calculation and local attention calculation on the first fusion result to obtain a global attention result and a local attention result, then adding the global attention result and the local attention result to obtain a second fusion result, and splitting the second fusion result into a second road image feature and a second crack mask image feature;
[0015] An output module is used to output the second predicted road image feature and the second predicted crack mask feature to obtain the crack depth of each crack in the road image, construct a loss function based on the output result of the output module and the annotation information, and correct the parameters of the road crack depth prediction model according to the loss function, wherein the road crack depth prediction model includes a crack segmentation module, a feature extraction module, a global-local attention fusion module and an output module.
[0016] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute a method for constructing a road crack depth prediction model with a dual-branch structure.
[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process, and the process includes a method for constructing a road crack depth prediction model with a dual-branch structure.
[0018] The main contributions and innovations of the present invention are as follows:
[0019] The embodiment of the present application uses a dual-branch architecture to extract the RGB features and mask features of the road image respectively, thereby combining the complementary information of the RGB image and the binary mask image to improve the accuracy of depth prediction; the embodiment of the present application calculates channel attention on the road image and the crack mask image, and combines a global-local attention fusion module to ensure that important local details are not lost and the global structure is taken into account, thereby maintaining the computational efficiency of the model while improving the expression ability of the model and improving the accuracy of crack depth prediction.
[0020] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0022] Figure 1 is a structural diagram of a road crack depth prediction model with a double-branch structure according to an embodiment of the present application;
[0023] Figure 2 is a structural diagram of an image enhancement module according to an embodiment of the present application;
[0024] Figure 3 is a structural diagram of a first segmentation branch in a crack segmentation module according to an embodiment of the present application;
[0025] Figure 4 is a structural diagram of a second segmentation branch in a crack segmentation module according to an embodiment of the present application;
[0026] Figure 5 is a structural diagram of an RGB image extraction branch according to an embodiment of the present application;
[0027] Figure 6 is a structural diagram of a crack mask extraction branch according to an embodiment of the present application;
[0028] Figure 7 is a structural diagram of a convolution block according to an embodiment of the present application;
[0029] Figure 8 is a structural diagram of a residual network according to an embodiment of the present application;
[0030] Fig. 9 is a structural diagram of a global-local attention fusion module according to an embodiment of the present application;
[0031] Fig.10 is a schematic diagram of a road image used for training according to an embodiment of the present application;
[0032] Fig.11 It is a structural block diagram of a device for constructing a road crack depth prediction model with a double-branch structure according to an embodiment of the present application;
[0033] Fig.12 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0034] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.
[0035] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.
[0036] Embodiment 1
[0037] The embodiment of the present application provides a method for constructing a road crack depth prediction model, which uses a dual-branch architecture to extract RGB features and mask features of road images respectively, and introduces a global-local attention fusion module to ensure that important local details are not lost and the global structure is taken into account, thereby improving the expression ability of the model while maintaining the calculation efficiency of the model, and improving the accuracy of crack depth prediction. Specifically, reference Figure 1 , the method comprising:
[0038] Acquire a road image with annotated crack depths and a crack mask image corresponding to the road image, wherein the road image is processed using a crack segmentation module to obtain a corresponding crack mask image, wherein the crack mask image only contains crack information in the road image;
[0039] Using a feature extraction module to extract features from the road image and the crack mask image to obtain a first road image feature and a first crack mask image feature;
[0040] In the global-local attention fusion module, the first road image feature and the first crack mask image feature are channel-connected to obtain a first fusion result, and global attention calculation and local attention calculation are performed on the first fusion result to obtain a global attention result and a local attention result, and then the global attention result and the local attention result are added to obtain a second fusion result, and the second fusion result is split into a second road image feature and a second crack mask image feature;
[0041] An output module is used to output the second predicted road image feature and the second predicted crack mask feature to obtain the crack depth of each crack in the road image, a loss function is constructed based on the output result of the output module and the annotation information, and the parameters of the road crack depth prediction model are corrected according to the loss function, wherein the road crack depth prediction model includes a crack segmentation module, a feature extraction module, a global-local attention fusion module and an output module.
[0042] In some specific embodiments, in addition to analyzing and predicting road surface images taken by cameras in the road network, this solution can also analyze and predict road surface images taken by handheld shooting devices of patrol personnel. Therefore, in order to ensure the clarity of the acquired road images, this solution uses an image enhancement module to denoise and enhance the captured images to obtain road images, and then annotates the crack depths of the road images, thereby improving the accuracy of analysis and prediction.
[0043] In this solution, the image enhancement module includes a convolutional neural unit and a digital image processing unit. The convolutional neural unit extracts features from a captured image to obtain image parameters of the captured image. The digital image processing unit optimizes and adjusts the image parameters of the captured image to obtain a road image. The captured image is an image directly acquired by a shooting device.
[0044] Specifically, the structure of the image enhancement module is as follows: Figure 2 As shown, the convolutional neural unit in the image enhancement module extracts the features of the captured image through three serially connected convolutional layers, and then a multi-layer perceptron outputs the image parameters of the captured image, which include white balance, contrast, brightness, sharpening, gamma, hue, and noise.
[0045] Specifically, the digital image processing unit is a DIP parameter predictor, which can adaptively learn and adjust image parameters to achieve the purpose of enhancing image quality.
[0046] In this solution, the crack segmentation module includes a first branch and a second branch. The first segmentation branch directly performs crack segmentation on the road image to obtain a first segmentation result. The second segmentation branch uses the road image as a guide to denoise the first segmentation result to obtain a second segmentation result. The second segmentation result is a crack mask image.
[0047] Specifically, the first segmentation result obtained by the first segmentation branch is a crack mask with a low degree of refinement, and the second segmentation result obtained by the second segmentation branch is a crack mask image with a high degree of refinement.
[0048] Furthermore, the structure of the first segmentation branch in the crack segmentation module is as follows: Figure 3 As shown, in the first segmentation branch, the Transformer pyramid structure is used as an encoder, and the multi-layer convolution structure is used as a decoder to extract and analyze the features of the road image to obtain a rough crack mask image.
[0049] Specifically, the encoder of the first segmentation branch is composed of 4 series-connected Transformer-convolutional layers to form a pyramid structure. The feature extraction results of each Transformer-convolutional layer are fused and input into the decoder by a multi-layer perceptron. The decoder uses 4 series-connected convolutional layers to decode to obtain a decoding result, and uses a multi-layer perceptron to output the decoding result to obtain the first segmentation result.
[0050] Furthermore, the Transformer-convolutional layer is a hierarchical structure consisting of a Transformer layer and a convolutional layer.
[0051] Specifically, since the encoder adopts the Transformer structure, it can better analyze the relationship between pixels in the enhanced road image, and the pyramid structure can better combine the global context relationship, thereby obtaining a more accurate first segmentation result after decoding.
[0052] In this scheme, the structure of the second segmentation branch in the crack segmentation module is as follows: Figure 4 As shown, the second segmentation branch uses the dilated convolution layer and the convolution extraction layer to extract features from the road image and then fuses them to obtain a fusion result. The fusion result then passes through the SENet network and two convolution layers in sequence to obtain the guided features. Finally, the guided image filter is used to denoise the first segmentation result based on the guided features to obtain the second segmentation result.
[0053] Specifically, the guided filter adapts to the shape, size and direction of the cracks on the road based on the guided features, thereby optimizing the crack boundary to improve the accuracy of semantic segmentation and obtain a second segmentation result.
[0054] In this solution, the feature extraction module includes an RGB image extraction branch, a crack mask extraction branch and a channel attention branch. The RGB image extraction branch performs feature extraction on the road image to obtain a road RGB image feature. The crack mask extraction branch performs feature extraction on the crack mask image to obtain a crack mask feature. The road image and the crack mask image are channel-connected to obtain a connected crack image. The connected crack image is input into the channel attention branch to perform channel attention calculation to obtain a channel attention result. The channel attention result and the road RGB image feature are channel-multiplied to obtain a first road image feature. The channel attention result and the crack mask feature are channel-multiplied to obtain a first crack mask image feature.
[0055] Specifically, the structure of the RGB image extraction branch is as follows: Figure 5As shown, it includes two convolution units and an output unit connected in series. Each convolution unit has the same structure, which is connected in series by two convolution blocks and a residual network layer. The output unit is connected by a convolution block and a channel attention layer.
[0056] The structure of the crack mask extraction branch is as follows: Figure 6 As shown in Figure 1, three convolution blocks are connected in series, and finally an activation function is used for output. The structure of the convolution block in the RGB image extraction branch is the same as that in the crack mask extraction branch, which is connected in sequence by a convolution layer, batch normalization, activation function and maximum pooling layer. The structure of the convolution block is as follows: Figure 7 As shown in Figure 2, the residual network structure in the RGB image extraction branch is as follows: Figure 8 shown.
[0057] Specifically, since the acquired channel image is an RGB image, which is a three-channel image, and the crack mask image is a single-channel image, the 3-channel road image and the single-channel crack mask image are channel-connected to obtain a 4-channel connected crack image, and the channel attention result can be obtained by performing channel attention calculation on the connected crack image through the channel attention branch. The channel attention result is the importance weight of each channel. Using the channel attention result to perform channel multiplication with the road RGB image features and the crack mask features respectively can enhance important channel features and weaken unimportant channel features, thereby improving the learning efficiency of the model.
[0058] In this solution, the structure of the global-local attention fusion module is as follows Fig. 9 As shown, in the global-local attention fusion module, the global attention is calculated after the global average pooling is performed on the first fusion result, and the result of the global attention calculation is multiplied element by element with the first fusion result to obtain the global attention result; the local attention is calculated for the first fusion result and then output by the Sigmoid activation function, and the output result of the Sigmoid activation function is multiplied element by element with the first fusion feature to obtain the local attention result.
[0059] Specifically, the first road image features of the three roads and the first crack mask image features of the single channel are channel-connected to obtain a first fusion result of four channels. That is to say, the first fusion result includes the road image features and the crack mask image features. When the second fusion result is split, the features of the RGB channels are split out as the second road image features, and the remaining single channel features are the second crack mask image features.
[0060] Specifically, the global-local attention fusion module allows the model to process data without losing important local details while taking into account the global structure, thereby improving the model's expressiveness while maintaining its computational efficiency.
[0061] In this scheme, the output module includes a road image output branch and a crack mask image output branch. In the road image output branch, the second predicted road image feature is further extracted and input into the road MLP to obtain a road prediction result. In the crack mask image output branch, the second predicted crack mask feature is further extracted and input into the mask MLP to obtain a crack mask prediction result. After adding the features of the road prediction result and the crack mask prediction result, the final MLP output obtains the crack depth of each crack in the road image.
[0062] Specifically, a plurality of serially connected CGB modules are constructed to further extract the second predicted road image features and the second predicted crack mask features, wherein the CGB module is composed of a convolutional layer, a batch normalization, and a GELU activation function.
[0063] Furthermore, the output results of each CGB module in the road image output branch are added layer by layer in the form of a pyramid and output to the road MLP.
[0064] Furthermore, the output results of each CGB module in the crack mask image output branch are added layer by layer in the form of a pyramid and output to the mask MLP.
[0065] In some specific embodiments, the road images used in the training of this solution are as follows: Fig.10 As shown, in Fig.10 The location and depth of each crack are marked in the figure, and the MSE and R 2 As loss functions, the MSE loss and R loss functions are as follows:
[0066]
[0067]
[0068] Where n is the number of road images used, is the true value of the crack depth of the road image, is the predicted value of the road crack depth prediction model.
[0069] Embodiment 2
[0070] Based on the same idea, refer to Fig.11 The present application also proposes a device for constructing a road crack depth prediction model with a double-branch structure, comprising:
[0071] An acquisition module, used to acquire a road image with annotated crack depths and a crack mask image corresponding to the road image, wherein the road image is processed using the crack segmentation module to obtain a corresponding crack mask image, and the crack mask image only contains crack information in the road image;
[0072] A feature extraction module, used for extracting features from the road image and the crack mask image to obtain a first road image feature and a first crack mask image feature;
[0073] A global-local attention fusion module, used for performing channel connection on the first road image feature and the first crack mask image feature to obtain a first fusion result, performing global attention calculation and local attention calculation on the first fusion result to obtain a global attention result and a local attention result, then adding the global attention result and the local attention result to obtain a second fusion result, and splitting the second fusion result into a second road image feature and a second crack mask image feature;
[0074] An output module is used to output the second predicted road image feature and the second predicted crack mask feature to obtain the crack depth of each crack in the road image, construct a loss function based on the output result of the output module and the annotation information, and correct the parameters of the road crack depth prediction model according to the loss function, wherein the road crack depth prediction model includes a crack segmentation module, a feature extraction module, a global-local attention fusion module and an output module.
[0075] Embodiment 3
[0076] This embodiment also provides an electronic device, referring to Fig.12 , comprises a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above method embodiments.
[0077] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0078] Among them, the memory 404 may include a large capacity memory 404 for data or instructions. For example, but not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 404 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 404 may be inside or outside the data processing device. In a specific embodiment, the memory 404 is a non-volatile memory. In a specific embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. In appropriate circumstances, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM may be a fast page mode dynamic random access memory 404 (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.
[0079] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .
[0080] The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement any one of the methods for constructing a road crack depth prediction model with a double-branch structure in the above embodiments.
[0081] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .
[0082] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.
[0083] The input / output device 408 is used to input or output information. In this embodiment, the input information may be a road image, a crack mask image, etc., and the output information may be the crack depth of each crack, etc.
[0084] Optionally, in this embodiment, the processor 402 may be configured to perform the following steps through a computer program:
[0085] Acquire a road image with annotated crack depths and a crack mask image corresponding to the road image, wherein the road image is processed using a crack segmentation module to obtain a corresponding crack mask image, wherein the crack mask image only contains crack information in the road image;
[0086] Using a feature extraction module to extract features from the road image and the crack mask image to obtain a first road image feature and a first crack mask image feature;
[0087] In the global-local attention fusion module, the first road image feature and the first crack mask image feature are channel-connected to obtain a first fusion result, and global attention calculation and local attention calculation are performed on the first fusion result to obtain a global attention result and a local attention result, and then the global attention result and the local attention result are added to obtain a second fusion result, and the second fusion result is split into a second road image feature and a second crack mask image feature;
[0088] An output module is used to output the second predicted road image feature and the second predicted crack mask feature to obtain the crack depth of each crack in the road image, a loss function is constructed based on the output result of the output module and the annotation information, and the parameters of the road crack depth prediction model are corrected according to the loss function, wherein the road crack depth prediction model includes a crack segmentation module, a feature extraction module, a global-local attention fusion module and an output module.
[0089] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.
[0090] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.
[0091] Embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, at this point, it should be noted that, for example, Fig.12Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.
[0092] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0093] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for constructing a road crack depth prediction model with a double-branch structure, characterized in that: The following steps are involved: Acquire a road image with annotated crack depths and a crack mask image corresponding to the road image, wherein a crack segmentation module is used to process the road image to obtain a corresponding crack mask image, wherein the crack mask image only contains crack information in the road image, wherein the crack segmentation module includes a first branch and a second branch, wherein the first segmentation branch directly performs crack segmentation on the road image to obtain a first segmentation result, and the second segmentation branch uses the road image as a guide to denoise the first segmentation result to obtain a second segmentation result, wherein the second segmentation result is a crack mask image, wherein the first segmentation branch uses a Transformer pyramid structure as an encoder and a multi-layer convolution structure as a decoder to extract and analyze features of the road image to obtain a rough crack mask image, wherein the second segmentation branch uses an expanded convolution layer and a convolution extraction layer to extract features from the road image and then fuses the features to obtain a fusion result, wherein the fusion result is then sequentially passed through a SENet network and two convolution layers to obtain a guide feature, and finally a guide image filter is used to denoise the first segmentation result based on the guide feature to obtain a second segmentation result; Using a feature extraction module to extract features from the road image and the crack mask image to obtain a first road image feature and a first crack mask image feature; In the global-local attention fusion module, the first road image feature and the first crack mask image feature are channel-connected to obtain a first fusion result, and global attention calculation and local attention calculation are performed on the first fusion result to obtain a global attention result and a local attention result, and then the global attention result and the local attention result are added to obtain a second fusion result, and the second fusion result is split into a second road image feature and a second crack mask image feature; An output module is used to output the second predicted road image feature and the second predicted crack mask feature to obtain the crack depth of each crack in the road image, a loss function is constructed based on the output result of the output module and the annotation information, and the parameters of the road crack depth prediction model are corrected according to the loss function, wherein the road crack depth prediction model includes a crack segmentation module, a feature extraction module, a global-local attention fusion module and an output module.
2. The method for constructing a road crack depth prediction model with a double-branch structure according to claim 1, characterized in that: After denoising and enhancing the captured image through an image enhancement module to obtain a road image, the crack positions and crack depths of the road image are annotated, wherein the image enhancement module includes a convolutional neural unit and a digital image processing unit, the convolutional neural unit extracts features from the captured image to obtain image parameters of the captured image, and the digital image processing unit optimizes and adjusts the image parameters of the captured image to obtain a road image.
3. The method for constructing a road crack depth prediction model with a double-branch structure according to claim 1, characterized in that: The feature extraction module includes an RGB image extraction branch, a crack mask extraction branch and a channel attention branch. The RGB image extraction branch performs feature extraction on a road image to obtain a road RGB image feature. The crack mask extraction branch performs feature extraction on a crack mask image to obtain a crack mask feature. The road image and the crack mask image are channel-connected to obtain a connected crack image. The connected crack image is input into the channel attention branch to perform channel attention calculation to obtain a channel attention result. The channel attention result and the road RGB image feature are channel-multiplied to obtain a first road image feature. The channel attention result and the crack mask feature are channel-multiplied to obtain a first crack mask image feature.
4. The method for constructing a road crack depth prediction model with a double-branch structure according to claim 1, characterized in that: The output module includes a road image output branch and a crack mask image output branch. In the road image output branch, the second predicted road image feature is further extracted and then input into the MLP to obtain a road prediction result. In the crack mask image output branch, the second predicted crack mask feature is further extracted and then input into the MLP to obtain a crack mask prediction result. After the road prediction result and the crack mask prediction result are feature added, the MLP outputs the crack depth of each crack in the road image.
5. A device for constructing a road crack depth prediction model with a double-branch structure, characterized in that: include: An acquisition module is used to acquire a road image with annotated crack depths and a crack mask image corresponding to the road image, wherein a crack segmentation module is used to process the road image to obtain a corresponding crack mask image, and the crack mask image only contains crack information in the road image, wherein the crack segmentation module includes a first branch and a second branch, wherein the first segmentation branch directly performs crack segmentation on the road image to obtain a first segmentation result, and the second segmentation branch uses the road image as a guide to denoise the first segmentation result to obtain a second segmentation result, and the second segmentation result is a crack mask image, wherein the first segmentation branch uses a Transformer pyramid structure as an encoder and a multi-layer convolution structure as a decoder to extract and analyze features of the road image to obtain a rough crack mask image, and the second segmentation branch uses an expanded convolution layer and a convolution extraction layer to extract features from the road image and then fuses them to obtain a fusion result, and the fusion result is then sequentially passed through a SENet network and two convolution layers to obtain a guide feature, and finally a guide image filter is used to denoise the first segmentation result based on the guide feature to obtain a second segmentation result; A feature extraction module, used for extracting features from the road image and the crack mask image to obtain a first road image feature and a first crack mask image feature; A global-local attention fusion module, used for performing channel connection on the first road image feature and the first crack mask image feature to obtain a first fusion result, performing global attention calculation and local attention calculation on the first fusion result to obtain a global attention result and a local attention result, then adding the global attention result and the local attention result to obtain a second fusion result, and splitting the second fusion result into a second road image feature and a second crack mask image feature; An output module is used to output the second predicted road image feature and the second predicted crack mask feature to obtain the crack depth of each crack in the road image, construct a loss function based on the output result of the output module and the annotation information, and correct the parameters of the road crack depth prediction model according to the loss function, wherein the road crack depth prediction model includes a crack segmentation module, a feature extraction module, a global-local attention fusion module and an output module.
6. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the method for constructing a road crack depth prediction model with a double-branch structure as described in any one of claims 1-4.
7. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, and the process includes a method for constructing a road crack depth prediction model with a double-branch structure according to any one of claims 1-4.
Citation Information
Patent Citations
Road crack detection method and road crack detection device
CN111127468A
Method and apparatus for identifying concrete crack based on video semantic segmentation technology
US20210319547A1
Road detection method based on internet of vehicles
WO2022083784A1
Pavement crack detection method, medium, and system
WO2024092968A1
