Pavement disease detection network training method and device, equipment and storage medium

Abstract

The application discloses a kind of road surface disease detection network training method, device and equipment and storage medium, belong to road surface disease identification technical field.The method comprises: obtaining multiple road surface sample images and corresponding edge true value matrix and area true value matrix;Road surface sample image is input into original detection network, and the corresponding edge prediction matrix and area prediction matrix are obtained;According to edge true value matrix and edge prediction matrix, obtain edge identification loss function value, and according to area true value matrix and area prediction matrix, obtain area identification loss function value;According to two loss function values, adjust network parameters, and obtain target joint detection network.The application combines edge identification mode and area identification mode, trains and optimizes original detection network from two different angles of edge grid and disease area, so that the network after training can detect disease from different angles, to ensure the detection accuracy of final detection by combining the detection results of different angles.

Description

Technical Field

[0001] The present invention relates to the technical field of pavement defect identification, and in particular to a pavement defect detection network training method, device, equipment and storage medium. Background Art

[0002] In related technologies, a variety of detection methods can be used to collect and identify road surface defects. First, the collection methods mainly include line scanning imaging, surface scanning imaging, laser digital imaging, three-dimensional data imaging, etc. The corresponding detection device is a camera equipped with auxiliary lighting. The recognition method after imaging is based on grayscale recognition method and deep learning method, and recognition and detection are performed based on the frequency of different components in the grayscale image.

[0003] However, in this method, water stains, signs and markings, uneven lighting, and road shadows on the road surface will greatly interfere with the identification of defects. In addition, the reference standards of this method are relatively simple, and the corresponding types of defects that can be detected are also relatively simple. Therefore, at this stage, it is impossible to identify and detect road defects from different angles, and the detection accuracy is low. Summary of the Invention

[0004] The main purpose of the present invention is to provide a road surface disease detection network training method, device, equipment and storage medium, aiming to solve the technical problem of low road surface disease detection accuracy in the existing technology.

[0005] According to a first aspect of the present invention, a method for training a road surface defect detection network is provided, the method comprising:

[0006] Acquire multiple pavement sample images and corresponding sets of edge truth matrices and region truth matrices; each element of the edge truth matrix corresponds to an edge grid, and the value of each element in the edge truth matrix indicates whether the corresponding edge grid has a specified pavement defect; each row of elements in the region truth matrix corresponds to a defect region, and the value of each element in each row of the region truth matrix indicates the coordinates of the circumscribed rectangle of the corresponding defect region; wherein the edge grid is obtained by segmenting the pavement sample image according to a fixed size, and the defect region is obtained based on the minimum circumscribed rectangle of the region in the pavement sample image that belongs to the specified pavement defect;

[0007] Inputting a plurality of the road sample images into the original detection network to obtain corresponding edge prediction matrices and region prediction matrices;

[0008] Obtaining an edge recognition loss function value according to the edge truth matrix and the edge prediction matrix, and obtaining a region recognition loss function value according to the region truth matrix and the region prediction matrix;

[0009] According to the edge recognition loss function value and the region recognition loss function value, the network parameters of the original detection network are adjusted to obtain a target joint detection network.

[0010] Optionally, obtaining an edge recognition loss function value according to the edge truth matrix and the edge prediction matrix, and obtaining a region recognition loss function value according to the region truth matrix and the region prediction matrix, includes:

[0011] The edge recognition loss function value is obtained according to the first formula, the edge truth matrix, and the edge prediction matrix, and the region recognition loss function value is obtained according to the second formula, the region truth matrix, and the region prediction matrix; wherein the first formula is:

[0012]

[0013] Among them, loss1 is the edge recognition loss function value, is the matrix vector of the edge truth matrix, is the matrix vector of the edge prediction matrix, and ε is a preset constant to avoid the denominator being 0;

[0014] The second formula is:

[0015]

[0016] Among them, loss2 is the region recognition loss function value, is the matrix vector of the region truth matrix, is the matrix vector of the region prediction matrix.

[0017] Optionally, adjusting the network parameters of the original detection network according to the edge recognition loss function value and the region recognition loss function value to obtain the target joint detection network includes:

[0018] Obtaining a total loss function value according to the edge recognition loss function value and the region recognition loss function value;

[0019] Adjusting the network parameters according to the total loss function value and updating the original detection network;

[0020] Returning to the step of inputting the plurality of road sample images into the original detection network, obtaining corresponding edge prediction matrices and region prediction matrices, and obtaining corresponding total loss function values, until the total loss function value reaches a minimum;

[0021] The network parameters are adjusted according to the minimum total loss function value to obtain the target joint detection network.

[0022] Optionally, adjusting the network parameters and updating the original detection network according to the total loss function value includes:

[0023] According to the third formula and the total loss function value, adjusting the network parameters and updating the original detection network;

[0024] Wherein, the third formula is:

[0025]

[0026] Among them, w i+1 is the adjusted network parameter, w i are the network parameters before adjustment, is the arithmetic mean of the total loss function values ​​corresponding to the multiple road sample images, and η is a constant coefficient.

[0027] Optionally, after adjusting the network parameters of the original detection network according to the edge recognition loss function value and the region recognition loss function value to obtain the target joint detection network, the method further includes:

[0028] Acquire an image of the road surface to be tested;

[0029] Inputting the road surface image to be tested into the target joint detection network to obtain a corresponding edge recognition result matrix and a region recognition result matrix;

[0030] A pavement defect detection result of the road surface image to be tested is obtained according to the edge recognition result matrix and the region recognition result matrix.

[0031] Optionally, inputting the road surface image to be tested into the target joint detection network to obtain a corresponding edge recognition result matrix and a region recognition result matrix includes:

[0032] The road surface image to be tested is input into the target joint detection network, so that the target joint detection network divides the road surface image to be tested into multiple edge grids and at least one defective area, and generates the edge recognition result matrix based on the multiple edge grids, and generates the area recognition result matrix based on the at least one defective area.

[0033] Optionally, obtaining the pavement defect detection result of the road surface image to be tested according to the edge recognition result matrix and the region recognition result matrix includes:

[0034] Obtaining matrix element values ​​in the edge recognition result matrix and the region recognition result matrix;

[0035] The pavement damage detection result is obtained according to the matrix element values.

[0036] According to a second aspect of the present invention, a road surface defect detection network training device is provided, the device comprising:

[0037] An acquisition module is configured to acquire multiple pavement sample images and corresponding sets of edge truth matrices and region truth matrices; each element of the edge truth matrix corresponds to an edge grid, and the value of each element in the edge truth matrix indicates whether the corresponding edge grid contains a specified pavement defect; each row of elements in the region truth matrix corresponds to a defect region, and the value of each element in each row of the region truth matrix indicates the coordinates of the circumscribed rectangle of the corresponding defect region; wherein the edge grids are obtained by segmenting the pavement sample images according to a fixed size, and the defect region is obtained based on the minimum circumscribed rectangle of the region in the pavement sample image that contains the specified pavement defect;

[0038] A prediction module, configured to input a plurality of the road sample images into an original detection network to obtain corresponding edge prediction matrices and region prediction matrices;

[0039] a calculation module, configured to obtain an edge recognition loss function value based on the edge truth matrix and the edge prediction matrix, and to obtain a region recognition loss function value based on the region truth matrix and the region prediction matrix;

[0040] The optimization module is used to adjust the network parameters of the original detection network according to the edge recognition loss function value and the region recognition loss function value to obtain a target joint detection network.

[0041] According to a third aspect of the present invention, a pavement disease detection network training device is provided, comprising: a memory, a processor, and a pavement disease detection network training program stored in the memory and executable on the processor, wherein the pavement disease detection network training program, when executed by the processor, implements the various steps described in any possible implementation of the first aspect.

[0042] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a pavement defect detection network training program is stored. When the pavement defect detection network training program is executed by a processor, the various steps described in any possible implementation of the first aspect are implemented.

[0043] An embodiment of the present invention proposes a pavement defect detection network training method, apparatus, device and storage medium, wherein a plurality of pavement sample images and a corresponding plurality of sets of edge truth matrices and region truth matrices are obtained through the pavement defect detection network training device; the plurality of pavement sample images are input into the original detection network to obtain corresponding edge prediction matrices and region prediction matrices; an edge recognition loss function value is obtained based on the edge truth matrix and the edge prediction matrix, and a region recognition loss function value is obtained based on the region truth matrix and the region prediction matrix; and the network parameters of the original detection network are adjusted based on the edge recognition loss function value and the region recognition loss function value to obtain a target joint detection network.

[0044] The present invention combines the edge recognition method and the area recognition method, and trains and optimizes the original detection network from two different perspectives: edge grid and diseased area. This allows the trained target joint detection network to perform disease detection from different angles, and to combine the detection results from different angles to ensure the final detection accuracy. In this way, when performing subsequent pavement disease detection, the target joint detection network can be used to detect strip-shaped diseases using edge grids, and to detect regional diseases using rectangular areas. This allows the two to be combined for comprehensive identification to obtain more comprehensive detection results, thereby improving the accuracy of pavement disease detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0046] Figure 1 A schematic diagram of the structure of a road surface disease detection network training device in the hardware operating environment involved in an embodiment of the present invention;

[0047] Figure 2 This is a flow chart of a first embodiment of a road surface disease detection network training method according to the present invention;

[0048] Figure 3 For the present invention Figure 2 A detailed flow chart of step S204;

[0049] Figure 4 For the present invention Figure 2 Schematic diagram of the process after step S204;

[0050] Figure 5Schematic diagram of the functional modules of the network training device for pavement defect detection according to an embodiment of the present invention;

[0051] Figure 6 A schematic diagram of manually marking a grid with cracked pavement defects according to an embodiment of the present invention;

[0052] Figure 7 A schematic diagram of a grid with cracked pavement damage obtained by a grayscale image-based pattern recognition method according to an embodiment of the present invention;

[0053] Figure 8 Schematic diagram of a grid with cracked pavement damage obtained by adopting the method of the present invention according to an embodiment of the present invention;

[0054] Figure 9 A schematic diagram of manually marking a grid with cracked pavement defects according to an embodiment of the present invention;

[0055] Figure 10 A schematic diagram of a grid with cracked pavement defects obtained by a grayscale image-based pattern recognition method according to an embodiment of the present invention;

[0056] Figure 11 Schematic diagram of a grid with cracked pavement damage obtained by adopting the method of the present invention according to an embodiment of the present invention;

[0057] Figure 12 This is an original result map for identifying areas with cracked pavement defects according to an embodiment of the present invention.

[0058] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0059] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0060] The main solution of the embodiment of the present invention is: obtaining multiple road sample images and corresponding multiple sets of edge truth matrices and region truth matrices; inputting the multiple road sample images into the original detection network to obtain corresponding edge prediction matrices and region prediction matrices; obtaining edge recognition loss function values ​​based on the edge truth matrix and the edge prediction matrix, and obtaining region recognition loss function values ​​based on the region truth matrix and the region prediction matrix; adjusting the network parameters of the original detection network based on the edge recognition loss function values ​​and the region recognition loss function values ​​to obtain a target joint detection network.

[0061] In related technologies, a variety of detection methods can be used to collect and identify road surface defects. First, the collection methods mainly include line scanning imaging, surface scanning imaging, laser digital imaging, and three-dimensional data imaging. The corresponding detection device is a camera equipped with auxiliary lighting. The post-imaging recognition method is based on grayscale recognition and deep learning methods, and the recognition and detection are based on the frequency of different components in the grayscale image. However, in this method, water stains, signs and markings, uneven lighting, and shadows on the road surface will greatly interfere with the identification of defects. In addition, the reference standard of this method is relatively simple, and the corresponding types of defects that can be detected are also relatively simple. The road surface defects are identified and detected from different angles, and the detection accuracy is low.

[0062] The present invention provides a solution for a pavement defect detection network training device, which combines edge recognition and area recognition methods, and trains and optimizes the original detection network from two different perspectives: edge grid and defect area. This allows the trained target joint detection network to perform defect detection from different angles, and combines the detection results from different angles to ensure the final detection accuracy. In this way, when subsequently performing pavement defect detection, the target joint detection network can be used to detect strip-shaped defects using edge grids, and to detect regional defects using rectangular areas. This allows the two to be combined for comprehensive identification to obtain more comprehensive detection results, thereby improving the accuracy of pavement defect detection.

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0064] The terms "first" and "second" in the description and claims of the embodiments of the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0065] Reference Figure 1 , Figure 1 This is a structural diagram of a road surface disease detection network training device in the hardware operating environment involved in an embodiment of the present invention.

[0066] like Figure 1As shown, the pavement defect detection network training device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a wireless fidelity (WI-FI) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0067] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the pavement defect detection network training device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0068] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, an acquisition module, a processing module, an optimization module, and a pavement disease detection network training program, wherein the processing module may be further divided into a prediction module and a calculation module.

[0069] exist Figure 1 In the pavement disease detection network training device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the pavement disease detection network training device of the present invention can be set in the pavement disease detection network training device, and the pavement disease detection network training device calls the pavement disease detection network training program stored in the memory 1005 through the processor 1001, and executes the pavement disease detection network training method provided by the embodiment of the present invention.

[0070] Based on the above hardware structure but not limited to the above hardware structure, the present invention provides a first embodiment of a road surface disease detection network training method. Figure 2 , Figure 2 Schematic diagram of the flow of the first embodiment of the road surface disease detection network training method of the present invention.

[0071] In this embodiment, the method includes:

[0072] Step S201, obtaining multiple road sample images and corresponding multiple sets of edge truth matrices and region truth matrices;

[0073] In this embodiment, the execution entity is a pavement defect detection network training device. This pavement defect detection network training device can be a PC or a dedicated electronic device with network model training capabilities, but this embodiment does not impose any restrictions. Furthermore, the multiple road sample images have the same resolution, namely, a standard resolution. This standard resolution can be set based on actual needs. It is understood that the resolution required by the target joint detection network is the standard resolution. If the resolutions are different, they can be converted to the standard resolution through scaling, grayscale padding, and other methods.

[0074] Every road sample image has a corresponding edge truth matrix and region truth matrix. These two matrices can be understood as the actual parameters of the corresponding road sample image. They are obtained through manual identification and marking to ensure the accuracy of the actual parameters. The following is a detailed description of these two matrices.

[0075] For the edge truth matrix, each element corresponds to an edge grid, and the specific value of each element indicates whether the corresponding edge grid has the specified pavement disease: if the edge grid has the specified pavement disease, the element of the edge truth matrix corresponding to the edge grid is marked as the first label value, otherwise the element of the edge truth matrix corresponding to the edge grid is marked as the second label value. Generally, in order to facilitate the processing of the pavement disease detection network training equipment, the first label value can be recorded as 1 and the second label value can be recorded as 0. Among them, the edge grid is obtained by segmenting the pavement sample image according to a fixed size. In other words, the value of the elements contained in the edge truth matrix is ​​either 1 or 0. If it is 1, it means that the corresponding edge grid has the specified pavement disease, and if it is 0, it means that the corresponding edge grid does not have the specified pavement disease, and different edge grids have the same shape, size, and dimension.

[0076] For the regional truth value matrix, each row element corresponds to a diseased area, and the value of the element in each row represents the coordinates of the circumscribed rectangle of the corresponding diseased area. It can be understood that the number of rows in the regional truth value matrix of each road sample image is the same as the number of specified road diseased areas in the image. For example, if the number of specified road diseased areas in the road sample image is 3, then the number of rows in the regional truth value matrix is ​​also 3. Since the location of the diseased area is determined by the circumscribed rectangle in this embodiment, in order to determine a diseased area, it is only necessary to determine the diagonal coordinates of the area. In this embodiment, the position of the corresponding area can be represented by the coordinate values ​​of the upper left corner and lower right corner of the minimum circumscribed matrix of the diseased area. Since the coordinates of a point include horizontal and vertical coordinates, each row requires four values ​​to represent a diseased area, that is, the number of columns in the regional truth value matrix is ​​generally 4. The coordinate values ​​mentioned here refer to the coordinate values ​​normalized to 0 to 1. That is to say, the number of rows of the regional truth value matrix is ​​the number of diseased areas, and the number of columns is generally 4. The values ​​of the elements it contains are between 0 and 1. They are obtained by normalizing the coordinates of the minimum bounding rectangle of the manually marked designated diseased areas. The shapes, sizes, and dimensions of the minimum bounding rectangles of different diseased areas may be the same or different.

[0077] From the above description, it is not difficult to understand that the edge recognition method corresponding to the edge truth matrix processes each small edge grid separately, so that the details of different small areas can be detected and judged separately, thereby accurately representing the trend, line, and path of the pavement disease. Therefore, the network model obtained through this training, that is, the target joint detection network, can accurately detect strip-type pavement diseases, such as cracks. On the other hand, the region recognition method corresponding to the region truth matrix processes the entire disease area. It does not focus on the details of the disease situation within the area (i.e., the trend line, etc.), but instead marks the entire block area as a whole. Therefore, the target joint detection network obtained through this training can also accurately detect block-type regional diseases, such as cracks. Therefore, based on this, it should be clear that the edge training and subsequent detection, as well as the region training and subsequent detection involved in this embodiment, are not simply differences in size. Instead, based on the different principles and characteristics described above, network training can be performed according to different types of pavement diseases, thereby obtaining a more comprehensive and more accurate network model. The corresponding target joint detection network obtained subsequently can also accurately detect different types of pavement diseases, improving the comprehensiveness and accuracy of disease detection.

[0078] Step S202: inputting a plurality of road sample images into the original detection network to obtain corresponding edge prediction matrices and region prediction matrices;

[0079] For any road sample image, input it into the original detection network to obtain the corresponding edge prediction matrix and region prediction matrix. It can be understood that the resolution of the road sample image needs to be converted to the above-mentioned standard resolution. The road sample image acts as a training set, so that parameter adjustment and network optimization can be performed based on the output results. Specifically, after the road sample image is input into the original detection network, the edge recognition branch of the original detection network can divide the road sample image into multiple edge grids and obtain the corresponding edge prediction matrix, where each element of the edge prediction matrix corresponds to each edge grid. The region recognition branch of the original detection network can divide the road sample image into multiple specified diseased areas and obtain the corresponding region prediction matrix, where each row element of the region prediction matrix corresponds to each diseased area.

[0080] Each element in the edge prediction matrix and the edge truth matrix corresponds to each edge grid, ensuring that the number of rows and columns of the edge prediction matrix and the edge truth matrix are the same, facilitating the accuracy of the edge recognition branch's recognition results. The value of each element in the edge prediction matrix indicates whether each edge grid has a specified pavement defect. When the edge recognition branch of the original detection network identifies an edge grid as having a specified pavement defect, the element in the prediction matrix corresponding to that edge grid is labeled with a first label value; otherwise, the element in the prediction matrix corresponding to that edge grid is labeled with a second label value. Specifically, after a road sample image is input into the original detection network, the edge recognition branch outputs the calculated value for each edge grid in the form of a matrix. Each calculated value is compared with a preset judgment threshold. This judgment threshold is an empirical value ranging from 0 to 1, typically 0.5. When the calculated value of the edge grid is less than the judgment threshold, the element in the edge prediction matrix corresponding to that edge grid is assigned the second label value. When the calculated value of the edge grid is not less than the judgment threshold, the element in the edge prediction matrix corresponding to that edge grid is assigned the first label value, ultimately outputting the edge prediction matrix. Generally, the first label value can be recorded as 1, and the second label value can be recorded as 0.

[0081] The number of rows in the region prediction matrix and the region truth matrix corresponds to the number of designated diseased areas predicted by the region recognition branch of the original detection network and the number of designated diseased areas manually marked in the road sample image, respectively. The number of columns in the region prediction matrix and the region truth matrix are both 4, which makes it easier to determine whether the recognition results of the joint network region recognition branch are accurate.

[0082] It should be noted that the structure of the original detection network can be based on the Yolo network, using the Yolo network's backbone network as the original detection network's backbone network. The output of the Yolo network serves as the region recognition branch of the original detection network, outputting a region prediction matrix representing the region recognition results. Another branch network is added as the edge recognition branch of the original detection network, outputting an edge prediction matrix representing the edge recognition results. Therefore, the original detection network can simultaneously perform edge recognition and region recognition on specified pavement defects in pavement sample images. Furthermore, the Yolo network is a convolutional neural network that can predict multiple locations and categories at once, enabling end-to-end target detection and recognition. Its greatest advantage is its speed. Therefore, when the trained network is subsequently used for detection, in addition to the aforementioned improvements in detection accuracy, detection efficiency can also be significantly improved.

[0083] Among them, the backbone network layers of the Yolo network used to extract features include: 1. BN layer (batch normalization layer), which normalizes the feature data extracted from the input road sample image and unifies the mean and variance of the data; 2. Convolutional layer, which performs the inner product between the image and the filter operator. Specifically, different filter operators will obtain different output data, which are used to extract different features of the image, such as color features and contour features. The convolution operation multiplies the feature map with the corresponding value of the filter operator and calculates the sum. The activation function ReLu (Rectified Linear Units) is used to determine whether the current neuron is activated or not; 3. Pooling layer, which reduces the size of the feature map and increases the model's receptive area.

[0084] Step S203: obtaining an edge recognition loss function value based on the edge truth matrix and the edge prediction matrix, and obtaining a region recognition loss function value based on the region truth matrix and the region prediction matrix;

[0085] As mentioned above, the role of the road sample image is equivalent to the training set, so that parameter adjustment and network optimization can be performed according to the output results. The main purpose of this embodiment is to obtain a network model with higher accuracy. Therefore, there will inevitably be errors between the prediction matrix calculated by the above-mentioned original detection network and the true value matrix obtained by manual labeling, that is, the above-mentioned loss function value. Therefore, in order to optimize the original detection network, it is necessary to first calculate the corresponding loss function value.

[0086] Specifically, the edge recognition loss function value can be calculated using the first formula, and the region recognition loss function value can be calculated using the second formula.

[0087] Among them, the first formula is as follows:

[0088]

[0089] The second formula is as follows:

[0090]

[0091] Among them, loss1 is the edge recognition loss function value, is the matrix vector of the marginal truth matrix, is the matrix vector of the edge prediction matrix, ε is a preset constant to avoid the denominator being 0, which is generally a small real number, and the default value can be 10 -5 ; loss2 is the region recognition loss function value, is the matrix vector of the regional truth matrix, is the matrix vector of the region prediction matrix.

[0092] Step S204: adjusting the network parameters of the original detection network according to the edge recognition loss function value and the region recognition loss function value to obtain a target joint detection network.

[0093] After obtaining the above-mentioned edge recognition loss function value and region recognition loss function value, it is equivalent to obtaining the error of the prediction result of the original detection network relative to the actual situation. The network parameters of the original detection network can be adjusted accordingly to achieve the purpose of optimizing the network model and obtain a more accurate target joint detection network. Among them, the network parameters refer to the inherent parameters of the network layer contained in the detection network. When the network structure of the selected network is determined, the network parameters of each network layer are determined, that is, which parameters to be used can be determined. For example, the network parameters contained in the above-mentioned BN layer, convolution layer, etc. are all well-known and will not be described in detail in this embodiment.

[0094] In one embodiment, referring to Figure 3 , Figure 3 For the present invention Figure 2 The detailed flow diagram of step S204 in FIG. 1 is a flow diagram of adjusting the network parameters of the original detection network according to the edge recognition loss function value and the region recognition loss function value to obtain the target joint detection network, including:

[0095] Step A10, obtaining a total loss function value according to the edge recognition loss function value and the region recognition loss function value;

[0096] It should be noted that in this embodiment, each road sample image is trained and tested. That is, each road sample image corresponds to a set of edge recognition loss function values ​​and region recognition loss function values. After obtaining the edge recognition loss function values ​​and region recognition loss function values, the two are added together to obtain the total loss function value, i.e., total loss function value loss = loss1 + loss2.

[0097] Step A20: adjusting the network parameters according to the total loss function value and updating the original detection network;

[0098] After obtaining the total loss function value corresponding to each road sample image, the network parameters can be optimized and adjusted based on all these total loss function values. Specifically, the adjusted network parameters can be calculated using the third formula. The third formula is as follows:

[0099]

[0100] Among them, w i+1 is the adjusted network parameter, w i are the network parameters before adjustment, is the arithmetic mean of the total loss function values ​​corresponding to all road sample images, and η is a constant coefficient. It can be understood that w i+1 and w i It should be noted that the initial values ​​of the network parameters are randomly generated, and in one optimization process, all network parameters of the network are optimized and updated.

[0101] Step A30, returning to the step of inputting the plurality of road sample images into the original detection network to obtain corresponding edge prediction matrices and region prediction matrices, and obtaining corresponding total loss function values, until the total loss function value reaches a minimum;

[0102] Step A40: Adjust the network parameters according to the minimum total loss function value to obtain the target joint detection network.

[0103] To achieve higher accuracy, network models often require multiple training cycles to meet requirements. Therefore, each time the network parameters are optimized and updated, all road sample images are re-input into the detection network for specific road defect recognition. As the number of optimization updates increases, the arithmetic mean of the loss function for all road sample images decreases. When the arithmetic mean of the loss function stabilizes, increasing the number of optimizations and the number of road sample images will not significantly improve the recognition accuracy of the joint network, indicating that the network has begun to converge and that the optimal network parameters can be obtained within a limited number of optimization cycles. Therefore, through continuous optimization until the arithmetic mean of the loss function for all road sample images is minimized, indicating that the network has reached optimality, the resulting network is the target joint detection network. This target joint detection network has higher accuracy and is conducive to more accurate detection results in subsequent disease detection.

[0104] Finally, it should be noted that through the above training and optimization process, it should be understood that for the same target joint detection network, when the designated pavement defects are different, the network parameters of the optimized target joint detection network will be different, so that the final optimized joint detection network can identify different types of designated pavement defects. When a specific pavement defect needs to be identified, the target joint detection network optimized for that pavement defect can be used for identification.

[0105] To further improve efficiency, this embodiment can also set the number of optimization updates. When the number of optimization updates is reached, optimization is no longer performed. At the current number of optimization updates, the target joint detection network corresponding to the network parameters that minimize the arithmetic mean of the loss function calculated for all road sample images is determined as the final network.

[0106] In this embodiment, the accuracy of the target joint detection network in identifying pavement defects is measured by a loss function, so that the network parameters of the joint network can be continuously optimized until the optimal edge recognition and region recognition joint network is obtained, making the network's recognition results increasingly accurate; and, the target joint detection network provided in this embodiment enables subsequent formal detection to be conducted simultaneously from different angles for comprehensive detection, and can simultaneously detect different types of pavement defects, which is not only conducive to improving the comprehensiveness and accuracy of subsequent detection, but also conducive to improving the efficiency of subsequent detection.

[0107] Further, as an embodiment, refer to Figure 4 , Figure 4 For the present invention Figure 2In the flowchart after step S204, after adjusting the network parameters of the original detection network according to the edge recognition loss function value and the region recognition loss function value to obtain the target joint detection network, the method further includes:

[0108] Step S401, obtaining an image of the road surface to be tested;

[0109] Step S402: inputting the road surface image to be tested into the target joint detection network to obtain a corresponding edge recognition result matrix and a region recognition result matrix;

[0110] After the target joint detection network is trained and optimized as described above, the target joint detection network can be used for formal detection. Specifically, after obtaining the road surface image to be tested, it is input into the target joint detection network. The edge recognition branch of the target joint detection network divides it into multiple edge grids of the same shape and size according to a fixed size, and then outputs the calculated value of each edge grid through calculation (generally output in the form of a matrix). Among them, the target joint detection network compares the calculated value of each edge grid with a preset judgment threshold. The judgment threshold is an empirical value, ranging from 0 to 1, usually 0.5. When the calculated value of the edge grid is less than the judgment threshold, the element of the edge recognition result matrix corresponding to the edge grid takes the second label value; when the calculated value of the edge grid is not less than the judgment threshold, the element of the edge recognition result matrix corresponding to the edge grid takes the first label value; thereby finally outputting the edge recognition result matrix. Generally, the first label value can be recorded as 1, and the second label value can be recorded as 0. At the same time, the region identification branch of the target joint detection network outputs the minimum bounding rectangle coordinate vector of each diseased area. This coordinate vector consists of the horizontal and vertical normalized coordinates of the upper left and lower right corners of the minimum bounding rectangle, and its value range is 0 to 1. It should be noted that in actual applications, there may be cases where there are only strip-type diseases but no regional diseases. In this case, the number of diseased areas is 0, and the corresponding region identification result matrix is ​​an empty matrix.

[0111] Step S403 : obtaining a pavement defect detection result of the road surface image to be tested according to the edge recognition result matrix and the region recognition result matrix.

[0112] For the edge recognition result matrix, the elements it contains are either 1 or 0. Since the first label value, i.e. 1, indicates that the specified pavement disease exists within the range of the edge grid corresponding to the element, all edge grids corresponding to the element value of 1 can be filtered out as the final edge detection result. It can be understood that the positional relationship between the multiple filtered edge grids can reflect the trend, line, shape, path, etc. of the pavement disease, so as to realize the detection of strip-type diseases.

[0113] The region recognition matrix contains elements that correspond to the coordinates of the top-left and bottom-right corners of the minimum bounding rectangle of a defect region. Vertically, the number of rows represents the number of defect regions. Therefore, the region recognition matrix can represent the locations of all defect regions, representing a region as a whole to detect block-type (i.e., region-type) defects and obtain regional detection results. Finally, the edge detection results and regional detection results are combined to form the aforementioned pavement defect detection results.

[0114] It should be noted that if the same identified grid has multiple pavement diseases or there are overlapping areas between the identified disease areas of different categories, the grid or area with the pavement disease with the highest priority will be determined based on the priority of each pavement disease.

[0115] Because when optimizing the detection network, the values ​​of the optimized network parameters are different according to different pavement diseases. Therefore, when using different optimized target joint detection networks to detect corresponding pavement diseases, different detection networks may identify different pavement diseases for the same grid. For example, a grid is identified as having cracks and repaired pavement diseases. At this time, it is only necessary to determine whether the grid has one pavement disease. Therefore, according to the priority of the pavement disease, the pavement disease with the highest priority is the pavement disease that the grid ultimately has. This priority is generally obtained based on experience. Different pavement roads have different types of pavement diseases. For example, the priority is ranked as follows:

[0116] 1) Asphalt pavement defects: cracks > fissures > potholes > looseness > subsidence > rutting > ripples > oil spills > repairs.

[0117] 2) For cement pavement defects: broken slabs > broken slab corners > misalignment > mud pumping > peeling edges and corners > damaged joints > potholes > exposed surfaces > repairs.

[0118] For example, when a certain grid of an asphalt pavement is identified as having cracks and repairs, the grid is prioritized as having cracked pavement disease.

[0119] In this embodiment, the target joint detection network obtained by the above training is used to perform disease detection. It can simultaneously perform comprehensive detection from different angles, thereby detecting different types of road diseases at the same time, which is not only conducive to improving the comprehensiveness and accuracy of subsequent detection, but also conducive to improving the efficiency of subsequent detection.

[0120] Finally, the results of pavement defect identification using the detection method described in the above embodiment were compared with those obtained using existing grayscale image-based pattern recognition methods. For example, in predicting cracks and fissures on asphalt pavement, the comparison of recognition efficiency and accuracy between the grayscale image-based pattern recognition method and the recognition method of the present invention is shown in Table 1. This recognition accuracy was evaluated using a similarity metric commonly used in the field of deep learning.

[0121] Table 1 compares the results of the traditional pattern recognition method and the method involved in this embodiment:

[0122]

[0123] As can be seen from Table 1, the method involved in this embodiment has greatly improved recognition efficiency and accuracy compared with traditional grayscale image-based pattern recognition methods, and its accuracy is close to that of manual labeling.

[0124] like Figures 6-8 The following are schematic diagrams of grids with cracked pavement defects obtained by manual labeling, traditional grayscale-based pattern recognition methods, and the target joint detection network involved in this embodiment. The black squares in the figure represent grids with cracked pavement defects. Figures 6-8 It can be seen that compared with the pattern recognition method based on grayscale images, the detection method of this embodiment can detect more cracked pavement diseases, is closer to the results of manual marking, and has more accurate recognition results.

[0125] like Figures 9-12 As shown in the figure, there are schematic diagrams of the grid with cracked pavement disease obtained by manual marking, traditional grayscale-based pattern recognition method and target joint detection network according to the present invention, and the original result map of regional recognition with cracked pavement disease. It can be seen that there is no grid in the original result map of regional recognition, and it is necessary to use the above method to convert it into a recognition map with a grid, that is, Figure 11 , in order to achieve accurate identification of the disease, wherein the grid in the detection result diagram of this embodiment is obtained by transforming the rectangular frame of the identification area. The black squares in the figure represent the grid with cracked pavement disease. Figures 9-11 It can be seen that, compared with the pattern recognition method based on grayscale images, the detection method of this embodiment is closer to the result of manual marking, and the recognition result is more accurate.

[0126] In summary, the above specific embodiments also show that the method of the embodiment of the present invention can accurately identify road surface defects.

[0127] Based on the same inventive concept, the embodiment of the present invention also provides a road disease detection network training device, referring to Figure 5 Shown, including:

[0128] An acquisition module is configured to acquire multiple pavement sample images and corresponding sets of edge truth matrices and region truth matrices; each element of the edge truth matrix corresponds to an edge grid, and the value of each element in the edge truth matrix indicates whether the corresponding edge grid contains a specified pavement defect; each row of elements in the region truth matrix corresponds to a defect region, and the value of each element in each row of the region truth matrix indicates the coordinates of the circumscribed rectangle of the corresponding defect region; wherein the edge grids are obtained by segmenting the pavement sample images according to a fixed size, and the defect region is obtained based on the minimum circumscribed rectangle of the region in the pavement sample image that contains the specified pavement defect;

[0129] A prediction module, configured to input a plurality of the road sample images into an original detection network to obtain corresponding edge prediction matrices and region prediction matrices;

[0130] a calculation module, configured to obtain an edge recognition loss function value based on the edge truth matrix and the edge prediction matrix, and to obtain a region recognition loss function value based on the region truth matrix and the region prediction matrix;

[0131] The optimization module is used to adjust the network parameters of the original detection network according to the edge recognition loss function value and the region recognition loss function value to obtain a target joint detection network.

[0132] It should be noted that the various implementations of the pavement disease detection network training device in this embodiment and the technical effects achieved can refer to the various implementations of the pavement disease detection network training method in the aforementioned embodiments, and will not be repeated here.

[0133] In addition, in one embodiment, the present application further provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the aforementioned method embodiment are implemented.

[0134] In some embodiments, the computer-readable storage medium may be a memory device such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface mount memory, optical disk, or CD-ROM; or various devices including any one or any combination of the above memories. The computer may be various computing devices including smart terminals and servers.

[0135] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0136] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as, for example, one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0137] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0138] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A road surface disease detection network training method, characterized in that: The method comprises: Acquire multiple pavement sample images and corresponding sets of edge truth matrices and region truth matrices; each element of the edge truth matrix corresponds to an edge grid, and the value of each element in the edge truth matrix indicates whether the corresponding edge grid has a specified pavement defect; each row of elements in the region truth matrix corresponds to a defect region, and the value of each element in each row of the region truth matrix indicates the coordinates of the circumscribed rectangle of the corresponding defect region; wherein the edge grid is obtained by segmenting the pavement sample image according to a fixed size, and the defect region is obtained based on the minimum circumscribed rectangle of the region in the pavement sample image that belongs to the specified pavement defect; Inputting a plurality of the road sample images into the original detection network to obtain corresponding edge prediction matrices and region prediction matrices; Obtaining an edge recognition loss function value according to the edge truth matrix and the edge prediction matrix, and obtaining a region recognition loss function value according to the region truth matrix and the region prediction matrix; Adjusting the network parameters of the original detection network according to the edge recognition loss function value and the region recognition loss function value to obtain a target joint detection network; After adjusting the network parameters of the original detection network according to the edge recognition loss function value and the region recognition loss function value to obtain the target joint detection network, the method further includes: Acquire an image of the road surface to be tested; Inputting the road surface image to be tested into the target joint detection network to obtain a corresponding edge recognition result matrix and a region recognition result matrix; A pavement defect detection result of the road surface image to be tested is obtained based on the edge recognition result matrix and the region recognition result matrix. If multiple pavement defects are identified in the same grid, or if there are overlapping areas between defect areas of different categories, the pavement defect with the highest priority is used as the pavement defect detection result, and the priority ranking of each pavement defect is different for different categories of road surfaces.

2. The method according to claim 1, characterized in that Obtaining an edge recognition loss function value according to the edge truth matrix and the edge prediction matrix, and obtaining a region recognition loss function value according to the region truth matrix and the region prediction matrix, includes: The edge recognition loss function value is obtained according to the first formula, the edge truth matrix, and the edge prediction matrix, and the region recognition loss function value is obtained according to the second formula, the region truth matrix, and the region prediction matrix; wherein the first formula is: Among them, loss1 is the edge recognition loss function value, is the matrix vector of the edge truth matrix, is the matrix vector of the edge prediction matrix, and ε is a preset constant to avoid the denominator being 0; The second formula is: Among them, loss2 is the region recognition loss function value, is the matrix vector of the region truth matrix, is the matrix vector of the region prediction matrix.

3. The method according to claim 1, characterized in that The adjusting the network parameters of the original detection network according to the edge recognition loss function value and the region recognition loss function value to obtain the target joint detection network includes: Obtaining a total loss function value according to the edge recognition loss function value and the region recognition loss function value; Adjusting the network parameters according to the total loss function value and updating the original detection network; Returning to the step of inputting the plurality of road sample images into the original detection network, obtaining corresponding edge prediction matrices and region prediction matrices, and obtaining corresponding total loss function values, until the total loss function value reaches a minimum; The network parameters are adjusted according to the minimum total loss function value to obtain the target joint detection network.

4. The method according to claim 3, characterized in that The adjusting the network parameters according to the total loss function value and updating the original detection network includes: According to the third formula and the total loss function value, adjusting the network parameters and updating the original detection network; Wherein, the third formula is: Among them, w i+1 is the adjusted network parameter, w i are the network parameters before adjustment, is the arithmetic mean of the total loss function values ​​corresponding to the multiple road sample images, and η is a constant coefficient.

5. The method according to claim 1, wherein The step of inputting the road surface image to be tested into the target joint detection network to obtain a corresponding edge recognition result matrix and a region recognition result matrix includes: The road surface image to be tested is input into the target joint detection network, so that the target joint detection network divides the road surface image to be tested into multiple edge grids and at least one defective area, and generates the edge recognition result matrix based on the multiple edge grids, and generates the area recognition result matrix based on the at least one defective area.

6. The method according to claim 1, characterized in that Obtaining a pavement defect detection result of the road surface image to be tested according to the edge recognition result matrix and the region recognition result matrix includes: Obtaining matrix element values ​​in the edge recognition result matrix and the region recognition result matrix; The pavement damage detection result is obtained according to the matrix element values.

7. A road surface disease detection network training device, characterized in that: The device comprises: An acquisition module is configured to acquire multiple pavement sample images and corresponding sets of edge truth matrices and region truth matrices; each element of the edge truth matrix corresponds to an edge grid, and the value of each element in the edge truth matrix indicates whether the corresponding edge grid contains a specified pavement defect; each row of elements in the region truth matrix corresponds to a defect region, and the value of each element in each row of the region truth matrix indicates the coordinates of the circumscribed rectangle of the corresponding defect region; wherein the edge grids are obtained by segmenting the pavement sample images according to a fixed size, and the defect region is obtained based on the minimum circumscribed rectangle of the region in the pavement sample image that contains the specified pavement defect; A prediction module, configured to input a plurality of the road sample images into an original detection network to obtain corresponding edge prediction matrices and region prediction matrices; a calculation module, configured to obtain an edge recognition loss function value based on the edge truth matrix and the edge prediction matrix, and to obtain a region recognition loss function value based on the region truth matrix and the region prediction matrix; an optimization module, configured to adjust network parameters of the original detection network according to the edge recognition loss function value and the region recognition loss function value, to obtain a target joint detection network; The pavement disease detection network training device is also used to obtain a pavement image to be tested; input the pavement image to be tested into the target joint detection network to obtain a corresponding edge recognition result matrix and a region recognition result matrix; obtain a pavement disease detection result of the pavement image to be tested based on the edge recognition result matrix and the region recognition result matrix; wherein, if the same grid is identified as having multiple pavement diseases, or there are overlapping areas between different types of disease areas, the pavement hazard with the highest priority is used as the pavement disease detection result, and the priority ranking of each pavement hazard is different for different types of pavements.

8. A road surface disease detection network training device, characterized in that: The method comprises a memory, a processor, and a pavement defect detection network training program stored in the memory and executable on the processor. When the pavement defect detection network training program is executed by the processor, the steps of the pavement defect detection network training method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a pavement defect detection network training program, which, when executed by a processor, implements the steps of the pavement defect detection network training method according to any one of claims 1 to 6.

Images