Crack detection method, device and computer equipment based on joint detection

Abstract

The application relates to a crack detection method and device based on joint detection, a computer device, a storage medium and a computer program product. The method comprises the following steps: obtaining a target image of an object to be detected; inputting the target image into a pre-trained crack joint detection network to obtain a first detection result and a second detection result; the first detection result is a crack identification result corresponding to each detection region in the target image; the crack identification result comprises a crack and no crack; the second detection result is a crack prediction frame comprising a category prediction confidence and position information; the detection region with the crack identification result being the crack is determined as a candidate detection region; and the target detection region containing the crack is determined in each candidate detection region according to the overlapping area of each candidate detection region and each crack prediction frame. The method can improve the accuracy of crack detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a crack detection method and device based on joint detection, a computer device, a storage medium and a computer program product. BACKGROUND

[0002] Crack detection, such as crack detection of the surface of engineering structures such as asphalt pavement, bridges, and other buildings, is a key link in the preventive maintenance of engineering structures. For example, asphalt pavement cracks not only affect driving safety and comfort, but also cause poor stability of the pavement layer and other secondary diseases due to water seepage. Therefore, timely maintenance and repair of early-stage cracks in asphalt pavement can prolong the service life of the asphalt pavement. Therefore, regular detection and evaluation of the surface of engineering structures such as asphalt pavement can provide decision-making basis for timely maintenance and repair of cracks and other diseases, and is of great significance to maintaining the health of the surface of engineering structures.

[0003] With the development of deep learning technology, target detection algorithms have emerged, which can be used for crack detection of the surface of engineering structures such as asphalt pavement. However, the accuracy of crack detection is not high, which is not conducive to the calculation of crack rate. SUMMARY

[0004] Therefore, it is necessary to provide a crack detection method and device based on joint detection, a computer device, a computer readable storage medium and a computer program product, which can improve the accuracy of crack detection.

[0005] In a first aspect, the present application provides a crack detection method based on joint detection. The method comprises:

[0006] obtaining a target image of a to-be-detected object;

[0007] inputting the target image into a pre-trained crack joint detection network to obtain a first detection result and a second detection result; the first detection result is a crack recognition result corresponding to each detection region in the target image; the crack recognition result includes a crack and no crack; the second detection result is a crack prediction box including a category prediction confidence and position information;

[0008] determining the detection region in which the crack recognition result is as a candidate detection region;

[0009] determining a target detection region containing a crack in each candidate detection region according to the overlapping area of each candidate detection region and each crack prediction box.

[0010] In one of the embodiments, the determining a target detection region in each of the candidate detection regions according to an overlapping area of each of the candidate detection regions and each of the crack prediction boxes comprises:

[0011] For each of the candidate detection regions, an overlapping area of the candidate detection region and each of the crack prediction boxes is calculated, and in a case that at least one of the overlapping areas is greater than a preset threshold, the candidate detection region is determined as a target detection region containing cracks.

[0012] In one of the embodiments, the detection region is a grid region of a preset size, and each of the grid regions constitutes the target image; the method further comprises:

[0013] The number of the target detection regions and the total number of the grid regions contained in the target image are determined, and a ratio of the number to the total number is calculated, and the ratio is determined as a total crack rate of the object to be detected.

[0014] In one of the embodiments, the second detection result further contains a crack category corresponding to the crack prediction box; after the determining a target detection region containing cracks in each of the candidate detection regions according to an overlapping area of each of the candidate detection regions and each of the crack prediction boxes, the method further comprises:

[0015] For each of the target detection regions, a target crack prediction box with a maximum overlapping area with the target detection region is determined in each of the crack prediction boxes, and a crack category corresponding to the target crack prediction box is determined as a crack category corresponding to the target detection region.

[0016] In one of the embodiments, the crack joint detection network comprises a backbone network, a first neck network and a second neck network respectively connected to the backbone network, a first head network connected to the first neck network, and a second head network connected to the second neck network; the inputting the target image into the pre-trained crack joint detection network to obtain a first detection result and a second detection result comprises:

[0017] The target image is input into the backbone network for feature extraction, and the extracted feature information is input into the first neck network and the second neck network respectively, a first feature map is obtained by the first processing of the first neck network, and a second feature map is obtained by the second processing of the second neck network; wherein each pixel point of the first feature map corresponds to a detection region;

[0018] input the first feature map into the first head network, perform first crack category prediction on each pixel point of the first feature map through the first head network, and obtain the first detection result;

[0019] input the second feature map into the second head network for second crack category prediction, and obtain the second detection result.

[0020] In one of the embodiments, the training process of the crack joint detection network comprises:

[0021] obtain a sample image of a sample object and annotation data corresponding to the sample image; the annotation data comprises first type annotation data and second type annotation data, the first type annotation data is crack annotation results corresponding to a plurality of sample detection regions contained in the sample image, and the crack annotation results comprise containing cracks and not containing cracks; the second type annotation data is an annotation box containing crack category information and position information;

[0022] input the sample image and the annotation data into an initial crack joint detection network for joint training, and obtain a trained crack joint detection network.

[0023] In a second aspect, the present application further provides a crack detection device based on joint detection. The device comprises:

[0024] a first obtaining module configured to obtain a target image of an object to be detected;

[0025] a detection module configured to input the target image into a pre-trained crack joint detection network to obtain a first detection result and a second detection result; the first detection result is a crack recognition result corresponding to each detection region in the target image; the crack recognition result comprises containing cracks and not containing cracks; and the second detection result is a crack prediction box containing category prediction confidence and position information;

[0026] a first determining module configured to determine the detection region whose crack recognition result is containing cracks as a candidate detection region;

[0027] a second determining module configured to determine a target detection region containing cracks in each candidate detection region according to the overlapping area of each candidate detection region and each crack prediction box.

[0028] In one of the embodiments, the second determining module is specifically configured to:

[0029] For each of the candidate detection regions, an overlapping area of the candidate detection region and each of the crack prediction boxes is calculated, and in a case that at least one of the overlapping areas is greater than a preset threshold, the candidate detection region is determined as a target detection region containing a crack.

[0030] In one of the embodiments, the detection regions are grid regions of a preset size, and each of the grid regions constitutes the target image; the device further comprises:

[0031] The computing module is configured to determine a number of the target detection regions and a total number of the grid regions contained in the target image, and calculate a ratio of the number to the total number, and determine the ratio as a total crack rate of the to-be-detected object.

[0032] In one of the embodiments, the second detection result further contains a crack category corresponding to the crack prediction box; the device further comprises:

[0033] The third determining module is configured to, for each of the target detection regions, determine a target crack prediction box with a maximum overlapping area with the target detection region from among the crack prediction boxes, and determine a crack category corresponding to the target crack prediction box as a crack category corresponding to the target detection region.

[0034] In one of the embodiments, the crack joint detection network comprises a backbone network, a first neck network and a second neck network respectively connected to the backbone network, a first head network connected to the first neck network, and a second head network connected to the second neck network; the detecting module is specifically configured to:

[0035] The target image is input to the backbone network for feature extraction, and the extracted feature information is input to the first neck network and the second neck network respectively, the first processing is performed through the first neck network to obtain a first feature map, and the second processing is performed through the second neck network to obtain a second feature map; wherein each pixel point of the first feature map corresponds to one of the detection regions; the first feature map is input to the first head network, the first crack category prediction is performed on each pixel point of the first feature map through the first head network to obtain the first detection result; the second crack category prediction is performed on the second feature map through the second head network to obtain the second detection result.

[0036] In one of the embodiments, the device further comprises:

[0037] The second acquisition module is configured to acquire a sample image of a sample object and annotation data corresponding to the sample image; the annotation data comprises first type annotation data and second type annotation data; the first type annotation data is a crack annotation result corresponding to a plurality of sample detection regions contained in the sample image, and the crack annotation result comprises a crack and no crack; and the second type annotation data is an annotation box containing crack category information and position information.

[0038] The training module is configured to input the sample image and the annotation data into an initial crack joint detection network for joint training to obtain a trained crack joint detection network.

[0039] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method of the first aspect when executing the computer program.

[0040] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program implements the steps of the method of the first aspect when executed by a processor.

[0041] In a fifth aspect, the present application further provides a computer program product. The computer program product comprises a computer program, and the computer program implements the steps of the method of the first aspect when executed by a processor.

[0042] The aforementioned crack detection method, apparatus, computer equipment, storage medium, and computer program product based on joint detection perform crack detection on the target image of the object to be detected through a pre-trained crack joint detection network, yielding two detection results. The first detection result is the crack identification result corresponding to each detection region of the target image, and the second detection result is a crack prediction bounding box containing category prediction confidence and location information. Then, by determining the detection regions containing cracks as candidate detection regions, and based on the overlap area between each candidate detection region and each crack prediction bounding box, the target detection region is determined within each candidate detection region, thus detecting that the target detection region in the target image contains a crack. In this method, a crack joint detection network is used to detect the target image to obtain a first detection result based on multiple detection regions and a second detection result based on the predicted bounding box. The two detection results are matched based on the overlap area between the candidate detection region and the crack prediction bounding box. The target detection region that both detection results indicate has a high probability of containing cracks is retained, which can improve the detection accuracy or confidence of the target detection region containing cracks. At the same time, since the target detection regions do not overlap, the same crack (or the same part of the crack) can be repeatedly detected by multiple target detection regions, thereby improving the accuracy of crack detection. Furthermore, the target detection region is more conducive to the calculation of crack rate, which can improve the accuracy of crack rate calculation. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating a crack detection method based on joint detection in one embodiment;

[0044] Figure 2a This is a schematic diagram of the target image in an example;

[0045] Figure 2b This is a schematic diagram of the first detection result in an example;

[0046] Figure 2c This is a schematic diagram of the second detection result in an example;

[0047] Figure 2d This is a schematic diagram comparing the first and second detection results in an example.

[0048] Figure 2e This is a schematic diagram of the target detection region in an example;

[0049] Figure 3 This is a schematic diagram of the process for obtaining the first detection result and the second detection result in one embodiment;

[0050] Figure 4 This is a schematic diagram of the structure of a joint crack detection network in an example;

[0051] Figure 5 a flowchart of a training process of a crack joint detection network in an embodiment;

[0052] Figure 6 a structural block diagram of a crack detection device based on joint detection in an embodiment;

[0053] Figure 7 an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0054] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0055] First, before specifically introducing the technical solutions of the embodiments of the present application, the technical background or technical evolution context based on which the embodiments of the present application are introduced. It is of great significance to maintain the health of the surface of engineering structures such as asphalt pavement by periodically detecting and evaluating the cracks on the surface of engineering structures in order to maintain and repair them in time. With the development of deep learning technology, target detection algorithms have emerged, which can be used for crack detection on the surface of engineering structures such as asphalt pavement. However, due to the topological morphological structure of cracks, there are obvious differences compared with common target detection objects (such as people, animals, vehicles, etc.), for example, cracks have more typical linear features, and the overlapping occlusion between cracks is manifested as the connectivity of cracks, etc. Therefore, the accuracy of using target detection algorithms to detect cracks is not high. Based on this background, the applicant proposes a crack detection method based on joint detection through long-term research and development and experimental verification, which can improve the accuracy of crack detection and improve the calculation accuracy and efficiency of crack rate. In addition, it should be noted that the applicant has made a lot of creative labor for the discovery of the technical problem of the present application and the technical solutions introduced in the following embodiments.

[0056] In an embodiment, as shown in Figure 1 a crack detection method based on joint detection is provided, and the present embodiment takes this method as an example applied to a terminal. It can be understood that this method can also be applied to a server, and can also be applied to a system including a terminal and a server, and can be realized through the interaction of the terminal and the server. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices, etc. The server can be realized by an independent server or a server cluster composed of multiple servers. In the present embodiment, the method includes the following steps:

[0057] Step 101, obtaining a target image of a to-be-detected object.

[0058] The to-be-detected object refers to an object that needs to be subjected to crack detection, and can be an asphalt road, a concrete road, a bridge, or other engineering structures. The target image of the to-be-detected object is generally an image obtained by collecting the surface of the aforementioned engineering structure. The image perspective can be a top-down perspective. If it is an image of other perspectives, a related algorithm can be used to process the image to obtain an image of a top-down perspective.

[0059] In implementation, the terminal can capture an image of the to-be-detected object by using a camera device (such as a camera or a detection device provided with a camera component) to obtain a target image. In an example, the target image is as shown in FIG. 2, which is a digital image obtained by capturing an asphalt pavement. The image contains four cracks a, b, c, and d. Figure 2a

[0060] Step 102: input the target image into the pre-trained crack joint detection network to obtain a first detection result and a second detection result.

[0061] The first detection result is a crack identification result corresponding to each detection region in the target image, and the crack identification result includes a crack-containing region and a crack-free region. The detection region is a unit region for crack detection in the target image, and each unit region is an independent region, that is, each unit region does not overlap with each other. The crack identification result corresponding to each detection region indicates whether the detection region contains a crack or not. The second detection result is a crack prediction box containing a class prediction confidence and position information.

[0062] In implementation, the terminal can use the pre-trained crack joint detection network to perform crack detection on the target image. The crack joint detection network can process the input target image and output two detection results (the first detection result based on the detection region and the second detection result based on the prediction box). In an example, the detection region can be a grid region. After detecting the target image as shown in FIG. 2 by using the crack joint detection network, the first detection result as shown in FIG. 3 and the second detection result as shown in FIG. 4 can be obtained. Figure 2a Figure 2b Figure 2c In the first detection result as shown in FIG. 3, the detection regions (such as the grid n 0101 in the first row and the first column) marked by a dashed line indicate crack-free regions, and the detection regions (such as the grid n 0408 in the fourth row and the eighth column and the grid n 1012 in the tenth row and the twelfth column) marked by a solid line indicate crack-containing regions. In the second detection result as shown in FIG. 4, the crack prediction box contains the class prediction confidence and the position information of the crack. Figure 2b 0408 ​​​​The second detection result shown contains five crack prediction boxes, Box1 to Box5. It can be seen that crack prediction box Box1 misdetected cracks a and b as one crack. Therefore, crack a is detected by both initial prediction boxes Box1 and Box2, and crack b is detected by both initial prediction boxes Box1 and Box3. That is, there is a case where the same crack is repeatedly detected by multiple prediction boxes.

[0063] In step 103, the crack identification result of the detection region containing the crack is determined as a candidate detection region.

[0064] In implementation, the terminal can filter out the object detection region with the crack identification result containing the crack as the candidate detection region according to the crack identification result corresponding to each object detection region in the first detection result. For example, Figure 2b In the first detection result shown, the candidate detection region with the crack identification result containing the crack (such as grid n 0408 , grid n 1012 , etc. in the 4th row and the 8th column) can be filtered out.

[0065] In step 104, the target detection region containing the crack is determined in each candidate detection region according to the overlapping area of each candidate detection region and each crack prediction box.

[0066] In implementation, the terminal can calculate the overlapping area of each candidate detection region and each crack prediction box according to the position information of each candidate detection region and the position information of each crack prediction box, and determine the target detection region containing the crack in each candidate detection region based on the calculated overlapping area. If the candidate detection region and the crack prediction box overlap, it means that both detection results indicate that the overlapping region has a high probability of containing the crack. Therefore, the target detection region determined according to whether there is an overlap or according to the size of the overlapping area has a high probability of containing the crack, that is, the target detection region containing the crack can be more accurately detected. Moreover, since each target detection region does not overlap, the same crack (or the same part of the crack, such as a long crack, which may be contained in multiple adjacent detection regions, but each detection region detects a different part of the crack) can be avoided from being repeatedly detected by multiple target detection regions. Therefore, the accuracy of crack detection can be improved, and the calculation accuracy of the crack rate can be improved based on the target detection region for crack rate calculation. In one example, Figure 2d As shown in the comparison diagram of the first detection result and the second detection result, candidate detection region n 0408 overlaps with crack prediction boxes Box1 and Box2. Therefore, detection region n 0408 is determined as the target detection region, while candidate detection regions n 1012 , n 1112 , and n1809 If there is no area overlap between each crack prediction box and each candidate detection region, the candidate detection region n 1012 , n 1112 , and n 1809 are filtered out as outlier grids and are not used as the final target detection region. Thus, the target detection region (the region marked by a solid line) shown in FIG. 8 can be obtained. Figure 2e

[0067] In the crack detection method based on joint detection, the target image of the object to be detected is detected by the pre-trained crack joint detection network, and two detection results can be obtained. The first detection result is the crack recognition result of each detection region of the target image, and the second detection result is the crack prediction box containing the class prediction confidence and the position information. Then, the detection region with the crack recognition result is determined as the candidate detection region, and the target detection region is determined in each candidate detection region according to the area overlap between each candidate detection region and each crack prediction box, that is, the target detection region in the target image is detected to contain the crack. In this method, the target image is detected by the crack joint detection network to obtain the first detection result based on multiple detection regions and the second detection result based on the prediction box, and the two detection results are matched based on the area overlap between the candidate detection region and the crack prediction box, and the target detection region that is indicated by both the detection results to contain the crack with a high probability is retained, which can improve the detection accuracy or confidence of the target detection region containing the crack. At the same time, since each target detection region does not overlap with each other, the same crack (or the same part of the crack) can be avoided to be repeatedly detected by multiple target detection regions, thereby improving the accuracy of crack detection. Moreover, the target detection region is more conducive to the calculation of the crack rate, and the calculation accuracy of the crack rate can be improved.

[0068] In one embodiment, the process of determining the target detection region in step 104 specifically includes the following steps: for each candidate detection region, calculating the area overlap between the candidate detection region and each crack prediction box, and in each area overlap, if there is at least one area overlap greater than a preset threshold, the candidate detection region is determined as the target detection region containing the crack.

[0069] ​In implementation, after the terminal determines that the crack identification result is each candidate detection region containing a crack, the terminal can calculate, for each candidate detection region, an overlapping area of the candidate detection region and each crack prediction box according to position information of the candidate detection region and position information of each crack prediction box, so as to obtain an overlapping area set, that is, each value in the overlapping area set corresponds to an overlapping area of a crack prediction box and the candidate detection region, and the number of values in the overlapping area set is the same as the number of crack prediction boxes. If there is a value greater than a preset threshold in the overlapping area set, that is, there is at least one overlapping area greater than the preset threshold in the overlapping area obtained by the foregoing calculation, the candidate detection region is determined as a target detection region containing a crack. The preset threshold can be set according to the situation, for example, it can be set to 0, that is, any crack prediction box overlaps with the candidate detection region, that is, the candidate detection region is determined as a target detection region, or it can be set to other values, which are not limited herein. It can be understood that if the overlapping area of the candidate detection region and each crack prediction box is less than or equal to the preset threshold, it can be considered that the candidate detection region does not contain a crack, and the candidate detection region can be used as a filtering region, that is, in the first detection result, the filtering region is removed from each detection region containing a crack in the crack identification result, and the remaining detection region is a target detection region containing a crack.

[0070] In another implementation, after the overlapping area of each candidate detection region and each crack prediction box is calculated, the ratio of each overlapping area to the candidate detection region can be further calculated to obtain an overlapping area ratio (denoted as IoA). Then, the overlapping area ratio can be compared with a preset threshold, or the overlapping area ratios are sorted in size, and the maximum overlapping area ratio is compared with the preset threshold, and if the maximum overlapping area ratio is greater than the preset threshold, the candidate detection region is determined as a target detection region.

[0071] In this embodiment, by calculating the overlapping area of the candidate detection region and each crack prediction box, if at least one overlapping area is greater than a preset threshold, the candidate detection region is used as a target detection region. That is, by correcting the first detection result based on the detection region based on the second detection result based on the prediction box, the detection region whose overlapping area with each crack prediction box does not meet the requirement (less than or equal to the preset threshold) is removed (or referred to as filtering out the outlier detection region), so as to improve the crack detection accuracy based on the detection region. Moreover, unlike the crack prediction box, each target detection region does not overlap with each other, which can avoid the case that multiple target detection regions repeatedly detect the same crack (or the same part of the crack), so as to improve the accuracy of crack detection.

[0072] In an embodiment, the detection region is a grid region of a preset size, and each grid region constitutes the target image. The method further includes the following steps: determining the number of target detection regions and the total number of grid regions included in the target image, and calculating the ratio of the number to the total number, and determining the ratio as the total crack rate of the object to be detected.

[0073] In implementation, each detection region is a grid region in the target image. For example, the target image can be divided into multiple grid regions, and each grid region is a detection region. The size of each grid region is a preset size, and the specific size can be pre-set according to the situation, for example, according to the actual size of the object to be detected corresponding to each grid region. For example, the preset size can be set such that the actual size of the object to be detected corresponding to each grid region is 100mm*100mm. That is, if the object to be detected is an asphalt pavement and the target image is an image of the asphalt pavement, the actual size of the asphalt pavement corresponding to each grid region is 100mm*100mm.

[0074] After the target detection region is determined, the terminal can further calculate the ratio of the number (which can be denoted as n) of target detection regions to the total number (which can be denoted as N) of grid regions included in the target image, and determine the ratio as the total crack rate (which can be denoted as r) of the object to be detected. The calculation formula of the total crack rate r can be expressed as:

[0075]

[0076] In the first detection result obtained by the crack joint detection network in the embodiment, the first detection result includes the crack recognition result corresponding to each grid region. Based on the overlap area of each recognition result between the grid region (candidate detection region) containing the crack and each crack prediction box, each grid region is filtered or screened to obtain a target grid region (target detection region). In this way, the crack detection accuracy based on the grid (detection region) can be improved, so that the crack rate r can be calculated according to the number n of target grid regions and the total number N of grid regions, and the calculation accuracy and efficiency of the crack rate can be improved.

[0077] In an embodiment, the second detection result further includes the crack category corresponding to the crack prediction box. After the target detection region is determined in step 104, the method further includes the following steps: for each target detection region, determining a target crack prediction box with the largest overlap area with the target detection region from the crack prediction boxes, and determining the crack category corresponding to the target crack prediction box as the crack category corresponding to the target detection region.

[0078] In implementation, a plurality of crack categories can be preset, and the crack joint detection network can be trained based on sample data labeled with crack categories, so that the second detection result will contain the category prediction confidence of the crack prediction box, the crack category and the position information. The specific crack category can be set according to the engineering requirements, for example, it can include longitudinal cracks (such as longitudinal cracks c shown in Figure 2a , transverse cracks (such as transverse cracks a and b shown in Figure 2a , and crack cracks (such as crack cracks d shown in Figure 2a , etc. Specifically, after inputting the target image into the crack joint detection network, the first detection result based on the detection area can be obtained, and a plurality of crack prediction boxes and the category prediction confidence of each crack category corresponding to each crack prediction box can be obtained, for example, the category prediction confidence of each category of crack prediction box Box1 is obtained: the category prediction confidence of the category of transverse crack is 0.6, the category prediction confidence of the category of longitudinal crack is 0.3, and the category prediction confidence of the category of crack crack is 0.1. Then the terminal can output the crack category with the maximum category prediction confidence and the corresponding category prediction confidence as the second detection result, that is, the output second detection result will contain the position information of the crack prediction box Box1, the category prediction confidence of the crack prediction box Box1 is 0.6, and the crack category corresponding to the crack prediction box Box1 is transverse crack.

[0079] After the terminal determines each target detection area, it can calculate the overlapping area of each crack prediction box and the target detection area for each target detection area, and take the crack prediction box with the maximum overlapping area as the target crack prediction box. Then, the terminal can determine the crack category corresponding to the target crack prediction box as the crack category corresponding to the target detection area.

[0080] It can be understood that the terminal can determine the crack category of the target detection area at the same time in the process of determining the target detection area. Specifically, the terminal can determine the maximum overlapping area in the overlapping areas after calculating the overlapping areas of the candidate detection area and each crack prediction box, and then compare the maximum overlapping area with the preset threshold. If it is greater than the preset threshold, the candidate detection area is determined as the target detection area, and the crack category of the crack prediction box corresponding to the maximum overlapping area is determined as the crack category corresponding to the target detection area.

[0081] It can be understood that if the number of target crack prediction boxes with the maximum overlapping area is more than one, that is, multiple target crack prediction boxes have the same and maximum overlapping area with the target detection region, the crack category with the highest priority level can be determined as the crack category corresponding to the target detection region according to the priority levels of the crack categories of the target crack prediction boxes. The priority levels can be set in advance, for example, the priority level of the crack is the highest, the priority level of the horizontal crack is the second, and the priority level of the vertical crack is the lowest. For example, if two target crack prediction boxes i and j are determined at the same time, and the crack category of the target crack prediction box i is the crack crack, and the crack category of the target crack prediction box j is the vertical crack, since the priority level of the crack crack is higher than that of the vertical crack, the crack crack can be taken as the crack category corresponding to the target detection region.

[0082] Further, each detection region can be a grid region. After determining the crack category corresponding to each target grid region (target detection region), the number of target grid regions under each crack category can be calculated, and then the ratio of the number of target grid regions under each crack category to the total number of grid regions contained in the target image can be calculated as the sub-crack rate of each crack category of the object to be detected.

[0083] In the embodiment, the crack category of the crack prediction box with the maximum overlapping area with the target detection region can be taken as the crack category corresponding to the target detection region, that is, it is considered that the target detection region contains a crack with the crack category. In this way, the detection result (the target detection region containing the crack and the crack category) with higher detection accuracy can be obtained by combining the respective advantages of the detection results based on the detection regions and the detection results based on the prediction boxes, and the sub-crack rate of each crack category of the object to be detected can be calculated based on the detection result.

[0084] In one embodiment, the crack joint detection network includes a backbone network, a first neck network and a second neck network connected to the backbone network respectively, and a first head network connected to the first neck network and a second head network connected to the second neck network. As shown in Figure 3 The process of obtaining the first detection result and the second detection result in step 102 specifically includes the following steps:

[0085] In step 301, the target image is input into the backbone network for feature extraction, and the extracted feature information is input into the first neck network and the second neck network respectively, the first processing is performed through the first neck network to obtain the first feature map, and the second processing is performed through the second neck network to obtain the second feature map.

[0086] Each pixel point of the first feature map corresponds to a detection region.

[0087] In implementation, the pre-constructed and trained crack joint detection network can include a backbone network, a first neck network and a second neck network connected with the backbone network respectively, and a first head network connected with the first neck network and a second head network connected with the second neck network. As shown in the structural diagram of the crack joint detection network, it includes a backbone network backbone, a first neck network neck1 and a second neck network neck2, and a first head network head1 and a second head network head2. Among them, the backbone network backbone can adopt VGG, ResNet, DensNet, DarkNet, etc. backbone network, neck1 and neck2 can adopt FPN+PAN structure, etc. Figure 4

[0088] The terminal can input the target image to the backbone network backbone for feature extraction, and input the extracted feature information to the first neck network neck1 and the second neck network neck2 respectively for next processing. Specifically, the first processing is performed through the first neck network neck1, such as convolution operation, etc., to further extract and reduce the dimension of the feature information output by the backbone, so as to downsample each detection region in the target image to a pixel point, and each pixel point constitutes a first feature map. Among them, the first feature map is consistent with the number of detection regions contained in the target image, and each pixel point of the first feature map corresponds to a detection region. According to the position information of the pixel point in the feature map and the position information of the detection region in the target image, the pixel point in the feature map and the detection region can be one-to-one corresponding.

[0089] After inputting the feature information extracted by the backbone into the second neck network neck2, the second processing such as feature combination, further feature extraction, etc. can be performed through the second neck network neck2 to obtain a second feature map. The specific processing process can adopt the processing process of the neck network (neck network) in the target detection algorithm (such as You Only Look Once, YOLO-v5), which will not be described here.

[0090] Step 302, input the first feature map into the first head network, and perform first crack class prediction on each pixel point of the first feature map through the first head network to obtain a first detection result.

[0091] ​In implementation, the terminal can input the first feature map into the first head network head1, perform first crack category prediction on each pixel point of the first feature map through the first head network head1, and obtain the first detection result. Specifically, head1 can convolve the channel number corresponding to each pixel point in the first feature map output by neck1 into the category number (such as the category being a crack or being non-crack), as the crack identification result of the detection region corresponding to the pixel point, thereby obtaining the first detection result.

[0092] Step 303, input the second feature map into the second head network to perform second crack category prediction, and obtain the second detection result.

[0093] In implementation, the terminal can input the second feature map output by neck2 into the second head network head2 to perform second crack category prediction, and obtain the crack prediction box containing position information and category prediction confidence as the second detection result. The specific processing process can adopt the processing process of the head network (head network) in the target detection algorithm (such as YOLO-v5), which will not be described here.

[0094] In this embodiment, by inputting the target image into the crack joint detection network, performing feature extraction through the backbone network of the crack joint detection network, inputting the extracted feature information into the first neck network, performing first processing through the first neck network to obtain the first feature map, and then inputting the first feature map into the first head network to perform crack category prediction on each pixel point in the first feature map, the crack identification result corresponding to each pixel point can be obtained at the same time. Since the pixel point corresponds to the detection region, the crack identification result corresponding to each detection region can be obtained at the same time as the first detection result based on the detection region. At the same time, the extracted feature information is input into the second neck network, the second processing is performed through the second neck network, and the processing result is input into the second head network to obtain the second detection result based on the prediction box. In this way, the feature information extracted by the same backbone network can be used for the processing of the next two branches, avoiding the use of respective backbone networks by each branch, so as to improve the efficiency of crack detection, save computing resources, and improve the accuracy of crack detection based on the next step of matching processing of the target detection region.

[0095] In one embodiment, as shown in FIG. 1, Figure 5 the training process of the crack joint detection network includes the following steps:

[0096] Step 501, obtaining a sample image of a sample object and annotation data corresponding to the sample image.

[0097] The labeled data includes first type labeled data and second type labeled data. The first type labeled data is a crack label result corresponding to a plurality of sample detection regions contained in the sample image. The crack label result includes containing a crack and not containing a crack. The second type labeled data is a labeled box containing crack category information and position information.

[0098] In implementation, a sample image of a sample object can be collected by a collection device. The sample object can be of the same category as the to-be-detected object. For example, if the to-be-detected object is an asphalt pavement, an image of the asphalt pavement can be collected as the sample image. The sample image can be taken at a top-down perspective. If the image is taken at other perspectives, a related algorithm can be used to process the image to obtain a sample image at a top-down perspective.

[0099] Then, the collected sample image can be labeled with first type data and second type data to obtain two types of labeled data. The first type labeled data is a crack label result corresponding to a plurality of sample detection regions contained in the sample image. The crack label result includes containing a crack (e.g., labeled as 1) and not containing a crack (e.g., labeled as 0). For example, the sample image can be divided into a plurality of grid regions (e.g., 20*20 grid regions) so that the actual size of the sample object (e.g., an asphalt pavement) corresponding to each grid region meets a preset size, such as 100mm*100mm. It can be understood that, considering the constraint of the image feature down-sampling rate (e.g., 32 times down-sampling) of the crack joint detection network and the constraint of the actual size of the grid region, the size of the sample image can be constrained or adjusted.

[0100] For example, the size (pixels) of the sample image can be constrained to 640*640, and the size of each 32*32 image meets the actual size requirement of the sample object (e.g., 100mm*100mm). That is, after 32 times down-sampling of the feature of the input sample image with a size of 640*640, a feature map with a dimension of 20*20 can be obtained, that is, the feature map contains 400 pixels, each pixel corresponds to a local image region of 32*32 in the sample image, and each local image region (detection region or grid region) meets the actual size requirement of the sample object. If the size of the collected image is 2048*2048, and the ground sample distance (GSD) of two pixels is about 1mm, the size of the image can be adjusted to 640*640 as a sample image.

[0101] The second type of annotation data is an annotation box containing crack category information and position information, that is, the cracks contained in the sample image can be annotated by a bounding box, and each annotation box contains crack category information (such as horizontal cracks, vertical cracks, or crack cracks) and position information. Then, adaptive anchor boxes can be obtained based on clustering algorithms and genetic algorithms, which are used as the second type of annotation data.

[0102] Step 502, input the sample image and the annotation data into the initial crack joint detection network for joint training to obtain the trained crack joint detection network.

[0103] In implementation, the terminal can input the sample image and the two types of annotation data into the initial crack joint detection network for joint training. The structure of the initial crack joint detection network can be as shown in Figure 4 The structure of the initial crack joint detection network can be as shown in

[0104] During the training process, the first loss value can be calculated based on the first prediction result (crack prediction result corresponding to each sample detection region) of the first head network head1 and the crack annotation result corresponding to each sample detection region in the first type of annotation data. Specifically, cross-entropy and focal loss functions can be used. The second loss value can be calculated based on the second prediction result of the second head network head2 and the second type of annotation data. Then, the sum of the first loss value and the second loss value is minimized as the optimization target. The crack joint detection network is trained jointly. When the training end condition is reached, such as the number of iterations reaching a preset number, or the loss value (sum) being less than a preset threshold, the trained crack joint detection network is obtained.

[0105] In this embodiment, by annotating the sample image in two types and using the annotation data and the sample image for joint training of the crack joint detection network, and by sharing one backbone network for the two detection results of the crack joint detection network, the training efficiency and training effect of the crack joint detection network can be improved, thereby improving the crack detection accuracy and detection efficiency of the two detection results. Further processing of the target detection region based on the two detection results can further improve the crack detection accuracy.

[0106] In one embodiment, a crack detection method based on a detection region is also provided, including the following steps:

[0107] Step 1, obtaining a target image of an object to be detected.

[0108] Step 2, input the target image into the pre-trained detection region-based crack detection network to obtain crack identification results corresponding to each detection region in the target image. The crack identification results include detection regions containing cracks and detection regions not containing cracks.

[0109] The detection region can be a grid region. The crack detection network can include a backbone network, a neck network connected to the backbone network, and a head network connected to the neck network. The backbone network backbone can adopt a backbone network with structures such as VGG, ResNet, DensNet, and DarkNet. The neck network neck can adopt a network with structures such as FPN+PAN. After the target image is input into the backbone network backbone for feature extraction, the feature information is input into the neck network neck for processing. The detection region (grid) is down-sampled to one pixel point, and each pixel point forms a feature map, i.e., the feature map contains the same number of pixel points as the number of detection regions (grids), and each pixel point corresponds to a detection region. Then, the head network head convolves the channel number of each pixel point in the feature map output by the neck into the class number. For example, for a target image of 2048*2048 pixels with a GSD of about 1 millimeter, the size of the image can be adjusted to 640*640, and then input into the crack detection network. After processing by the backbone network backbone and the neck network neck, the neck outputs a feature map with a dimension of 20*20, i.e., the input image is down-sampled by 32 times to obtain a feature map containing 400 pixel points, and each pixel point corresponds to a detection region (grid) with a size of 32*32, which corresponds to an actual size of about 100mm*100mm of the object to be detected, meeting the requirements of the highway technical condition evaluation standard and facilitating subsequent crack rate calculation. Then, the head network head convolves the channel number of each pixel point in the feature map into the class number, and each channel represents the probability of a different class. The classes can include two classes of cracks and non-cracks. The sample data used to train the crack detection network can further be annotated with crack classes such as transverse cracks, longitudinal cracks, and crack cracks, so that the trained crack detection network can further identify the crack class corresponding to the detection region containing the crack.

[0110] Step 3, determining the detection region containing the crack as the target detection region according to the crack identification results.

[0111] Further, the training process of the detection region-based crack detection network includes the following steps:

[0112] Step 1, obtaining a sample image of a sample object.

[0113] Step 2, performing crack annotation on each sample detection region in the sample image to obtain annotation data.

[0114] Specifically, the detection region can be a grid, the sample image can be divided into multiple grids (for example, a 640*640 image is divided into 20*20 grids), and then each grid is labeled, for example, 1 if it contains a crack and 0 if it does not contain a crack. Further, if it contains a crack and is a transverse crack, it can be labeled as 101, if it is a longitudinal crack, it can be labeled as 102, and if it is a crack, it can be labeled as 103.

[0115] Optionally, an online data augmentation algorithm can be used to further expand the sample image, which is used as a sample image to ensure the richness and diversity of the sample image. If the data augmentation algorithm involves geometric transformation operations, attention needs to be paid to maintaining the position and size of the grid to meet the standard grid division. The so-called standard grid division is that for an image, it is divided into n rows and m columns of grids, and the actual size of each grid is about 100mm*100mm. For example, in mosaic (a data augmentation method), four images are cropped and spliced, and each image needs to be cropped along the edge of the standard grid and then spliced to ensure that the image after data augmentation still meets the standard grid division.

[0116] Step 3, input the sample image and the labeled data into the initial crack detection network for training to obtain the trained crack detection network based on the detection region.

[0117] Specifically, the sample image can be input into the initial crack detection network for processing, wherein each detection region (grid) is down-sampled to one pixel point on the feature map output by the neck network, and then the head network is used to predict the crack for each pixel point to obtain the crack identification result corresponding to each sample detection region. Then, the loss value is calculated according to the crack identification result and the labeled data, and the parameters (weights, etc.) of the crack detection network are updated according to the loss value to minimize the loss value, and the crack detection network based on the detection region is obtained.

[0118] Optionally, the cross-entropy loss function (BCEWithLogitsLoss) can be used to train the network, and the focal loss function (focalloss) can be used to further process the problem of unbalanced positive and negative samples. The formula of the cross-entropy loss function is as follows:

[0119]

[0120] Wherein, the size of the feature map output by the neck network is S1*S2, pi(c) represents the probability that the pixel point i in the feature map is of class c, and * represents the corresponding predicted value.

[0121] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0122] Based on the same inventive concept, the embodiments of the present application also provide a joint detection based crack detection device for implementing the joint detection based crack detection method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more joint detection based crack detection device embodiments provided below can refer to the limitations of the joint detection based crack detection method described above, which will not be repeated here.

[0123] In one embodiment, as shown in Figure 6 a joint detection based crack detection device 600 is provided, comprising: a first acquisition module 601, a detection module 602, a first determination module 603 and a second determination module 604, wherein:

[0124] The first acquisition module 601 is configured to acquire a target image of a to-be-detected object.

[0125] The detection module 602 is configured to input the target image into a pre-trained crack joint detection network to obtain a first detection result and a second detection result; the first detection result is a crack recognition result corresponding to each detection region in the target image; the crack recognition result includes a crack and does not include a crack; the second detection result is a crack prediction box including a category prediction confidence and position information.

[0126] The first determination module 603 is configured to determine the detection region with the crack recognition result as a candidate detection region.

[0127] The second determination module 604 is configured to determine a target detection region containing a crack in each candidate detection region according to the overlapping area of each candidate detection region and each crack prediction box.

[0128] In one of the embodiments, the second determining module 604 is specifically configured to: for each candidate detection region, calculate overlapping areas between the candidate detection region and the crack prediction boxes, and determine the candidate detection region as a target detection region containing cracks in a case where there is at least one overlapping area greater than a preset threshold value among the overlapping areas.

[0129] In one of the embodiments, the detection region is a grid region of a preset size, and each grid region constitutes the target image. The device further includes a calculating module configured to determine a number of the target detection regions and a total number of the grid regions contained in the target image, and calculate a ratio of the number to the total number, and determine the ratio as a total crack rate of the object to be detected.

[0130] In one of the embodiments, the second detection result further contains a crack category corresponding to the crack prediction box. The device further includes a third determining module configured to, for each target detection region, determine a target crack prediction box with a maximum overlapping area with the target detection region among the crack prediction boxes, and determine a crack category corresponding to the target crack prediction box as a crack category corresponding to the target detection region.

[0131] In one of the embodiments, the crack joint detection network includes a backbone network, a first neck network and a second neck network connected to the backbone network respectively, and a first head network connected to the first neck network and a second head network connected to the second neck network. The detecting module 602 is specifically configured to: input the target image into the backbone network for feature extraction, and input the extracted feature information into the first neck network and the second neck network respectively, perform first processing through the first neck network to obtain a first feature map, and perform second processing through the second neck network to obtain a second feature map; wherein each pixel point of the first feature map corresponds to a detection region; input the first feature map into the first head network, perform first crack category prediction on each pixel point of the first feature map through the first head network to obtain a first detection result; and input the second feature map into the second head network for second crack category prediction to obtain a second detection result.

[0132] In one of the embodiments, the device further includes a second obtaining module and a training module, wherein:

[0133] The second obtaining module is configured to obtain a sample image of a sample object and annotation data corresponding to the sample image; the annotation data includes first type annotation data and second type annotation data, the first type annotation data is crack annotation results corresponding to a plurality of sample detection regions contained in the sample image, the crack annotation results include containing cracks and not containing cracks; and the second type annotation data is an annotation box containing crack category information and position information.

[0134] The training module is used to input sample images and labeled data into the initial joint crack detection network for joint training, resulting in the trained joint crack detection network.

[0135] The modules in the aforementioned crack detection device based on joint detection can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0136] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a crack detection method based on joint detection. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device casing, or an external keyboard, touchpad, or mouse.

[0137] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0138] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0139] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0140] In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of any of the above method embodiments.

[0141] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0142] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0143] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, as long as the combinations of technical features do not have contradictions, they shall be considered within the scope of the present disclosure.

[0144] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it shall not be understood as a limitation on the patent scope of the present application. It shall be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these shall be within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A crack detection method based on joint detection, characterized by, The method comprises: acquiring a target image of an object to be detected; inputting the target image into a pre-trained joint crack detection network to obtain a first detection result and a second detection result; the first detection result is a crack identification result corresponding to each detection region in the target image; the crack identification result includes a crack-containing region and a crack-non-containing region; the second detection result is a crack prediction box including a category prediction confidence and position information; determining the crack-containing region as a candidate detection region; for each candidate detection region, calculating the overlapping area of the candidate detection region and each crack prediction box, and determining the candidate detection region as a target detection region containing a crack in the case that at least one of the overlapping areas is greater than a preset threshold.

2. The method of claim 1, wherein, The detection region is a grid region of a preset size, and each grid region constitutes the target image; the method further comprises: determining the number of target detection regions and the total number of grid regions contained in the target image, and calculating the ratio of the number to the total number, and determining the ratio as the total crack rate of the object to be detected.

3. The method of claim 1, wherein, The second detection result further includes a crack category corresponding to the crack prediction box; after determining the target detection region containing a crack in each candidate detection region according to the overlapping area of each candidate detection region and each crack prediction box, the method further comprises: for each target detection region, determining a target crack prediction box with the largest overlapping area with the target detection region in each crack prediction box, and determining the crack category corresponding to the target crack prediction box as the crack category corresponding to the target detection region.

4. The method of claim 1, wherein, The joint crack detection network comprises a backbone network, a first neck network and a second neck network connected to the backbone network respectively, and a first head network connected to the first neck network and a second head network connected to the second neck network; The method comprises: inputting the target image into the backbone network for feature extraction, and inputting the extracted feature information into the first neck network and the second neck network respectively, performing first processing through the first neck network to obtain a first feature map, and performing second processing through the second neck network to obtain a second feature map; wherein each pixel point of the first feature map corresponds to a detection region; inputting the first feature map into the first head network to perform first crack category prediction on each pixel point of the first feature map to obtain the first detection result; inputting the second feature map into the second head network to perform second crack category prediction to obtain the second detection result.

5. The method of claim 1, wherein, The training process of the joint crack detection network comprises: Obtaining a sample image of a sample object and annotation data corresponding to the sample image; the annotation data includes first type annotation data and second type annotation data, the first type annotation data is a crack annotation result corresponding to a plurality of sample detection regions contained in the sample image, and the crack annotation result includes containing a crack and not containing a crack; the second type annotation data is an annotation box containing crack category information and position information; The sample image and the annotation data are input into an initial crack joint detection network for joint training to obtain a trained crack joint detection network.

6. A crack detection device based on joint detection, characterized by, The device comprises: A first obtaining module configured to obtain a target image of an object to be detected; A detection module configured to input the target image into a pre-trained crack joint detection network to obtain a first detection result and a second detection result; the first detection result is a crack identification result corresponding to each detection region in the target image; the crack identification result includes containing a crack and not containing a crack; and the second detection result is a crack prediction box containing category prediction confidence and position information; A first determining module configured to determine the detection region whose crack identification result is containing a crack as a candidate detection region; A second determining module configured to, for each candidate detection region, calculate the overlapping area of the candidate detection region and each crack prediction box, and determine the candidate detection region as a target detection region containing a crack in the case that there is at least one overlapping area greater than a preset threshold among the overlapping areas.

7. The apparatus of claim 6, wherein, The detection region is a grid region of a preset size, and each grid region constitutes the target image; the device further comprises: A calculation module configured to determine the number of target detection regions and the total number of grid regions contained in the target image, and calculate the ratio of the number to the total number, and determine the ratio as the total crack rate of the object to be detected.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 5.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

Images