Method for crack identification and display of large-scale planar scene based on semantic segmentation

Abstract

The application discloses a kind of based on semantic segmentation algorithm's large-scale plane scene's crack identification and display method, which comprises the following steps: color camera is carried by unmanned aerial vehicle to obtain the color picture sequence and coordinate sequence of scene;Utilize semantic segmentation algorithm to identify the crack pixel of color picture sequence, and utilize color picture sequence and coordinate sequence to downsample;The spatial position of crack pixel is used to calculate the crack pixel of interconnection, and the corresponding crack pixel set is constructed;The view frame sequence and the spatial distribution attribute corresponding to view frame are calculated using crack pixel set;Further merging of view frame is carried out using view frame sequence and spatial distribution attribute, and crack display picture sequence is obtained.According to the display effect of crack, select the crack that needs to be removed, and obtain the picture sequence of crack display again.The application can identify and display the position of crack in large-scale plane scene, which facilitates infrastructure maintenance personnel to analyze the severity of crack distribution.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to a method for crack recognition and display in large-scale planar scenes based on semantic segmentation. Background Technology

[0002] With the rapid development of the economy and society, the amount of infrastructure has gradually increased, and the maintenance of infrastructure has become a major focus. Currently, crack identification and marking are often done manually, which is inefficient and cannot cope with the surge in maintenance needs brought about by the ever-increasing number of infrastructures. At the same time, with the development of drones and deep learning technology, large-scale scene scanning and high-precision crack identification have become possible. Therefore, there is an urgent need for an automated crack identification and display algorithm. Summary of the Invention

[0003] The present invention provides a method for crack recognition and display in large-scale planar scenes based on semantic segmentation. The large-scale planar scene refers to a scene in which the crack width is less than one pixel when it cannot be captured by a camera in one shot, so as to solve the problem of crack recognition and display in large-scale planar scenes.

[0004] In one aspect of the present invention, a crack recognition method for large-scale planar scenes based on semantic segmentation is proposed. According to an embodiment of the present invention, the method includes the following steps:

[0005] Step 1: Obtain a sequence of color images and coordinates of the scene using a color camera mounted on a drone;

[0006] Step 2: Use a semantic segmentation algorithm to identify crack pixels in the color image sequence, and then use the color image sequence and coordinate sequence for downsampling;

[0007] Step 3: Calculate the connected crack pixels using the spatial location of the crack pixels, and construct the corresponding crack pixel set;

[0008] Step 4: Calculate the view frame sequence and the corresponding spatial distribution attributes of the view frames using the crack pixel set;

[0009] Step 5: Further merge the view frames using the view frame sequence and spatial distribution attributes to obtain a sequence of crack display images.

[0010] In addition, the crack recognition method for large-scale planar scenes based on semantic segmentation according to the above embodiments of the present invention may also have the following additional technical features:

[0011] In some embodiments of the present invention, step 1, the method for obtaining the color image sequence and coordinate sequence of the scene includes the following steps:

[0012] Step 1.1: Obtain images collected by the camera, and use a drone to fly around and film the scene, keeping the drone's altitude above the ground constant, with the camera facing downwards, covering an area of ​​H. c ×W c The corresponding color image is an H p ×W p A tensor P of ×3, where H c W c H represents the height and width of the actual scene covered by the camera, respectively. p W p These represent the height and width of the image obtained through the camera, respectively, where 3 indicates that the image has 3 channels;

[0013] Step 1.2: Obtain the two-dimensional coordinates C during drone photography. C contains two values ​​X. c Y c The drone maintains a constant distance from the ground while moving, and its initial position is set to the origin, where X... c Y c These represent the horizontal and vertical distances of the shooting position from the origin, respectively.

[0014] Step 1.3: Based on the chronological order, construct a color image sequence LP = {P1, P2, ..., P...} using images P collected at different times and coordinates C. n} and the coordinate sequence LC={C1,C2,…C n}, n=1,2,3…, where n represents the final number of images taken, P n Let C represent the nth image collected. n This represents the nth coordinate collected when the drone moves horizontally by W. p Or move H in the vertical direction p At that time, take a picture and record the coordinates. The color picture should be able to cover the entire scene.

[0015] In some embodiments of the present invention, step 2, which involves identifying crack pixels using a semantic segmentation algorithm and downsampling them, includes the following steps:

[0016] Step 2.1: Set LP = {P1, P2, ..., P} n The input semantic segmentation network produces semantic segmentation results for each image, LS = {S1, S2, ... S}. i ,…,S n}, n = 1, 2, 3…, where S i It is a binary image, where i represents the index of the semantic segmentation result, and its size is the same as the image in the LP, which is H. p ×W p However, the number of channels is 1, which is one H.p ×W p A tensor of size ×1, S i The value of each pixel in the image is either 0 or 1, where 0 indicates that the pixel does not belong to the crack, and 1 indicates that the pixel belongs to the crack.

[0017] Step 2.2: Stitch LP and LS together according to their corresponding coordinates C to obtain the original image P of the complete scene. t and the overall segmentation result S t ;

[0018] Step 2.3: Analyze the overall crack identification results S t S' is obtained by scaling proportionally. t If the scaled pixel contains a crack pixel in the s×s region of the original overall segmentation result, the value of the pixel is 1; otherwise, it is 0, where s represents the downsampling factor.

[0019] In some embodiments of the present invention, step 3 includes the following steps:

[0020] Step 3.1: Initialize the crack instance set Cracks to an empty set, and the crack pixel access list Visited = [v1, v2, ..., v k All elements in the list are 0, the Search queue is empty, and the current crack pixel number is i. c The set of neighboring crack pixels is 0, N. s Let v be an empty set, where v k Indicates whether the k-th crack pixel has been visited; the element crack instance of Crack is S'. t The set of pixels in the middle crack, k is S' t The number of pixels in the crack, the elements in Visited and S' t The pixels in the crack correspond one-to-one, and the element in the queue to be explored is S'. t The crack pixels in the image;

[0021] Step 3.2: Obtain the current crack pixel number i c The corresponding item v in the Visited list ic If the value is 1, proceed to step 3.4; if it is 0, number the current crack pixel in S'. t in i c The corresponding pixels are placed in Search, and an empty set Cracks is added to Cracks. cur To Cracks cur Add the current crack pixel number i c Corresponding S' t The crack pixels in the image;

[0022] Step 3.3: Obtain the first element in Search and set the corresponding S' t The pixels surrounding the center in eight directions, namely top, bottom, left, right, upper right, lower right, upper left, and lower left, form set N. r = {n1,n2,…,n8}, where n1,n2,…,n8 are S' t The eight pixels are: top center, bottom center, left center, right center, top right center, bottom right center, top left center, and bottom left center; iterate through N. r If an element in N has a pixel value of 1, add it to N. s Then, towards Cracks cur Add N s The pixel in the Visited list will be the current crack pixel number i. c Corresponding item v ic Setting it to 1 will delete the first element of Search and add N to it. s The pixels in, where N s This represents the crack pixels surrounding the current pixel; if Search is empty, proceed to step 3.4, otherwise proceed to step 3.3;

[0023] Step 3.4: Number the current crack pixel i c Increment the value by 1 if i c If the value is greater than k, proceed to step 4; otherwise, proceed to step 3.2.

[0024] In some embodiments of the present invention, step 4 includes the following steps:

[0025] Step 4.1: Initialize the view frame set Boxes to an empty set, where each view frame is an octet (x, y, y). min ,x max ,y min ,y max ,lu,ld,ru,rd), where x min The minimum horizontal coordinate of the viewport's coverage area, x max The y-coordinate represents the maximum horizontal coordinate of the viewport's coverage area. min This refers to the minimum vertical coordinate of the viewport's coverage area, y max This refers to the maximum vertical coordinate of the view frame coverage area. lu represents the distribution coefficient of crack pixels in the upper left direction, ld represents the distribution coefficient of crack pixels in the lower left direction, ru represents the distribution coefficient of crack pixels in the upper right direction, and rd represents the distribution coefficient of crack pixels in the lower right direction.

[0026] Step 4.2: Construct an octet Box, iterate through each element Crack in Cracks, calculate the maximum and minimum values ​​of the horizontal and vertical coordinates of the pixel positions in Crack, and fill them into the first four items of Box;

[0027] Step 4.3: Calculate the distribution coefficients in the four directions and initialize Count. lu Count ld Count rd Count ru If the value is 0, iterate through each pixel in the Crack, setting the x and y coordinates of the pixel to 0. and Count lu Add one; if and Count rd Add one; if and Count ru Add one; if and Count ld Add one, and assign the value to the last four items of the Box. Place the Box into Boxes. Where Count... lu Count ld Count rd Count ru , representing the number of crack pixels in the top left, bottom left, bottom right, and top right positions, respectively, where x min The minimum horizontal coordinate of the viewport's coverage area, x max The y-coordinate represents the maximum horizontal coordinate of the viewport's coverage area. min This refers to the minimum vertical coordinate of the viewport's coverage area, y max This refers to the maximum vertical coordinate of the viewport's coverage area.

[0028] In some embodiments of the present invention, step 5 includes the following steps:

[0029] Step 5.1: Sort the Boxes in Boxes in descending order based on their area. Place the sorted results into the LBoxes list, setting `ismerged` to 0. `ismerged` indicates whether merging occurred during the calculation; it is 1 if merging occurred, and 0 otherwise. The formula for calculating the area of ​​a Box is as follows: (x...) max -x min )*(y max -y min ), where xmin x max y min y max The same meaning as in step 4;

[0030] Step 5.2: Iterate through the next element Box in LBoxes in sequence. s Construct the search scope Query, where Query is a four-tuple (qx min ,qx max ,qy min ,qy max ), where qx min =max(x min -δ x ,0), qy min =max(y min -δ y ,0), Where qx min qx max qy min qy max These represent the minimum and maximum values ​​of the search range on the horizontal and vertical axes, respectively. The functions `max()` and `min()` retrieve the maximum and minimum values ​​from these ranges, respectively. δ x δ represents the extended lateral distance. y Indicates the extended vertical distance. S' t width, S' t If there is a next element, proceed to step 5.3; if there is no next element and ismerged is 1, proceed to step 5.1; otherwise, proceed to step 5.7.

[0031] Step 5.3: Traverse all elements in LBoxes. If an element intersects with the search scope Query, denote it as a Box. c The condition for intersection is that both conditions must be met simultaneously: max(qx) min ,x min )>min(qx max ,x max ), max(qy min ,y min )>min(qy max ,y max ), put Box c Add it to the Cross collection;

[0032] Step 5.4: Traverse the next element Box in the Cross sequence.c If there is no next element, update the spatial distribution property of the viewbox corresponding to the Query using the method in step 4.3, and then execute step 5.1. If the Box... c center If the result falls within the scope of the query, proceed to step 5.6; otherwise, proceed to step 5.5.

[0033] Step 5.5: Determine the Query Center Compared to Box c The center location, and query Box c The distribution coefficient in the corresponding direction, if the distribution coefficient exceeds the threshold λ b If the above steps are not executed, proceed to step 5.6; otherwise, proceed to step 5.4.

[0034] Step 5.6: For Query and Box c Perform a merge operation, updating the values ​​of the first four items representing the range of the Box element in the LBoxes corresponding to the Query as follows:

[0035] min(x1 min x2 min ),max(x1 max x2 max ),min(y1 min ,y2 min ),max(y1 max ,y2 max );

[0036] Where x1 min This represents the minimum horizontal coordinate of the area covered by the Box, x1. max This represents the maximum horizontal coordinate of the area covered by the Box, y1 min This refers to the minimum vertical coordinate of the area covered by the Box, y1 max This refers to the maximum vertical coordinate of the area covered by the Box, where x2 min Box c Minimum horizontal coordinate of the coverage area, x2 max Box c The maximum horizontal coordinate of the coverage area, y2 min Refers to Box c Minimum vertical coordinate of the coverage area, y2 max Refers to Box c The maximum vertical coordinate of the coverage area is then set, and the value of ismerged is set to 1. Simultaneously, the Box and Box in Cracks are... c Merge the corresponding collections and remove the Box from LBoxes. c Proceed to step 5.3;

[0037] Step 5.7: Obtain the final display effect using the remaining viewports in LBoxes. f The first four items are used to extract the original image P from step 2. t The corresponding regions are used to obtain the final output image sequence LP. f ={P f1 ,P f2 ,…P fm}, where P fm This refers to the m-th output image, where m is the number of output image sequences, which is the same as the number of elements in LBoxes.

[0038] In another aspect, this invention proposes a method for displaying cracks in large-scale planar scenes based on semantic segmentation. According to embodiments of the invention, crack display image sequences obtained using the aforementioned method for crack identification in large-scale planar scenes based on semantic segmentation are used for interactive display.

[0039] In addition, the method for displaying cracks in large-scale planar scenes based on semantic segmentation according to the above embodiments of the present invention may also have the following additional technical features:

[0040] In some embodiments of the present invention, the user selects the cracks to be removed based on the display effect of the cracks, and obtains a new image sequence displaying the cracks; the user then views the final output image sequence LP. f Check if there are any cracks spanning the entire scene that cause the viewport to be too large. If so, select the crack, delete the corresponding set of crack pixels in Cracks, and set the P... t The color of the corresponding pixel is changed to red, indicating that the pixel belongs to the category of removing display cracks. Then, steps 4 and 5 are repeated to obtain LP'. f ={P' f1 ,P' f2 ,…P' fj}, where P' fj This refers to the j-th image obtained after re-execution, where j is the number of new images obtained after re-executing steps 4 and 5.

[0041] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0042] 1. This invention utilizes drones to scan the scene and leverages the drones' positioning capabilities to accurately display the location of the crack area.

[0043] 2. This invention utilizes a method for calculating connected pixels to reveal cracks that are too large to be identified in a single color image.

[0044] 3. The method for calculating connected pixels in this invention can be completed in one scan, avoiding the uncertainty and high computational load caused by repeated iterations of calculating crack display images using clustering algorithms.

[0045] 4. After obtaining the segmentation results, the present invention performs downsampling, which reduces the required computing resources while minimizing the reduction in accuracy.

[0046] 5. The view frame merging operation of the present invention enables cracks to be displayed in an interactive, compact, and non-repetitive manner. Interactive means that users can adjust the display scheme of cracks according to their own needs. Compact means that the algorithm tries its best to reduce the blank areas in the displayed crack images. Non-repetitive means that a crack can be avoided from appearing in multiple images. Attached Figure Description

[0047] Figure 1 This is an overall flowchart of the method for crack recognition and display in a large-scale planar scene based on semantic segmentation in an embodiment of the present invention.

[0048] Figure 2 This is a flowchart of semantic segmentation in an embodiment of the present invention;

[0049] Figure 3 This is a flowchart illustrating the calculation of connected crack pixels in an embodiment of the present invention;

[0050] Figure 4 This is a flowchart illustrating the merging of view frames in an embodiment of the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0052] In this embodiment, the method for crack identification and display in large-scale planar scenes based on semantic segmentation utilizes technologies such as computer vision, deep learning, and connected region computation. It identifies and displays cracks in images collected by drones and can also remove large cracks and update the display results based on user feedback on the display effect. Figure 1 As shown, this method is executed in the following manner:

[0053] Step 1: Use a drone equipped with a color camera to acquire a sequence of color images and coordinates of a large-scale scene.

[0054] Step 1.1: Obtain images collected by the camera, and use a drone to fly around and film the scene, keeping the drone's altitude above the ground constant, with the camera facing downwards, covering an area of ​​H. c ×W c The corresponding color image is an H p ×W p A tensor P of ×3, where H c W c H represents the height and width of the actual scene covered by the camera, respectively. p W p These represent the height and width of the image obtained through the camera, respectively, where 3 indicates that the image has 3 channels.

[0055] Step 1.2: Obtain the two-dimensional coordinates C during drone photography. C contains two values ​​X. c Y c The drone moves in the same vertical direction as the camera's coverage area, and its initial position is set to the origin. Where X... c Y c These are the horizontal and vertical distances from the shooting position to the origin, respectively.

[0056] In this embodiment, the distance between the drone and the ground remains constant at 1m, the camera covers an area of ​​1m x 1m, and the size of each collected image is 1080 x 1080.

[0057] Step 1.3: Based on the chronological order, construct a color image sequence LP = {P1, P2, ..., P...} using images P collected at different times and coordinates C. n} and the coordinate sequence LC={C1,C2,…C n}, n=1,2,3…, where n represents the final number of images taken, P n Let C represent the nth image collected. n This represents the nth coordinate collected when the drone moves horizontally by W. p Or move H in the vertical direction p At that time, take a picture and record the coordinates. The color picture should be able to cover the entire scene.

[0058] Step 2: Use a semantic segmentation algorithm to identify crack pixels and downsample them.

[0059] Step 2.1: Set LP = {P1, P2, ..., P} n The input semantic segmentation network produces semantic segmentation results for each image, LS = {S1, S2, ... S}. i ,…,S n}, n = 1, 2, 3…, where S iIt is a binary image, where i represents the index of the semantic segmentation result, and its size is the same as the image in the LP, which is H. p ×W p However, the number of channels is 1, which is one H. p ×W p A tensor of size ×1, S i The value of each pixel in the image is either 0 or 1, where 0 indicates that the pixel does not belong to the crack, and 1 indicates that the pixel belongs to the crack.

[0060] In this embodiment, the input image size is 1080x1080 with a bit depth of 24 bits. The semantic segmentation network used is the open-source UET algorithm, which was implemented by fine-tuning the structure and adjusting the output image size to 1080x1080. Modifications included adding dropout at specified locations and adjusting the number of channels in each layer according to actual needs. Figure 2 As shown.

[0061] Step 2.2: Stitch LP and LS together according to their corresponding coordinates C to obtain the original image P of the complete scene. t and the overall segmentation result S t .

[0062] Step 2.3: Scale the overall crack identification result proportionally to obtain S' t If the scaled pixel contains a crack pixel in the s×s region of the original overall segmentation result, the value of the pixel is 1; otherwise, it is 0, where s represents the downsampling factor.

[0063] In this embodiment, the value of s is set to 4, that is, one pixel after downsampling corresponds to a 4x4 pixel area of ​​the original image.

[0064] Step 3: Calculate the connected crack pixels using the spatial location of the crack pixels and place them into the corresponding sets. The specific process is as follows: Figure 3 As shown.

[0065] Step 3.1: Initialize the crack instance set Cracks to an empty set, and the crack pixel access list Visited = [v1, v2, ..., v k All elements in the list are 0, the Search queue is empty, and the current crack pixel number is i. c The set of neighboring crack pixels is 0, N. s Let v be an empty set, where v k Indicates whether the k-th crack pixel has been visited; the element crack instance of Crack is S'. t The set of pixels in the middle crack, k is S' t The number of pixels in the crack, the elements in Visited and S' tThe pixels in the crack correspond one-to-one, and the element in the queue to be explored is S'. t The crack pixels in the image;

[0066] Step 3.2: Obtain the current crack pixel number i c Current crack pixel number i in the Visited list c Corresponding item If the value is 1, proceed to step 3.4; if it is 0, change the current crack pixel number i. c In S' t The corresponding pixels are placed in Search, and an empty set Cracks is added to Cracks. cur To Cracks cur Add the current crack pixel number i c Corresponding S' t The crack pixels in the image.

[0067] Step 3.3: Obtain the first element in Search and set the corresponding S' t The values ​​of the surrounding pixels, specifically those in eight directions: top, bottom, left, right, top right, bottom right, top left, and bottom left, forming set N. r = {n1, n2, ..., n8}, traverse N r If an element in N has a pixel value of 1, add it to N. s Then, towards Cracks cur Add N s The pixel in the Visited list will be the current crack pixel number i. c Corresponding item Setting it to 1 will delete the first element of Search and add N to it. s The pixels in the Search. If Search is empty, proceed to step 3.4; otherwise, proceed to step 3.3.

[0068] Step 3.4: Number the current crack pixel i c Increment the value by 1 if i c If the value is greater than k, proceed to step 4; otherwise, proceed to step 3.2.

[0069] Step 4: Calculate the view frame sequence and the corresponding spatial distribution attributes using the merged set of crack pixels. The specific process is as follows: Figure 4 As shown.

[0070] Step 4.1: Initialize the view frame set Boxes to an empty set, where each view frame is an octet (x, y, y). min ,x max ,y min ,y max,lu,ld,ru,rd), where x min The minimum horizontal coordinate of the viewport's coverage area, x max The y-coordinate represents the maximum horizontal coordinate of the viewport's coverage area. min This refers to the minimum vertical coordinate of the viewport's coverage area, y max This refers to the maximum vertical coordinate of the viewport coverage area. lu represents the distribution coefficient of crack pixels in the upper left direction, ld represents the distribution coefficient of crack pixels in the lower left direction, ru represents the distribution coefficient of crack pixels in the upper right direction, and rd represents the distribution coefficient of crack pixels in the lower right direction.

[0071] Step 4.2: Construct an octet Box, iterate through each element Crack in Cracks, calculate the maximum and minimum values ​​of the horizontal and vertical coordinates of the pixel positions in Crack, and fill them into the first four items of Box.

[0072] Step 4.3: Calculate the distribution coefficients in the four directions and initialize Count. lu Count ld Count rd Count ru If the value is 0, iterate through each pixel in the Crack, setting the x and y coordinates of the pixel to 0. and Count lu Add one; if and Count rd Add one; if and Count ru Add one; if and Count ld Add one, and assign the value to the last four items of the Box. Place the Box into Boxes. Where Count... lu Count ld Count rd Count ru , representing the number of crack pixels in the top left, bottom left, bottom right, and top right positions, respectively, where x min The minimum horizontal coordinate of the viewport's coverage area, x max The y-coordinate represents the maximum horizontal coordinate of the viewport's coverage area. min This refers to the minimum vertical coordinate of the viewport's coverage area, y max This refers to the maximum vertical coordinate of the viewport's coverage area.

[0073] Step 5: Use the position and spatial attributes of the viewframes to further merge the viewframes and obtain the final sequence of crack display images.

[0074] Step 5.1: Sort the Boxes in Boxes in descending order based on their area. Place the sorted results into the LBoxes list, setting `ismerged` to 0. `ismerged` indicates whether merging occurred during the calculation; it is 1 if merging occurred, and 0 otherwise. The formula for calculating the area of ​​a Box is as follows: (x...) max -x min )*(y max -y min ), where x min x max y min y max The same meaning as in step 4;

[0075] Step 5.2: Iterate through the next element Box in LBoxes in sequence. s Construct the search scope Query, where Query is a four-tuple (qx min ,qx max ,qy min ,qy max ), where qx min =max(x min -δ x ,0), qy min =max(y min -δ y ,0), Among them, qx min qx max qy min qy max These represent the minimum and maximum values ​​of the search range on the horizontal and vertical axes, respectively. The functions `max()` and `min()` retrieve the maximum and minimum values ​​from these ranges, respectively. δ x δ represents the lateral distance of the expansion. y Indicates the extended vertical distance. S' t width, S' t If there is a next element, proceed to step 5.3; if there is no next element and ismerged is 1, proceed to step 5.1; otherwise, proceed to step 5.7.

[0076] In this embodiment, δ x and δy The values ​​are respectively

[0077] Step 5.3: Traverse all elements in LBoxes. If an element intersects with the search scope Query, denote it as a Box. c The condition for intersection is that both satisfy max(qx) min ,x min )>min(qx max ,x max ), max(qy min ,y min )>min(qy max ,y max ), put Box c Add it to the Cross collection.

[0078] Step 5.4: Traverse the next element Box in the Cross sequence. c If there is no next element, update the spatial distribution property of the viewbox corresponding to the Query using the method in step 4.3, and then execute step 5.1. If the Box... c center If the query falls within the scope of the query, proceed to step 5.6. Otherwise, proceed to step 5.5.

[0079] Step 5.5: Determine the Query Center Compared to Box c The center location, and query Box c The distribution coefficient in the corresponding direction, if the distribution coefficient exceeds the threshold λ b If the above steps are not executed, proceed to step 5.6; otherwise, proceed to step 5.4.

[0080] In this embodiment, λ b The value is 0.4.

[0081] Step 5.6: For Query and Box c Perform a merge operation, updating the values ​​of the first four items representing the range of the Box element in the LBoxes corresponding to the Query as follows:

[0082] min(x1 min x2 min ),max(x1 max x2 max ),min(y1 min ,y2 min ),max(y1 max ,y2 max );

[0083] Where x1 minThis represents the minimum horizontal coordinate of the area covered by the Box, x1. max This represents the maximum horizontal coordinate of the area covered by the Box, y1 min This refers to the minimum vertical coordinate of the area covered by the Box, y1 max This refers to the maximum vertical coordinate of the area covered by the Box, where x2 min Box c Minimum horizontal coordinate of the coverage area, x2 max Box c The maximum horizontal coordinate of the coverage area, y2 min Refers to Box c Minimum vertical coordinate of the coverage area, y2 max Refers to Box c The maximum vertical coordinate of the coverage area is then set, and the value of ismerged is set to 1. Simultaneously, the Box and Box in Cracks are... c Merge the corresponding collections and remove the Box from LBoxes. c Proceed to step 5.3;

[0084] Step 5.7: Obtain the final display effect using the remaining viewports in LBoxes. f The first four items are used to extract the original image P from step 2. t The corresponding regions are used to obtain the final output image sequence LP. f ={P f1 ,P f2 ,…P fm}, where P fm This refers to the m-th output image, where m is the number of output image sequences, which is the same as the number of elements in LBoxes.

[0085] Step 6: Interactive display allows users to select cracks to remove based on the crack display effect, resulting in a new sequence of crack images; users can view the LP (Loop View). f If there are no particularly large cracks causing the viewport to be too large, and the user selects the crack, this invention can delete the corresponding set of crack pixels from Cracks and set P... t The color of the corresponding pixel is changed to red, indicating that the pixel belongs to the category of removing display cracks. Then, steps 4 and 5 are repeated to obtain LP'. f ={P' f1 ,P' f2 ,…P' fj}, where P' fj This refers to the j-th image obtained after re-execution, where j is the number of new images obtained after re-executing steps 4 and 5.

[0086] In this embodiment, a crack spanning the entire scene would result in an excessively large image, making it inconvenient to observe the distribution of cracks with finer granularity. The user can click on one of the pixels of the crack that needs to be removed from the image. Then, the present invention finds the corresponding set of crack pixels in Cracks based on that pixel and performs the deletion operation. In this way, the crack will not participate in the subsequent construction of the view frame and will not affect the merging operation of other view frames.

[0087] The above description is merely an example and illustration of the method of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described or use similar methods to replace them, as long as they do not deviate from the structure of the present invention or exceed the scope defined in the claims, they should all fall within the protection scope of the present invention.

Claims

1. A crack recognition method for large-scale planar scenes based on semantic segmentation, characterized in that, Includes the following steps: Step 1: Obtain a sequence of color images and coordinates of the scene using a color camera mounted on a drone; Step 2: Use a semantic segmentation algorithm to identify crack pixels in the color image sequence, and then use the color image sequence and coordinate sequence for downsampling; Step 3: Calculate the connected crack pixels using the spatial location of the crack pixels, and construct the corresponding crack pixel set; Step 4: Calculate the view frame sequence and the corresponding spatial distribution attributes of the view frames using the crack pixel set; Step 5: Further merge the view frames using the view frame sequence and spatial distribution attributes to obtain the crack display image sequence; Specifically, step 5 includes the following steps: Step 5.1: Sort the view boxes Box in the view box set Boxes in descending order based on the area of ​​the Box. Put the sorted result into the LBoxes list and set ismerged to 0. Ismerged indicates whether merging occurred during the calculation. If merging occurred, ismerged is 1, otherwise it is 0. Step 5.2: Iterate through the next element in LBoxes sequentially. Construct the search scope Query. If there is a next element, proceed to step 5.3; if there is no next element and ismerged is 1, proceed to step 5.1; otherwise, proceed to step 5.

7. Step 5.3: Iterate through all elements in LBoxes. If an element intersects with the search scope Query, record it as... ; Step 5.4: Traverse the next element in the Cross sequence. If there is no next element, update the spatial distribution property of the viewbox corresponding to the Query using the method in step 4.3, and then execute step 5.

1. If the center of the query is within the scope of the query, proceed to step 5.6; otherwise, proceed to step 5.

5. Step 5.5: Determine the Query center relative to... Find the center location and query it. The distribution coefficient in the corresponding direction; if the distribution coefficient exceeds the threshold... If the condition is met, proceed to step 5.6; otherwise, proceed to step 5.

4. Step 5.6, for Query and Perform a merge update operation; Step 5.7: Obtain the final display effect.

2. The crack recognition method for large-scale planar scenes based on semantic segmentation according to claim 1, characterized in that, In step 1, the method for obtaining the color image sequence and coordinate sequence of the scene includes the following steps: Step 1.1: Obtain images collected by the camera. Use a drone to fly around and film the scene, keeping the drone's altitude above the ground constant, with the camera facing downwards, covering an area of... The corresponding color image is one The tensor P, where These represent the height and width of the actual scene covered by the camera, respectively. These represent the height and width of the image obtained through the camera, respectively, where 3 indicates that the image has 3 channels; Step 1.2: Obtain the two-dimensional coordinates C during drone photography. C contains two values. The drone maintains a constant distance from the ground while moving, and its initial position is set to the origin. These represent the horizontal and vertical distances of the shooting position from the origin, respectively. Step 1.3: Based on the time sequence, construct a color image sequence using images P collected at different times and coordinates C. and coordinate sequence n = 1, 2, 3…, where n represents the final number of images taken. This represents the nth image collected. This represents the nth coordinate collected when the drone is moving horizontally. Or move in the vertical direction At that time, take a picture and record the coordinates. The color picture should be able to cover the entire scene.

3. The crack recognition method for large-scale planar scenes based on semantic segmentation according to claim 2, characterized in that, Step 2, which uses a semantic segmentation algorithm to identify crack pixels and perform downsampling, includes the following steps: Step 2.1, Input the semantic segmentation network to obtain the semantic segmentation results for each image. n=1,2,3…, where It is a binary image, where i represents the index of the semantic segmentation result, and its size is the same as the image in the LP. However, the number of channels is 1, which is one. tensor, The value of each pixel in the image is either 0 or 1, where 0 indicates that the pixel does not belong to the crack, and 1 indicates that the pixel belongs to the crack. Step 2.2: Stitch LP and LS together according to their corresponding coordinates C to obtain the original image of the complete scene. and overall segmentation results ; Step 2.3: Scale the overall crack identification result proportionally to obtain... If the scaled pixels correspond to the original overall segmentation result If a region contains a crack pixel, then the value of that pixel is 1; otherwise, it is 0, where s represents the downsampling factor.

4. The crack recognition method for large-scale planar scenes based on semantic segmentation according to claim 3, characterized in that, Step 3 includes the following steps: Step 3.1: Initialize the crack instance set Cracks to an empty set, and the crack pixel access list. All elements in the list are 0, the Search queue is an empty list, and the current crack pixel number is 0. =0, neighboring crack pixel set An empty set, in which Indicates whether the k-th crack pixel has been visited. The element crack instance of Crack is... The set of pixels in the crack, k is The number of pixels in the crack. elements in The pixels in the crack correspond one-to-one, and the elements in the queue to be explored are... The crack pixels in the image; Step 3.2: Obtain the current crack pixel number. The corresponding item in the Visited list If the value is 1, proceed to step 3.4; if it is 0, number the current crack pixel. middle The corresponding pixels are placed in Search, and an empty set is placed in Cracks. ,Past Add the current crack pixel number corresponding The crack pixels in the image; Step 3.3: Get the first element in Search, and set the corresponding... The pixels surrounding the center in eight directions, namely top, bottom, left, right, top right, bottom right, top left, and bottom left, form a set. , They are respectively The eight pixels are: top center, bottom center, left center, right center, top right center, bottom right center, top left center, and bottom left center; traverse these regions. If an element in the array has a pixel value of 1, add it to the array. In the middle, then, towards Add The pixels in the Visited list will be the current crack pixel numbers. Corresponding item Setting it to 1 will delete the first element of Search and add it inside. The pixels in, where This represents the crack pixels surrounding the current pixel; if Search is empty, proceed to step 3.4, otherwise proceed to step 3.3; Step 3.4: Number the current crack pixel. The value is incremented by 1 if If the value is greater than k, proceed to step 4; otherwise, proceed to step 3.

2.

5. The crack recognition method for large-scale planar scenes based on semantic segmentation according to claim 4, characterized in that, Step 4 includes the following steps: Step 4.1: Initialize the view box set Boxes to an empty set, where each view box is an octet ( ),in This represents the minimum horizontal coordinate of the viewport's coverage area. This represents the maximum horizontal coordinate of the viewport's coverage area. This refers to the minimum vertical coordinate of the viewport's coverage area. This refers to the maximum vertical coordinate of the view frame coverage area. lu represents the distribution coefficient of crack pixels in the upper left direction, ld represents the distribution coefficient of crack pixels in the lower left direction, ru represents the distribution coefficient of crack pixels in the upper right direction, and rd represents the distribution coefficient of crack pixels in the lower right direction. Step 4.2: Construct an octet Box, iterate through each element Crack in Cracks, calculate the maximum and minimum values ​​of the horizontal and vertical coordinates of the pixel positions in Crack, and fill them into the first four items of Box; Step 4.3: Calculate the distribution coefficients in the four directions and initialize. , If the value is 0, iterate through each pixel in the Crack, setting the x and y coordinates of the pixel to 0. and , Add one; if and , Add one; if and , Add one; if and , Add one, and assign the value to the last four items of the Box. , , , Place the Box into Boxes; where, , These represent the number of crack pixels in the top left, bottom left, bottom right, and top right positions, respectively. This represents the minimum horizontal coordinate of the viewport's coverage area. This represents the maximum horizontal coordinate of the viewport's coverage area. The minimum vertical coordinate representing the viewport's coverage area. This represents the maximum vertical coordinate of the viewport's coverage area.

6. The crack recognition method for large-scale planar scenes based on semantic segmentation according to claim 5, characterized in that: In step 5.1, the formula for calculating the area of ​​the Box is as follows: ,in , , , The same meaning as in step 4; In step 5.2, Query is a quadruple ( ),in = , = , = , = ,in , , , These represent the minimum and maximum values ​​of the search range on the horizontal axis and the vertical axis, respectively. The functions max() and min() retrieve the maximum and minimum values ​​of the elements within these ranges, respectively. Indicates the lateral distance of the expansion. Indicates the extended vertical distance. ; In step 5.3, the condition for determining intersection is that both conditions simultaneously satisfy max( )>min( ), max( min () ),Will Add it to the Cross collection; In step 5.4, The coordinates of the center are ( , ); In step 5.5, the coordinates of the Query center are ( , ); In step 5.6, the values ​​representing the range of the first four items of the Box element in the LBoxes corresponding to the Query are updated as follows: my( , ),max( ),my( , ),max( ); in, This represents the minimum horizontal coordinate of the area covered by the Box. This represents the maximum horizontal coordinate of the area covered by the Box. This refers to the minimum vertical coordinate of the area covered by the Box. This refers to the maximum vertical coordinate of the area covered by the Box, where express Minimum horizontal coordinate of the coverage area express The maximum horizontal coordinate of the coverage area. It refers to Minimum vertical coordinate of the coverage area It refers to The maximum vertical coordinate of the coverage area is then set, and the value of ismerged is set to 1. Simultaneously, the Box and... Merge the corresponding sets and delete them from LBoxes. Proceed to step 5.3; In step 5.7, the remaining viewboxes in LBoxes are used. The first four items are extracted from the original image in step 2. The corresponding regions are used to obtain the final output image sequence. ,in This refers to the m-th output image, where m is the number of output image sequences, which is the same as the number of elements in LBoxes.

7. A method for displaying cracks in large-scale planar scenes based on semantic segmentation, characterized in that: The crack display image sequence obtained by the crack recognition method for large-scale planar scenes based on semantic segmentation as described in any one of claims 1-6 is used for interactive display.

8. The method for displaying cracks in large-scale planar scenes based on semantic segmentation according to claim 7, characterized in that: Based on the display effect of the cracks, the user selects the cracks that need to be removed, and a new sequence of images showing the cracks is obtained; the user then views the final output image sequence. Check for cracks spanning the entire scene that cause the viewport to be too large. If so, select the crack, delete the corresponding set of crack pixels in Cracks, and then... The color of the corresponding pixel is changed to red, indicating that the pixel belongs to the category of removing display cracks. Then, steps 4 and 5 are repeated to obtain... ,in This refers to the j-th image obtained after re-execution, where j is the number of new images obtained after re-executing steps 4 and 5.

Images