Image analysis system and method for municipal roads

Abstract

The invention provides an image analysis system and method for a municipal road. The method comprises the steps of obtaining an image sequence of a target municipal road section; generating a plurality of disease candidate areas on the target municipal road section based on the image sequence; performing space-time consistency analysis on each disease candidate region according to the structural similarity of each disease candidate region to obtain a region stable value of each disease candidate region; semantic tags of all pixel points in each disease candidate area are determined, and then semantic boundary information of road diseases in different image frames in each disease candidate area is generated; carrying out confidence analysis on semantic boundary information of the road diseases of each disease candidate area in different image frames according to all area stable values to obtain boundary confidence of each disease candidate area; and constructing a confidence thermodynamic diagram of road diseases in the target municipal road section through all boundary confidence coefficients. By adopting the scheme of the invention, the road disease can be dynamically detected based on the space-time correlation characteristics of the continuous images.

Description

Technical Field

[0001] The present application relates to the field of image analysis technology, and more specifically, to an image analysis system and method for municipal roads. Background Art

[0002] Image analysis is a method that uses computer vision and image processing algorithms to identify, extract, analyze and understand information such as targets, structures, features, etc. in static images or dynamic image sequences. It is widely used in medical diagnosis, industrial testing, agricultural monitoring, traffic management, security monitoring and other fields. Its basic goal is to convert the visual information contained in the image into data results with semantic meaning, so as to achieve automated perception and intelligent judgment of the real world. With the development of artificial intelligence, edge computing and multimodal fusion technology, image analysis is evolving towards a more intelligent, real-time and high-precision direction, becoming one of the key supporting technologies for the rapid development of smart healthcare, intelligent manufacturing, precision agriculture and other fields.

[0003] Image analysis for municipal roads is an intelligent analysis method that combines computer vision, image processing, and artificial intelligence technologies to perform structural perception, disease detection, maintenance assessment, and safety warnings on urban road environments. This method automatically processes road images or video sequences to efficiently extract road surface structural features and identify surface defects (such as cracks, potholes, subsidence, fading, missing markings, etc.), providing intelligent support for daily inspections and decision-making management of municipal roads. After completing image analysis, the image analysis system for municipal roads usually performs accurate detection and intelligent identification of road diseases through a multi-stage processing flow. In the existing road disease detection process, most road disease detection is based on static images (single frame). This method often relies on only a single frame image to extract disease features, resulting in high feature instability of static detection methods (i.e., the extracted features are highly sensitive to factors such as environmental changes and image quality fluctuations), thereby reducing the accuracy of road disease detection. Therefore, how to dynamically detect road diseases based on the spatiotemporal correlation characteristics of continuous images has become a difficult problem facing the industry. Summary of the Invention

[0004] The present application provides an image analysis system and method for municipal roads, which can dynamically detect road defects based on the spatiotemporal correlation characteristics of continuous images.

[0005] In a first aspect, the present application provides a road damage detection method, comprising the following steps: Acquire an image sequence of the target municipal road section; generating a plurality of candidate disease areas on the target municipal road section based on pixel grayscale differences and local texture changes in different adjacent image frames of the image sequence; According to the structural similarity of each disease candidate area, the spatiotemporal consistency analysis is performed on each disease candidate area to obtain the regional stability value of each disease candidate area; Determining the semantic labels of all pixels in each candidate defect area through a semantic segmentation model, and then generating semantic boundary information of road defects in each candidate defect area in different image frames based on all semantic labels; Based on the stability values ​​of all regions, confidence analysis is performed on the semantic boundary information of road defects in each candidate defect region in different image frames to obtain the boundary confidence of each candidate defect region; Construct a confidence heat map of road damage in the target municipal road segment using all boundary confidences.

[0006] In some embodiments, generating multiple candidate disease areas on the target municipal road section based on pixel grayscale differences and local texture changes in different adjacent image frames of the image sequence specifically includes: Determining feature change information of the image sequence based on pixel grayscale differences of all adjacent image frames in the image sequence of the target municipal road section; A plurality of candidate disease areas on the target municipal road section are generated according to the feature change information and the change amplitude of the local texture in the image sequence.

[0007] In some embodiments, generating multiple candidate disease areas on the target municipal road section based on the feature change information and the change amplitude of the local texture in the image sequence specifically includes: Dividing each image frame in the image sequence into a plurality of local regions, and then determining a variation range of local textures in different adjacent image frames of the image sequence; generating a plurality of change significance indices of the image sequence according to the change amplitude of the local texture in different adjacent image frames of the image sequence and the feature change information; Based on all the change significance indicators, multiple candidate disease areas on the target municipal road section are generated.

[0008] In some embodiments, a spatiotemporal consistency analysis is performed on each candidate disease region based on the structural similarity of each candidate disease region to obtain a regional stability value for each candidate disease region, specifically including: Obtaining spatial feature information of each disease candidate area in the image sequence; The structural similarity between each two disease candidate areas is determined through all spatial feature information; Determine the temporal co-occurrence frequency of each disease candidate region in the image sequence through all structural similarities; The regional stability value of each disease candidate area is determined by the co-occurrence frequency of all time series.

[0009] In some embodiments, generating semantic boundary information of road defects in each candidate defect area in different image frames based on all semantic labels specifically includes: selecting a disease candidate area as a selected disease candidate area; Determine the semantic consistency area of ​​the selected disease candidate area in different image frames by using the semantic labels of all pixels in the selected disease candidate area; Perform contour extraction on all semantically consistent areas to obtain semantic boundary information of road defects in selected candidate defect areas in different image frames; Continue to determine the semantic boundary information of road diseases in the remaining disease candidate areas in different image frames.

[0010] In some embodiments, confidence analysis is performed on the semantic boundary information of road defects in each candidate defect region in different image frames based on all region stability values ​​to obtain the boundary confidence of each candidate defect region, specifically including: selecting a disease candidate area as a selected disease candidate area; Determine the boundary overlap between multiple semantic boundary information corresponding to the selected disease candidate area; Determine the boundary confidence of the selected disease candidate area according to all boundary coincidences and the regional stability value of the selected disease candidate area; Continue to determine the boundary confidence of the remaining disease candidate areas.

[0011] In some embodiments, constructing a confidence heat map of road hazards in a target municipal road segment using all boundary confidences specifically includes: Obtain the coordinate information of all semantic boundary information corresponding to each disease candidate area; All coordinate information is adjusted by the boundary confidence corresponding to each candidate defect area, and then a confidence heat map of road defects in the target municipal road section is constructed based on the adjustment results.

[0012] In a second aspect, the present application provides an image analysis system for municipal roads, the system including a road damage detection unit, the road damage detection unit including: An acquisition module, used to acquire an image sequence of a target municipal road section; A processing module, configured to generate a plurality of candidate disease areas on a target municipal road section based on pixel grayscale differences and local texture changes in different adjacent image frames of the image sequence; The processing module is further configured to perform a spatiotemporal consistency analysis on each disease candidate region based on the structural similarity of each disease candidate region to obtain a regional stability value of each disease candidate region; The processing module is further configured to determine the semantic labels of all pixels in each candidate defect area through a semantic segmentation model, and then generate semantic boundary information of road defects in each candidate defect area in different image frames based on all the semantic labels; The processing module is further configured to perform confidence analysis on the semantic boundary information of road defects in each candidate defect region in different image frames based on the stability values ​​of all regions, and obtain the boundary confidence of each candidate defect region; An execution module is used to construct a confidence heat map of road damage in the target municipal road segment using all boundary confidences.

[0013] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned image analysis system method for municipal roads when executing the computer program.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-mentioned image analysis system method for municipal roads.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: In the image analysis system and method for municipal roads provided in the present application, an image sequence of a target municipal road section is obtained; a plurality of candidate disease areas on the target municipal road section are generated based on the pixel grayscale differences and local texture changes in different adjacent image frames of the image sequence; a spatiotemporal consistency analysis is performed on each candidate disease area according to the structural similarity of each candidate disease area to obtain a regional stability value of each candidate disease area; the semantic labels of all pixels in each candidate disease area are determined by a semantic segmentation model, and then the semantic boundary information of road diseases of each candidate disease area in different image frames is generated according to all semantic labels; a confidence analysis is performed on the semantic boundary information of road diseases of each candidate disease area in different image frames according to all regional stability values ​​to obtain the boundary confidence of each candidate disease area; and a confidence heat map of road diseases in the target municipal road section is constructed using all boundary confidences.

[0016] It can be seen that in the present application, firstly, after generating multiple candidate disease areas on the target municipal road section through the pixel grayscale differences and local texture changes in different adjacent image frames of the image sequence of the target municipal road section, the regional stability value can be obtained by calculating the structural similarity of each candidate disease area in adjacent frames, thereby quantifying the grayscale distribution and texture structure consistency of the candidate disease area in the time dimension through the regional stability value, thereby effectively filtering out artifacts caused by sudden changes in illumination, shadow interference or vehicle shaking; secondly, a high stability value area means that the morphology and texture of the disease features such as cracks and potholes at this location are always consistent in multiple frames, indicating that they are real diseases rather than occasional noise, that is: by threshold screening the regional stability value, false positive areas that appear briefly or discontinuously can be dynamically eliminated, so that the detection system can focus on persistent and continuously occurring diseases; Subsequently, after obtaining the regional stability value, this application further uses the semantic segmentation model to extract the semantic boundaries of the candidate defect areas in each frame, and performs confidence analysis on these boundaries in combination with the stability value, thereby obtaining the boundary confidence, that is: through the boundary confidence, the spatiotemporal consistency information is integrated into the edge extraction, thereby suppressing the single-frame segmentation error or model prediction drift, and ensuring that the boundary position is more accurate and reliable; secondly, the boundaries with high boundary confidence often correspond to the defect contours that appear consistently in multiple consecutive frames, thereby effectively filtering out false edges caused by road gloss, water accumulation or shadows; thirdly, the boundary confidence can also be used to dynamically update the heat map weights, so that the defect hot zones are concentrated in those areas with continuous, stable and high confidence boundaries; in summary, this scheme can dynamically detect road defects based on the spatiotemporal correlation characteristics of continuous images. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of a road damage detection method according to some embodiments of the present application; Figure 2 is a schematic diagram of a process for determining candidate disease areas according to some embodiments of the present application; Figure 3 This is a schematic diagram of a structure for determining semantically consistent regions according to some embodiments of the present application; Figure 4 is a schematic structural diagram of a road damage detection unit according to some embodiments of the present application; Figure 5 This is a diagram of the internal structure of a computer device for implementing a road hazard detection method according to some embodiments of the present application. DETAILED DESCRIPTION

[0018] In order to better understand the technical solution in this embodiment, the technical solution in this embodiment will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0019] refer to Figure 1 , which is a flow chart of a road damage detection method according to some embodiments of the present application. The road damage detection method 100 mainly includes the following steps: In step 101 , an image sequence of a target municipal road section in a municipal road is acquired.

[0020] In specific implementation, the image sequence of the target municipal road section can be obtained through the road monitoring system of the municipal road; it should be noted that the road monitoring system refers to a monitoring system composed of cameras, transmission equipment, background platforms, etc. in the municipal road, and the road monitoring system is used to monitor the road conditions of the municipal road. In addition, the image sequence refers to the monitoring images continuously collected at set time intervals (for example, every 2 seconds), and the shooting positions between image frames change with the movement of the vehicle or the acquisition equipment. Therefore, the image sequence has both temporal continuity and spatial position gradualness, that is: each frame of the image sequence corresponds to a spatial position segment on the target municipal road section, and the image frames have a strict temporal sequence, and the corresponding road sections can be inferred in space according to the equipment movement parameters (such as speed, frame rate); therefore, the image sequence can be described as a group of time-series image frames with spatial position information, which are used to reflect the changes in the appearance status along the entire target municipal road section.

[0021] In step 102, a plurality of candidate disease areas on the target municipal road section are generated based on the pixel grayscale differences and local texture changes in different adjacent image frames of the image sequence.

[0022] In some embodiments, generating multiple candidate disease areas on a target municipal road section based on pixel grayscale differences and local texture changes in different adjacent image frames of the image sequence can be achieved by using the following steps: Determining feature change information of the image sequence based on pixel grayscale differences of all adjacent image frames in the image sequence of the target municipal road section; A plurality of candidate disease areas on the target municipal road section are generated according to the feature change information and the change amplitude of the local texture in the image sequence.

[0023] In some embodiments, determining the feature change information of an image sequence based on the pixel grayscale differences of all adjacent image frames in an image sequence of a target municipal road section in a municipal road can be achieved by using the following steps: Acquire an image sequence of a target municipal road section in a municipal road; determining pixel grayscale differences between all adjacent image frames in the image sequence; Determining a pixel change rate of each pair of adjacent image frames in the image sequence according to the pixel grayscale difference; Feature change information of the image sequence is generated according to all pixel change rates.

[0024] In a specific implementation, the pixel grayscale difference of all adjacent image frames in the image sequence is determined, that is, the degree of difference between the grayscale values ​​of corresponding pixel positions in the adjacent image frames is used as the pixel grayscale difference of the corresponding adjacent image frames; for example, first, each frame of the image sequence is converted into a grayscale image, and then the grayscale difference of each pixel point in the grayscale image corresponding to all adjacent image frames is determined, and finally, the set consisting of the grayscale difference values ​​of all pixel points is used as the pixel grayscale difference of the corresponding adjacent image frames.

[0025] In a specific implementation, the pixel change rate of each pair of adjacent image frames in the image sequence is determined by the pixel grayscale difference, that is: the proportion of the number of pixels with significant grayscale changes in the image pixels is used as the pixel change rate of the corresponding adjacent image frames; for example, first, a pixel change threshold is set, and then, a plurality of changed pixel points are extracted from the grayscale images corresponding to each adjacent image frame through the pixel change threshold, and then the total number of pixel points of the monitoring image of the target municipal road section is obtained, and finally, the ratio of the number of changed pixel points to the total number of pixel points is used as the pixel change rate of the corresponding adjacent image frames, thereby determining the pixel change rate of each pair of adjacent image frames in the image sequence, preferably, by The pixel change threshold extracts multiple changed pixel points from the grayscale images corresponding to each adjacent image frame, that is: when the pixel grayscale difference of the corresponding pixel points in the adjacent image frames is greater than or equal to the pixel change threshold, the corresponding pixel point is regarded as the changed pixel point; when the pixel grayscale difference of the corresponding pixel points in the adjacent image frames is less than the pixel change threshold, no processing is performed; in addition, the pixel change threshold can be preset based on the noise level of the image sequence. When the noise level is high (such as images collected in a low-light environment), a larger pixel change threshold can be set to avoid misjudging noise as changes. When the noise level is low, a smaller pixel change threshold can be set to capture more subtle changes.

[0026] In specific implementation, the characteristic change information of the image sequence can be generated based on all pixel change rates in the following manner, namely: the pixel change rates corresponding to each pair of adjacent image frames in the image sequence are arranged in sequence according to the chronological order of the corresponding image frame acquisition moments to form a time series, and the time series is used as the characteristic change information of the image sequence.

[0027] It should be noted that the pixel grayscale difference mentioned in this application refers to the set of differences between the grayscale values ​​of any two adjacent image frames at the same pixel position in the image sequence, which is used to reflect the local brightness changes between adjacent image frames; the pixel change rate refers to the proportion of the number of pixels whose pixel grayscale differences between corresponding adjacent image frames exceed the set threshold in the total number of pixels, and the pixel change rate can be used to reflect the overall degree of change between the adjacent image frames; in addition, the feature change information refers to the time series formed by arranging the pixel change rates between all adjacent frames in the image sequence in chronological order, which can be used to reflect the change trend of the pixels in the target municipal road section.

[0028] In some embodiments, reference Figure 2 As shown in the figure, this figure is a schematic diagram of the process of determining candidate disease areas according to some embodiments of the present application. The following steps can be used to generate multiple candidate disease areas on the target municipal road section based on the feature change information combined with the change amplitude of the local texture in the image sequence: First, in 1021, each image frame in the image sequence is divided into a plurality of local regions, and then a variation range of the local texture in different adjacent image frames of the image sequence is determined; Then, in 1022, a plurality of change significance indices of the image sequence are generated based on the change amplitude of the local texture in different adjacent image frames of the image sequence and the feature change information; Finally, in step 1023, multiple candidate disease areas on the target municipal road section are generated based on all change significance indicators.

[0029] Preferably, each image frame in the image sequence can be divided into multiple local areas through fixed grid division in the prior art, wherein the fixed grid division is an image processing method that divides an image into multiple uniform sub-areas (local areas) based on a regular structure; this method generates horizontally and vertically arranged rectangular grids on the image frame according to a preset size and step size according to fixed rules, thereby dividing the entire image into a number of sub-image areas of uniform size and orderly distribution; for example, each image frame in the image sequence can be divided into multiple local areas of uniform size through fixed grid division, such as constructing a regular grid with 32×32 pixels as a unit, thereby ensuring that all image frames have a regional matching relationship of corresponding positions.

[0030] In specific implementation, determining the variation amplitude of the local texture of the image sequence in different adjacent image frames can be achieved in the following manner, namely: first, for each pair of adjacent image frames, determining the texture features of each local area in each pair of adjacent image frames, then determining the degree of difference in the texture features of each local area in each pair of adjacent image frames, and finally, taking the average value of the degree of difference in the texture features of all local areas as the variation amplitude of the local texture of the image sequence in each adjacent image frame; preferably, the texture features of each local area in each pair of adjacent image frames can be determined by statistically analyzing the joint probability distribution of pixel grayscale pairs in each local area in combination with the grayscale co-occurrence matrix method in the prior art. In other embodiments, other methods can also be used, which are not limited here.

[0031] In a specific implementation, a plurality of change significance indices of the image sequence are generated according to the change amplitude of the local texture in different adjacent image frames of the image sequence and the feature change information, that is: the change significance of the corresponding adjacent image frames is measured by the texture change and pixel difference of the image sequence in the corresponding adjacent image frames; for example: first, an adjacent image frame is selected, and the pixel change rate of the adjacent image frame and the change amplitude of the local texture in the adjacent image frame are obtained from the feature change information of the image sequence; secondly, the pixel change rate and the change amplitude of the local texture are weightedly fused to obtain the change significance index of the image sequence in the adjacent image frame; the above steps are repeated to determine the change significance index of the image sequence in the remaining adjacent image frames. The change significance indicators in the remaining adjacent image frames are combined to obtain multiple change significance indicators of the image sequence; preferably, the weight of each feature (pixel change rate and texture change amplitude) can be learned through machine learning methods in the prior art (such as regression models, neural networks, etc.). For example, the pixel change rate and the texture change amplitude are used as input features, and the change significance indicator is used as a label. After the weights of the pixel change rate and the texture change amplitude are obtained through training with a machine learning model (such as linear regression, support vector machine, etc.), the pixel change rate and the change amplitude of the local texture are weightedly fused through a linear weighted algorithm in the prior art. In other embodiments, other methods can also be used, which are not limited here.

[0032] In specific implementation, the following method can be used to generate multiple candidate disease areas on the target municipal road section based on all change significance indicators, namely: first, a pair of adjacent image frames in the image sequence are selected to obtain the spatial position segments of the target municipal road section corresponding to the pair of adjacent image frames. Subsequently, a mapping table of the spatial position segments of the pair of adjacent image frames in the target municipal road section is established, and then all the change significance indicators in the image sequence are arranged in spatial order to obtain a change significance sequence that changes with the length of the target municipal road section. Secondly, a significance threshold of road disease is set, and then the significance threshold is used to traverse the entire municipal road section. A sequence of significant changes is obtained, and spatial position segments that continuously exceed the significance threshold are identified, and all these spatial position segments are marked as pre-processed disease candidate areas. Finally, all pre-processed disease candidate areas are merged based on the interval distance of all pre-processed disease candidate areas (for example, when the interval distance between two adjacent pre-processed disease candidate areas is less than 1 meter, the corresponding two pre-processed disease candidate areas are merged into one pre-processed disease candidate area), and then all the merged pre-processed disease candidate areas are used as the disease candidate areas in this application, thereby obtaining multiple disease candidate areas on the target municipal road section.

[0033] It should be noted that the image frames are collected by a mobile collection device at fixed time intervals (for example, 2 seconds). Preferably, the actual spatial position segment corresponding to each pair of adjacent image frames can be calculated by combining the GPS information or driving speed data obtained in real time by the vehicle during the collection process. The spatial position segment can be expressed as a road section start and end position interval (for example, in meters), thereby realizing the mapping of image frames to road section spatial segments; in addition, the mapping table is used to establish a correspondence between image frame numbers and spatial position segments; its structure can be in the form of a two-dimensional table, and each row records adjacent frame number pairs and corresponding spatial start and end positions; finally, the significance threshold can be set by dynamic calculation, for example, the average of all change significance indicators is used as the significance threshold.

[0034] In addition, it should be noted that the amplitude of change of the local texture described in this application refers to the degree of difference in texture features in each local area in the adjacent image frames, and the structural changes of the image in the local area can be reflected by the local texture; the change significance index is a quantitative value used to measure the degree of change of each pair of adjacent image frames in the image sequence; the candidate defect area refers to the target municipal road section, by analyzing the change significance index in the image sequence, identifying the area with a change significance greater than a set threshold, and the potential defect area can be located by locating the candidate defect area, thereby realizing subsequent disease detection and processing.

[0035] In step 103, a spatiotemporal consistency analysis is performed on each candidate disease region based on the structural similarity of each candidate disease region to obtain a regional stability value of each candidate disease region.

[0036] In some embodiments, performing spatiotemporal consistency analysis on each disease candidate region based on the structural similarity of each disease candidate region to obtain the regional stability value of each disease candidate region can be achieved by the following steps: Obtaining spatial feature information of each disease candidate area in the image sequence; The structural similarity between each two disease candidate areas is determined through all spatial feature information; Determine the temporal co-occurrence frequency of each disease candidate region in the image sequence through all structural similarities; The regional stability value of each disease candidate area is determined by the co-occurrence frequency of all time series.

[0037] It should be noted that the spatial feature information is an information set used to characterize the morphological and structural characteristics of the disease candidate area in its spatial distribution. The spatial feature information can reflect the shape contour, arrangement trend and spatial texture distribution law of the disease candidate area at the corresponding spatial position in the image sequence. Preferably, the shape feature, texture feature and color feature of each disease candidate area can be used as the spatial feature information of each disease candidate area in the image sequence, wherein the area and perimeter of the disease candidate area can be determined as the shape feature of the corresponding disease candidate area; the texture feature of the corresponding disease candidate area can be determined by the gray level co-occurrence matrix of the disease candidate area; and the color feature of the corresponding disease candidate area can be used as the color feature of the corresponding disease candidate area by the color histogram analysis result of the disease candidate area. In other embodiments, other methods can also be used for implementation, which is not limited here.

[0038] In a specific implementation, the structural similarity between each two disease candidate regions is determined through all spatial feature information, that is, the similarity score between the spatial feature information corresponding to each two disease candidate regions is used as the structural similarity between each two disease candidate regions; for example, first, after converting the spatial feature information corresponding to each two disease candidate regions into a vector form, the cosine similarity of the spatial feature information between the two disease candidate regions is determined by the cosine similarity algorithm in the prior art, and then the cosine similarity is used as the structural similarity between the two disease candidate regions. In other embodiments, other methods can also be used, which are not limited here.

[0039] It should be noted that the structural similarity is a quantitative value used to measure the similarity of spatial feature information between two disease candidate regions.

[0040] In a specific implementation, the temporal co-occurrence frequency of each candidate defect region in the image sequence is determined, that is, the number of co-occurrence frames of the candidate defect region in the image sequence (that is, the frequency of the region appearing with a region having similar structural features in the image frame) is used as the temporal co-occurrence frequency of the corresponding candidate defect region in the image sequence. For example, first, a structural similarity threshold is set. For each candidate defect region, the image sequence is sequentially detected frame by frame. If a region with similar structural features to the candidate defect region exists in the current frame (the structural similarity is greater than or equal to the structural similarity threshold), the frame is marked as a co-occurrence frame of the candidate defect region (if the structural similarity is less than the structural similarity threshold, no processing is performed). Then, multiple co-occurrence frames of each candidate defect region are obtained. Finally, the ratio of the total number of co-occurrence frames to the total number of image frames in the image sequence is used as the temporal co-occurrence frequency of the corresponding candidate defect region, thereby obtaining the temporal co-occurrence frequency of each candidate defect region in the image sequence.

[0041] It should be noted that the temporal co-occurrence frequency refers to the frequency at which the disease candidate region and other regions with similar structural features co-occur in the entire image sequence.

[0042] In a specific implementation, the regional stability value of each candidate disease region is determined by all temporal co-occurrence frequencies, that is: the stability (regional stability value) of the corresponding candidate disease region in the time dimension is measured by the consistency of the candidate disease region in the image sequence; for example, the mapping value after converting the temporal co-occurrence frequencies corresponding to all candidate disease regions to a unified scale (such as [0, 1]) is used as the regional stability value of the corresponding candidate disease region; preferably, after obtaining the maximum and minimum values ​​of all temporal co-occurrence frequencies, the normalization algorithm in the prior art can be used to map the temporal co-occurrence frequencies corresponding to all candidate disease regions in [0, 1] to obtain the mapping value of the temporal co-occurrence frequency corresponding to each candidate disease region in [0, 1]. In other embodiments, other methods can also be used for determination, which is not limited here.

[0043] It should be noted that the regional stability value mentioned in this application refers to measuring the stability of the region in the time dimension by analyzing the temporal co-occurrence frequency of the candidate disease region in the image sequence. Through the regional stability value, it is possible to effectively distinguish unstable regions of diseases that continue to appear in the image sequence.

[0044] In step 104, the semantic labels of all pixels in each candidate defect area are determined by a semantic segmentation model, and then the semantic boundary information of the road defect in each candidate defect area in different image frames is generated based on all the semantic labels.

[0045] It should be noted that the semantic segmentation model described in this application is a computer vision task model, which classifies images at the pixel level through a neural network algorithm, thereby assigning each pixel in the image to a specific category; the semantic label refers to the category identifier assigned to each pixel in the image of the target municipal road section, which usually indicates the semantic category to which the pixel belongs; in the detection of municipal road defects, the semantic labels include "cracks", "potholes" and "road damage", etc., indicating the type of defect area; the label of each pixel can be represented by an integer or a string, for example, "0" represents a non-defective area, "1" represents a crack area, and "2" represents a pothole area; as a preferred embodiment, the U-Net network in the prior art can be selected as the semantic segmentation network architecture, and a training data set can be constructed using historical road defect images and their corresponding pixel-level semantic labels. The semantic segmentation network is trained based on the training data set to obtain a semantic segmentation model that can be used for road defect detection. In other embodiments, other methods can also be used for implementation, which is not limited here.

[0046] In some embodiments, generating semantic boundary information of road defects in each candidate defect area in different image frames based on all semantic labels can be achieved by using the following steps: selecting a disease candidate area as a selected disease candidate area; Determine the semantic consistency area of ​​the selected disease candidate area in different image frames by using the semantic labels of all pixels in the selected disease candidate area; Perform contour extraction on all semantically consistent areas to obtain semantic boundary information of road defects in selected candidate defect areas in different image frames; Continue to determine the semantic boundary information of road diseases in the remaining disease candidate areas in different image frames.

[0047] For specific implementation, refer to Figure 3As shown, this figure is a structural schematic diagram of determining semantic consistency areas shown in some embodiments of the present application, wherein the semantic consistency areas of the selected disease candidate area in different image frames are determined by selecting the semantic labels of all pixels in the disease candidate area, that is: first, the semantic label information of all pixels in the disease candidate area is used as semantic features, and then the semantic areas that match the semantic features of the corresponding disease candidate area are extracted from different image frames as the semantic consistency areas of the corresponding disease candidate area; for example, first, the semantic labels of all pixels in the selected disease candidate area are extracted and constructed into a semantic feature set as a reference semantic template for the disease candidate area, and a semantic segmentation operation is performed on each image frame in the image sequence to obtain the semantic feature set corresponding to all pixels in each frame of the image, and then the semantic consistency measure between the reference semantic template and the semantic features of the current image frame is calculated, and a semantic consistency threshold is set. When the semantic consistency value between the reference semantic template and the current image frame is greater than or equal to the consistency threshold, the image frame is marked as the semantic consistency area of ​​the selected disease candidate area; conversely, when the semantic consistency value between the reference semantic template and the current image frame is less than the consistency threshold, no processing is performed.

[0048] It should be noted that the semantic consistency region refers to the region that matches the semantic label information of the selected disease candidate region in different frames of the image sequence. In addition, the semantic label in this application refers to the category identifier of each pixel point. Preferably, the semantic feature set can be represented by a histogram of the proportion of semantic labels of all pixels. In addition, the semantic features extracted from the selected disease candidate region (such as category distribution vector, i.e., histogram) can be used as a template; as a preferred embodiment, the semantic consistency measure value can be obtained by obtaining the probability distribution of different semantic labels from the reference semantic template of the selected disease candidate region (histogram of semantic labels of different categories), and then using the KL divergence (Kullback-Leibler) in the prior art. Divergence) algorithm is used to determine the divergence of semantic features between the selected disease candidate area and the corresponding image frame, and then the divergence is used as the semantic consistency measure between the reference semantic template and the semantic features of the current image frame. In other embodiments, other methods can also be used to determine it, which are not limited here; finally, the semantic consistency threshold can be obtained by statistically analyzing the measurement values ​​between a large number of reference semantic templates and semantic areas of the image frame. For example, when KL divergence is used as the consistency measure, the demarcation threshold (such as 0.1 or 0.15) can be set according to the divergence distribution of positive and negative samples; in addition, the threshold can be determined by a dynamic adjustment strategy based on the mean and standard deviation according to the dynamic fluctuation range of the semantic features in the actual image frame, thereby improving the robustness of the semantic consistency area determination.

[0049] In a specific implementation, contour extraction is performed on all semantically consistent regions to obtain semantic boundary information of road diseases in the selected candidate disease region in different image frames, that is, the contour structure of the semantically consistent region that matches the semantic features of the selected candidate disease region is used as the semantic boundary information of road diseases in the selected candidate region in different image frames; for example, a semantically consistent region is selected, and a binary mask map is constructed according to the pixel values ​​of all pixels in the semantically consistent region. Subsequently, the binary mask map is used as input, and a boundary tracking algorithm in the prior art (such as a boundary extraction method based on chain coding) is used to obtain its boundary contour information, so that the boundary contour information is used as the semantic boundary information of the selected candidate disease region in the image frame. , repeat the above steps to determine the semantic boundary information of the selected candidate defect area in the remaining image frames; preferably, the pixels in the semantic consistency area with the same semantic label (i.e., the semantic category label consistent with the reference semantic template of the selected candidate defect area) are set as foreground pixels (value 1) in the mask image, and the remaining pixels are set as background pixels (value 0), thereby constructing a binary mask image of the semantic consistency area; in addition, as a preferred embodiment, the contour of the connected foreground area can be extracted by the Freeman chain code method, wherein preferably, a minimum contour length threshold can be set to eliminate small non-target areas to ensure that the boundary contour is stable and effective.

[0050] It should be noted that the semantic boundary information refers to the boundary structure data with clear semantic label affiliation extracted around the boundary of a specific semantic area (such as a selected disease candidate area), which usually includes information such as the position of the boundary contour points and the morphological parameters of the area corresponding to the boundary. The semantic boundary information is used to characterize the spatial range and structural characteristics of the semantic area.

[0051] In step 105, confidence analysis is performed on the semantic boundary information of the road damage in each candidate damage region in different image frames based on the stability values ​​of all regions to obtain the boundary confidence of each candidate damage region.

[0052] In some embodiments, confidence analysis is performed on the semantic boundary information of road defects in each candidate defect region in different image frames based on the stability values ​​of all regions. Obtaining the boundary confidence of each candidate defect region can be achieved by the following steps: selecting a disease candidate area as a selected disease candidate area; Determine the boundary overlap between multiple semantic boundary information corresponding to the selected disease candidate area; Determine the boundary confidence of the selected disease candidate area according to all boundary coincidences and the regional stability value of the selected disease candidate area; Continue to determine the boundary confidence of the remaining disease candidate areas.

[0053] In specific implementation, determining the boundary overlap between multiple semantic boundary information corresponding to the selected disease candidate area can be achieved in the following manner, namely: taking the overlapping pixel ratio of the pixel overlapping part of each semantic boundary information corresponding to the selected disease candidate area as the boundary overlap. For example, first, obtain the corresponding semantic boundary information from each image frame of the selected disease candidate area, then perform boundary overlap analysis on each two semantic boundary information to obtain the boundary overlap area corresponding to the two semantic boundary information, and then divide the total number of pixels in the boundary overlap area by the smaller value of the number of pixels in the binary mask images corresponding to the two image frames as the boundary overlap between the corresponding two semantic boundary information.

[0054] It should be noted that the boundary overlap described in the present application is a measure of the degree of overlap between two semantic boundary information (usually the boundary area in the binary mask image), and the boundary overlap can be used to characterize the consistency of the selected disease candidate area in the corresponding image frame in the image sequence; in addition, as a preferred embodiment, boundary overlap analysis is performed on every two semantic boundary information, and after obtaining the binary mask images of the two semantic boundary information, the intersection area of ​​the two binary mask images (that is, the overlapping part of the two) is determined as the boundary overlap area corresponding to the two semantic boundary information, wherein the intersection area of ​​the two binary mask images can be determined by the pixel-level Boolean operation ("AND" operation) in the prior art. In other embodiments, other methods can also be used to determine it, which is not limited here.

[0055] In specific implementation, the boundary confidence of the selected disease candidate area is determined based on all boundary overlaps and the regional stability value of the selected disease candidate area, that is, the boundary confidence of the selected disease candidate area is determined by quantifying the degree of change (regional stability value) and consistency (boundary overlap) of the selected disease candidate area in the image sequence. For example, first, the average value of all boundary overlaps corresponding to the selected disease candidate area is determined, and then the product of the average value and the regional stability value is used as the boundary confidence of the selected disease candidate area.

[0056] It should be noted that the boundary confidence described in this application is a quantitative value used to measure the reliability of the boundary of the selected candidate disease area in the image sequence. Among them, this application provides the consistency of the disease area in different frames by determining the average value of the boundary overlap, and then provides the stability of the disease area in the time series through the regional stability value, and uses the product of the average value and the regional stability value as the boundary confidence of the selected candidate disease area, that is: the boundary confidence of the selected candidate disease area is measured by comprehensively considering the degree of change (regional stability value) and consistency (boundary overlap) of the selected candidate disease area in the image sequence.

[0057] In step 106, a confidence heat map of road defects in the target municipal road section is constructed using the boundary confidence of all candidate defect areas.

[0058] In some embodiments, constructing a confidence heat map of road defects in a target municipal road section using the boundary confidence of all candidate defect regions can be achieved by the following steps: Obtain the coordinate information of all semantic boundary information corresponding to each disease candidate area; All coordinate information is adjusted by the boundary confidence of each candidate defect area, and then a confidence heat map of road defects in the target municipal road section is constructed based on the adjustment results.

[0059] In a specific implementation, the coordinate information of all semantic boundary information corresponding to each candidate defect area is obtained, that is, the specific pixel coordinate set of the boundary coordinates of all semantic boundary information corresponding to the candidate defect area in the image sequence is used as the corresponding coordinate information; as a preferred embodiment, after obtaining a binary mask map of all semantic boundary information corresponding to the candidate defect area, a contour extraction algorithm in the prior art, such as (such as OpenCV's findContours function), an edge detection algorithm (such as Canny) or a morphological gradient algorithm is used to obtain a set of pixel points of all semantic boundary information, and then the two-dimensional coordinate information of each boundary point in all semantic boundary information is extracted as the coordinate information of all semantic boundary information corresponding to the candidate defect area.

[0060] In specific implementation, all coordinate information is adjusted by the boundary confidence of each candidate disease area, and then a confidence heat map of road diseases in the target municipal road section is constructed based on the adjustment results. This can be achieved in the following way: first, a candidate disease area is selected, and the coordinate information corresponding to the candidate disease area is used as the candidate coordinate point of the heat map. Then, the heat map weight value of the candidate coordinate point is set to the boundary confidence value corresponding to the candidate disease area as the importance measure of the pixel point. Subsequently, the boundary coordinate points in all candidate areas are superimposed and drawn on the heat map image corresponding to the entire target municipal road section, and the confidence values ​​of points with the same coordinates are fused (for example, weighted average, maximum value strategy, etc.). Finally, a two-dimensional heat map of the disease distribution in the target municipal road section is generated according to the fusion result of all pixel points as the confidence heat map of road diseases in the target municipal road section.

[0061] It should be noted that the candidate coordinate points of the heat map described in this application refer to the two-dimensional coordinate set of semantic boundary pixel points extracted from the disease candidate area in the image sequence, which are usually expressed as pixel coordinates of (x, y), and are used to participate in weighted drawing as key position points when constructing the confidence heat map; wherein, as a preferred embodiment, the superimposed drawing of the candidate coordinate points of the heat map of all disease candidate areas can be achieved in the following way, namely: first, for the two-dimensional coordinate point (x, y), it is mapped to the corresponding position in the target heat map, and then, if there is a pixel-level offset between the coordinates of different frames, a neighboring pixel expansion strategy (such as 3×3 window diffusion) can be selected. ) to diffuse the heat value of the local area. In addition, when the resolution of the heat map is different from that of the original image, the coordinate projection mapping can be completed by the nearest neighbor interpolation or bilinear interpolation method; in addition, the weighted average refers to the sum of the boundary confidence values ​​of all the coordinate points appearing as the final heat value; the maximum value strategy refers to taking the maximum value of the boundary confidence in all candidate areas where the point appears, indicating the area with the strongest confidence; in summary, the solution of the present application can be used to accurately integrate the semantic boundary information of the candidate area of ​​the disease based on its boundary confidence into the spatial distribution map of the target municipal road section, and construct a confidence heat map that truly reflects the credibility of road diseases in each area.

[0062] In addition, in another aspect of the present application, in some embodiments, the present application provides an image analysis system for municipal roads, the system including a road disease detection unit, referring to Figure 4 , which is a schematic diagram of the structure of a road damage detection unit according to some embodiments of the present application. The road damage detection unit 200 includes: an acquisition module 201, a processing module 202, and an execution module 203, which are described as follows: Acquisition module 201, in this application, acquisition module 201 is mainly used to acquire an image sequence of a target municipal road section; Processing module 202, in this application, the processing module 202 is mainly used to generate multiple candidate disease areas on the target municipal road section based on the pixel grayscale differences and local texture changes in different adjacent image frames of the image sequence; In addition, the processing module 202 in the present application is further configured to perform spatiotemporal consistency analysis on each candidate disease region based on the structural similarity of each candidate disease region to obtain a regional stability value of each candidate disease region; In addition, the processing module 202 in the present application is further configured to determine the semantic labels of all pixels in each candidate defect area through a semantic segmentation model, and then generate semantic boundary information of road defects in each candidate defect area in different image frames based on all the semantic labels; In addition, the processing module 202 in the present application is further configured to perform confidence analysis on the semantic boundary information of road defects in each candidate defect region in different image frames based on the stability values ​​of all regions, and obtain the boundary confidence of each candidate defect region; Execution module 203: In this application, execution module 203 is mainly used to construct a confidence heat map of road hazards in the target municipal road section through all boundary confidences.

[0063] In addition, the present application also provides a computer device, which includes a memory and a processor, wherein the memory stores code, and the processor is configured to obtain the code and execute the above-mentioned road damage detection method.

[0064] In some embodiments, reference Figure 5 , which is an internal structure diagram of a computer device for implementing a road damage detection method according to some embodiments of the present application. The road damage detection method in the above embodiment can be Figure 5 The computer device 300 shown in FIG. 1 is implemented as shown in FIG. 1 , and the computer device 300 includes at least one processor 301 , a communication bus 302 , a memory 303 , and at least one communication interface 304 .

[0065] The processor 301 may be a general-purpose central processing unit (CPU), or an application-specific integrated circuit (ASIC) or one or more processors for controlling the execution of the road damage detection method of the present application.

[0066] The communication bus 302 is used to transmit information between the above components.

[0067] Memory 303 may be, but is not limited to, a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, an optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer. Memory 303 may exist independently and be connected to processor 301 via communication bus 302. Memory 303 may also be integrated with processor 301.

[0068] Memory 303 is used to store program code for implementing the present invention, and is controlled by processor 301 for execution. Processor 301 is configured to execute the program code stored in memory 303. The program code may include one or more software modules. The road defect detection method in the above embodiment can be implemented using processor 301 and one or more software modules in the program code stored in memory 303.

[0069] The communication interface 304 uses any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0070] In a specific implementation, as an example, a computer device may include multiple processors, each of which may be a single-CPU processor or a multi-CPU processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0071] The aforementioned computer device may be a general-purpose computer device or a dedicated computer device. In a specific implementation, the computer device may be a desktop computer, a portable computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiments of this application do not limit the type of computer device.

[0072] In addition, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned road damage detection method.

[0073] In summary, in the image analysis system and method for municipal roads disclosed in the embodiments of the present application, an image sequence of a target municipal road section is obtained; a plurality of candidate disease areas on the target municipal road section are generated based on the pixel grayscale differences and local texture changes in different adjacent image frames of the image sequence; a spatiotemporal consistency analysis is performed on each candidate disease area according to the structural similarity of each candidate disease area to obtain a regional stability value of each candidate disease area; the semantic labels of all pixels in each candidate disease area are determined by a semantic segmentation model, and then the semantic boundary information of road diseases of each candidate disease area in different image frames is generated according to all semantic labels; a confidence analysis is performed on the semantic boundary information of road diseases of each candidate disease area in different image frames according to all regional stability values ​​to obtain the boundary confidence of each candidate disease area; a confidence heat map of road diseases in the target municipal road section is constructed through all boundary confidences; and road diseases can be dynamically detected based on the spatiotemporal correlation characteristics of continuous images.

[0074] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0075] Obviously, those skilled in the art may make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if such changes and modifications fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such changes and modifications.

Claims

1. A road damage detection method for detecting road damage using an image analysis system for municipal roads, characterized in that: The method comprises the following steps: Acquire an image sequence of the target municipal road section; generating a plurality of candidate disease areas on the target municipal road section based on pixel grayscale differences and local texture changes in different adjacent image frames of the image sequence; According to the structural similarity of each disease candidate area, the spatiotemporal consistency analysis is performed on each disease candidate area to obtain the regional stability value of each disease candidate area; Determining the semantic labels of all pixels in each candidate defect area through a semantic segmentation model, and then generating semantic boundary information of road defects in each candidate defect area in different image frames based on all semantic labels; Based on the stability values ​​of all regions, confidence analysis is performed on the semantic boundary information of road defects in each candidate defect region in different image frames to obtain the boundary confidence of each candidate defect region; The confidence heat map of road diseases in the target municipal road section is constructed based on the boundary confidence of all candidate disease areas.

2. The method according to claim 1, wherein Generating multiple candidate disease areas on the target municipal road section based on the pixel grayscale differences and local texture changes in different adjacent image frames of the image sequence specifically includes: Determining feature change information of the image sequence based on pixel grayscale differences of all adjacent image frames in the image sequence of the target municipal road section; A plurality of candidate disease areas on the target municipal road section are generated according to the feature change information and the change amplitude of the local texture in the image sequence.

3. The method according to claim 2, wherein Generating multiple candidate disease areas on the target municipal road section according to the feature change information and the change amplitude of the local texture in the image sequence specifically includes: Dividing each image frame in the image sequence into a plurality of local regions, and then determining a variation range of local textures in different adjacent image frames of the image sequence; generating a plurality of change significance indices of the image sequence according to the change amplitude of the local texture in different adjacent image frames of the image sequence and the feature change information; Based on all the change significance indicators, multiple candidate disease areas on the target municipal road section are generated.

4. The method according to claim 1, wherein Based on the structural similarity of each disease candidate area, a spatiotemporal consistency analysis is performed on each disease candidate area, and the regional stability value of each disease candidate area is obtained, which specifically includes: Obtaining spatial feature information of each disease candidate area in the image sequence; The structural similarity between each two disease candidate areas is determined through all spatial feature information; Determine the temporal co-occurrence frequency of each disease candidate region in the image sequence through all structural similarities; The regional stability value of each disease candidate area is determined by the co-occurrence frequency of all time series.

5. The method according to claim 1, wherein Generating semantic boundary information of road defects in each candidate defect area in different image frames according to all semantic labels specifically includes: selecting a disease candidate area as a selected disease candidate area; Determine the semantic consistency area of ​​the selected disease candidate area in different image frames by using the semantic labels of all pixels in the selected disease candidate area; Perform contour extraction on all semantically consistent areas to obtain semantic boundary information of road defects in selected candidate defect areas in different image frames; Continue to determine the semantic boundary information of road diseases in the remaining disease candidate areas in different image frames.

6. The method according to claim 1, wherein Based on the stability values ​​of all regions, the confidence analysis of the semantic boundary information of road defects in each candidate defect region in different image frames is performed to obtain the boundary confidence of each candidate defect region, including: selecting a disease candidate area as a selected disease candidate area; Determine the boundary overlap between multiple semantic boundary information corresponding to the selected disease candidate area; Determine the boundary confidence of the selected disease candidate area according to all boundary coincidences and the regional stability value of the selected disease candidate area; Continue to determine the boundary confidence of the remaining disease candidate areas.

7. The method according to claim 1, wherein The confidence heat map of road diseases in the target municipal road section is constructed by the boundary confidence of all candidate disease areas, including: Obtain the coordinate information of all semantic boundary information corresponding to each disease candidate area; All coordinate information is adjusted by the boundary confidence corresponding to each candidate defect area, and then a confidence heat map of road defects in the target municipal road section is constructed based on the adjustment results.

8. An image analysis system for municipal roads, comprising a road damage detection unit, characterized in that: The road hazard detection unit comprises: An acquisition module, used to acquire an image sequence of a target municipal road section; A processing module, configured to generate a plurality of candidate disease areas on a target municipal road section based on pixel grayscale differences and local texture changes in different adjacent image frames of the image sequence; The processing module is further configured to perform a spatiotemporal consistency analysis on each disease candidate region based on the structural similarity of each disease candidate region to obtain a regional stability value of each disease candidate region; The processing module is further configured to determine the semantic labels of all pixels in each candidate defect area through a semantic segmentation model, and then generate semantic boundary information of road defects in each candidate defect area in different image frames based on all the semantic labels; The processing module is further configured to perform confidence analysis on the semantic boundary information of road defects in each candidate defect region in different image frames based on the stability values ​​of all regions, and obtain the boundary confidence of each candidate defect region; An execution module is used to construct a confidence heat map of road damage in the target municipal road segment using all boundary confidences.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the road damage detection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the road damage detection method according to any one of claims 1 to 7 are implemented.

Images