A highway facility anomaly identification and real-time early warning method

Abstract

This invention relates to the field of monitoring and early warning, and particularly to a method for anomaly identification and real-time early warning of highway facilities. The invention divides the road contour into sub-windows and analyzes the image parameter variation characteristics along the extension direction to determine the visual interference type of each sub-target area. Subsequently, based on the visual interference type corresponding to the sub-window, the inspection image data stream is analyzed. Under the condition of interference stability, a mapping relationship between sub-windows and coded blocks is established, and the stability characteristics of the coded data in the time domain are directly analyzed to quickly identify abnormal time periods. Decoding is performed only on images from abnormal time periods, significantly reducing computational load. For cases where interference stability is not met, direct decoding and target identification are performed to ensure continuous detection reliability. This invention improves image analysis efficiency while ensuring the accuracy of anomaly identification, and is suitable for processing large volumes of inspection image data streams, improving the efficiency of large-scale intelligent early warning for highway facilities.

Description

Technical Field

[0001] This invention relates to the field of monitoring and early warning, and in particular to a method for identifying and providing real-time early warning of anomalies in highway facilities. Background Technology

[0002] As the lifeline of national transportation, the integrity of highways' roadside facilities (such as guardrails, signs, and road surfaces) directly affects traffic safety and operational efficiency. Traditional manual inspection methods have inherent drawbacks such as limited coverage, low efficiency, and poor real-time performance, making it difficult to promptly detect sudden damage or safety hazards and issue early warnings. In recent years, drone inspection and early warning technology has been gradually applied, enabling rapid and wide-area image acquisition through aerial photography, significantly improving the mobility of inspection and early warning operations and enhancing the timeliness and accuracy of early warnings.

[0003] For example, Chinese Patent Publication No. CN119323869A discloses a dynamic early warning system for highway defects using a drone equipped with a sensor-computing device. This system, belonging to the field of intelligent transportation technology, includes an operation platform, a drone, a sensor-computing integrated roadside base station, a drone housing, and an onboard unit. The drone, equipped with the sensor-computing integrated roadside base station, collects real-time images of the highway surface and reports them to the drone housing via an image transmission system. The drone housing then transmits the highway surface images back to the operation center via a local area network. The sensor-computing integrated roadside base station identifies road defect information in real time, generates a V2X standard message set containing road anomaly information, and dynamically broadcasts it to the onboard unit. This invention utilizes a drone carrying a sensor-computing intelligent device flying on a highway. The sensor-computing intelligent roadside device collects real-time images of the highway surface, greatly improving inspection efficiency and revolutionizing the traditional "vehicle-to-roadside" model to "roadside-to-vehicle," significantly reducing the risk of accidents.

[0004] However, the following problems still exist in the existing technology. Due to the nature of the environment, highways cover a wide area and generate a large number of inspection images. However, anomaly analysis of these images is inefficient and computationally expensive. When using drones for monitoring, the dynamic movement of drones and the high consistency of the highway scene along their movement direction result in relatively stable inspection images in the time domain. Existing technologies do not utilize the continuous acquisition characteristics of drones to analyze the anomalies of inspection images from a stable perspective, leading to low efficiency in analysis and early warning. Summary of the Invention

[0005] To address this, the present invention provides a method for identifying and providing real-time early warning of anomalies in highway facilities, thereby overcoming the problem in the prior art where, due to the wide coverage of highways and the large number of inspection images collected by drones, the mobile characteristics of drones and the consistency of highway scenes are not utilized for anomaly analysis, resulting in low efficiency in analysis and early warning.

[0006] To achieve the above objectives, the present invention provides a method for identifying and providing real-time early warning of highway facility anomalies, comprising: Control the drone to pass through several target areas of the highway and collect inspection image data streams in real time and upload them to the cloud; In response to the drone moving to the edge of the target area, a panoramic feature acquisition action is performed, including increasing the drone's flight altitude, acquiring a panoramic image of the adjacent target area, and then restoring the reference altitude; Several visual windows are constructed within the panoramic image based on the direction of highway extension; Extended interferometry analysis is performed based on several sub-target regions in a panoramic image, including determining the variation characteristics of image parameters of sub-windows within each sub-target region in the direction of highway extension. The sub-windows are determined based on road contour features and are classified into a first type of sub-window and a second type of sub-window. Based on the aforementioned change characteristics, determine the visual interference type of the sub-windows within each sub-target region; The sub-target area where the UAV is located is determined, and the inspection image data stream is analyzed based on the visual interference type corresponding to the sub-view window, including: The mapping relationship between sub-windows and coding blocks is determined, and the stable characteristics of the coding data corresponding to the coding block in the time domain are analyzed to lock the abnormal time domain. The inspection image data stream in the abnormal time domain is decoded to perform anomaly identification and early warning. Decode all inspection image data streams to identify and warn of anomalies; Among them, the anomaly identification and early warning includes identifying whether there are abnormal targets in the decoded frame, marking the decoded frame containing abnormal targets as a potential abnormal frame, and issuing an early warning message.

[0007] Furthermore, the process of constructing a plurality of visual windows within the panoramic image based on the highway's extension direction includes, The centerline of the highway is determined as the baseline in the panoramic image; A plurality of visual windows are constructed with the baseline as the center line extending in the direction of the highway, so that the visual windows cover the area of ​​the highway in which there are no vehicle outlines. Each of the aforementioned visual windows is a rectangular area used to select the content of the panoramic image.

[0008] Furthermore, the process of determining sub-windows based on road contour features and determining the type of the resulting sub-windows includes, Determine the road contour features within the visual window of the panoramic image, including guardrail contours and road line contours; A strip-shaped area of ​​predetermined width is determined on both sides of the guardrail outline to obtain the first type of sub-window; A strip-shaped area that does not include the road line outline is determined between the guardrail outlines to obtain a second type of sub-window.

[0009] Furthermore, the process of determining the variation characteristics of the image parameters of the sub-window in the direction of highway extension includes, Visual windows are determined one by one along the direction of the highway in the panoramic image; Determine the rate of change of the first image parameter of the first type of sub-window and the rate of change of the second image parameter of the second type of sub-window among several adjacent visual windows contained in the sub-target region; The mean rate of change of the first image parameters and the mean rate of change of the second image parameters are calculated and used as the change features.

[0010] Furthermore, the process of determining the visual interference type of sub-windows within each sub-target region based on the aforementioned change characteristics includes, Determine the mean rate of change of the first image parameter and the mean rate of change of the second image parameter corresponding to the sub-target region; If the average rate of change of the first image parameter of the sub-target region is greater than the predetermined first rate of change threshold, the first type of sub-window of the sub-target region is determined to be a strong visual interference type. If the average rate of change of the second image parameter of the sub-target region is greater than the predetermined second rate of change threshold, the second type of sub-window of the sub-target region is determined to be a strong visual interference type.

[0011] Furthermore, the analysis of the inspection image data stream based on the visual interference type corresponding to the sub-window includes, If the interference stability condition is met, the mapping relationship between the sub-window and the coding block is determined, and the stability characteristics of the coding data corresponding to the coding block in the time domain dimension are analyzed to lock the abnormal time domain. The inspection image data stream in the abnormal time domain is decoded to perform anomaly identification and early warning. If the interference stability condition is not met, decode all inspection image data streams and perform anomaly identification and early warning. The interference stability condition is that there are no sub-windows with strong visual interference.

[0012] Furthermore, the process of determining the mapping relationship between sub-windows and coded blocks includes, The inspection images collected by the UAV when it flies at a reference altitude are predetermined, and sub-windows are labeled on the inspection images, including a first type of sub-window and a second type of sub-window; Determine the position of the sub-window in the inspection image, determine the corresponding coding block, and construct the mapping relationship between the sub-window and the coding block.

[0013] Furthermore, the process of analyzing the stable characteristics of the encoded data corresponding to the encoded block in the time domain includes, Determine the coded block mapped by the sub-window and extract the coded parameters; The temporal variation curve of the coding parameters is constructed based on the temporal sequence of the inspection image data stream; Extract the stability characteristics of several curve segments corresponding to the time-domain variation curve, including amplitude variance and average amplitude.

[0014] Furthermore, the process of locking onto the anomalous time domain includes, Identify the time domain segment corresponding to the curve segment that meets the anomaly condition, and define the time domain segment as the abnormal time domain; Among them, the abnormal conditions include the amplitude variance being greater than the predetermined variance threshold or / and the average amplitude not belonging to the predetermined stable amplitude range.

[0015] Furthermore, the early warning information includes the location and time of the collection of potentially abnormal frames by the drone.

[0016] Compared with existing technologies, this invention controls a drone to collect inspection image data streams along a target area of ​​a highway. It acquires panoramic images by increasing the altitude at the edge of the area, divides the road contour into sub-windows, and analyzes the image parameter variation characteristics along the extension direction. Based on these variation characteristics, it determines the visual interference type of each sub-window within the target area. Subsequently, it analyzes the inspection image data stream based on the visual interference type corresponding to the sub-window. Under the condition of interference stability, it establishes a mapping relationship between sub-windows and coded blocks, directly analyzing the stability characteristics of the coded data in the time domain to quickly identify abnormal time periods. Only images from abnormal time periods are decoded and anomaly identification and early warning are performed, significantly reducing computational load. For cases where interference stability is not met, it directly decodes and performs target identification, ensuring continuous detection reliability. This invention improves image analysis efficiency while ensuring the accuracy of anomaly identification. It is suitable for processing large volumes of inspection image data streams, improving the efficiency of large-scale intelligent inspection of highway facilities, ensuring the accuracy of early warnings, and thus improving highway safety.

[0017] In particular, this invention acquires panoramic images of the target area to be entered by performing panoramic feature acquisition, providing data support for subsequent extended interferometry analysis. It constructs continuously arranged visual windows to select road areas and builds sub-windows based on road contour features. In reality, the overall image parameters of highway road areas in the extension direction are highly stable, but the repeatability and stability of different areas vary. Based on this, this invention uses road contour features to determine and classify sub-windows. The first type of sub-window reflects the strip-shaped areas on both sides of the road, mainly reflecting the guardrail situation. The second type of sub-window reflects the non-marking areas on the road. The construction of sub-windows is to observe the stability of the image parameters of the corresponding areas in continuous space and determine the visual interference type of the sub-windows corresponding to different sub-target areas. This provides data support for subsequent dynamic inspection by UAVs and selection of different analysis methods for the inspection image data stream.

[0018] In particular, this invention analyzes the inspection image data stream based on the visual interference type of the sub-window. In practice, UAVs have dynamic shooting characteristics, and the acquired inspection images have spatial continuity. Furthermore, combined with the mechanism of high image parameter stability in the direction of highway extension, different analysis methods can be adopted. When the sub-target area meets the interference stability condition, due to the high stability of the image parameters, the corresponding encoded data also exhibits highly stable properties. Moreover, due to the construction of the sub-window, background interference is reduced, making the data representation of the encoded data stronger. By using the pre-constructed mapping relationship between the sub-window and the encoding block, only the encoding parameters of the corresponding encoding block are extracted for analysis. By analyzing the anomalies of the encoding parameters in the time domain, abnormal time domains are locked before decoding. Such decoded frames have a high probability of potential anomalies. Therefore, when dealing with a large number of highway inspection image data streams, selective decoding can be performed, and anomalies can be identified. This method is reliable, reduces computing power consumption, and improves analysis efficiency.

[0019] In particular, when the sub-target region does not meet the interference stability condition, it reflects the poor stability of image parameters in the direction of highway extension. For example, there may be vegetation encroachment on the highway guardrail, light and shadow on the road, and other situations. In such cases, a large amount of background noise will be introduced, resulting in poor data representation of the coded data corresponding to the inspection image. Therefore, in this case, after full decoding, an image processing model is used to identify abnormal targets, avoid feature omissions, and improve reliability. Furthermore, by adaptively selecting the analysis method of the inspection image data stream, the efficiency of highway inspection can be improved. Attached Figure Description

[0020] Figure 1 A schematic diagram illustrating the steps of a method for identifying and providing real-time early warning of highway facility anomalies according to an embodiment of the invention; Figure 2A logic block diagram for determining the visual interference type of sub-windows within each sub-target region in an embodiment of the invention; Figure 3 This is a logic block diagram illustrating the analysis of inspection image data streams based on the visual interference type corresponding to sub-windows, as described in an embodiment of the invention. Figure 4 This is a logic block diagram for locking the abnormal time domain according to an embodiment of the invention. Detailed Implementation

[0021] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0022] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0023] It should be noted that in the description of this invention, the terms "upper," "lower," "left," "right," "inner," and "outer," etc., which indicate directional or regional relationships, are based on the directional or regional relationships shown in the accompanying drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0024] Please see Figure 1 The diagram shown illustrates the steps of a highway facility anomaly identification and real-time early warning method according to an embodiment of the present invention. The highway facility anomaly identification and real-time early warning method according to an embodiment of the present invention includes: Step S1: Control the drone to pass through several target areas of the highway and collect inspection image data streams in real time and upload them to the cloud. Step S2, in response to the drone moving to the edge of the target area, performs a panoramic feature acquisition action, including increasing the drone's flight altitude, acquiring a panoramic image of the adjacent target area, and then restoring the reference altitude; Step S3: Construct several visual windows within the panoramic image based on the direction of highway extension; Step S4, based on several sub-target regions in the panoramic image, performs extended interferometry analysis, including determining the variation characteristics of image parameters of sub-windows in each sub-target region in the direction of highway extension. The sub-windows are determined based on road contour features and are classified into a first type of sub-window and a second type of sub-window. Step S5: Determine the visual interference type of the sub-windows within each sub-target region based on the change characteristics; Step S6: Determine the sub-target area where the UAV is located, and analyze the inspection image data stream based on the visual interference type corresponding to the sub-view window, including: The mapping relationship between sub-windows and coding blocks is determined, and the stable characteristics of the coding data corresponding to the coding block in the time domain are analyzed to lock the abnormal time domain. The inspection image data stream in the abnormal time domain is decoded to perform anomaly identification and early warning. Decode all inspection image data streams to identify and warn of anomalies; Among them, the anomaly identification and early warning includes identifying whether there are abnormal targets in the decoded frame, marking the decoded frame containing abnormal targets as a potential abnormal frame, and issuing an early warning message.

[0025] Specifically, there are no restrictions on the method of identifying abnormal targets. Existing open-source target detection image models can be used to identify abnormal targets in the image, such as road cracks, road depressions, guardrail cracks, guardrail depressions, etc. Alternatively, an image processing model that can identify abnormal targets can be trained independently. This is existing technology and will not be elaborated further.

[0026] Specifically, the process of constructing a plurality of visual windows within the panoramic image based on the highway's extension direction includes: The centerline of the highway is determined as the baseline in the panoramic image; A plurality of visual windows are constructed with the baseline as the center line extending in the direction of the highway, so that the visual windows cover the area of ​​the highway in which there are no vehicle outlines. Each of the aforementioned visual windows is a rectangular area used to select the content of the panoramic image.

[0027] Specifically, the visual window is symmetrical to the centerline of the highway, with a length of 1.2 times the corresponding width of the highway to cover the highway. The width is determined based on the length of the highway in the panoramic image and is set between 0.05 and 0.1 times the corresponding length of the highway. In practice, 0.05 times is selected to observe the sub-target areas of the highway in the panoramic image.

[0028] Specifically, for panoramic images, the vehicle outline can be identified first. If the constructed visual window does not coincide with the vehicle outline, it is considered a qualified visual window. If the constructed visual window coincides with the vehicle outline, the visual window is deleted. The purpose of this is to reduce the interference of the vehicle outline.

[0029] Specifically, inspections should be conducted at times with less traffic to minimize the impact of vehicles on the inspection process.

[0030] Specifically, since highways are fixed targets, panoramic images can be collected at the same height and from a fixed angle at fixed locations on the highway. Thus, the panoramic images acquired at each fixed location during each inspection have the same scale and shooting angle. Therefore, the positions of the first type of sub-window and the second type of sub-window in the panoramic image of each fixed location can be predetermined, which will not be elaborated further.

[0031] Specifically, when controlling the drone for inspection, the drone is controlled to fly along the center line of the highway to avoid obstruction and capture the guardrails on both sides. The flight altitude is kept relatively constant relative to the highway surface. Different flight parameters can be set for different road sections after manual flight in advance to maintain the relative altitude between the drone and the highway as much as possible. This altitude is the reference altitude, which is selected within 30m to 40m to capture more details. In practice, 35m is preferred. When not performing panoramic feature acquisition, the viewing angle of the inspection images captured by the drone remains fixed. Based on this, the inspection route, flight altitude, and shooting angle of the drone in each road section are fixed, and the scale and shooting angle of the inspection images are very similar. Similarly, the positions of the first type of sub-window and the second type of sub-window can be pre-determined and applied to quickly calibrate the first type of sub-window and the second type of sub-window during subsequent inspections. This will not be elaborated further.

[0032] In situations involving tunnels, drones need to adapt to the tunnel altitude and decode all the inspection image data streams acquired within the tunnel area to directly identify and issue early warnings for anomalies.

[0033] Specifically, there is no limitation on the division of the target area; it is only necessary to cover the highways to be inspected. For example, in implementation, the highway can be divided at intervals of 500m to 1000m to construct several target areas. In this implementation, 5000m is selected to ensure that the panoramic images collected by the UAV are analyzable and to reduce the loss of details when the range is too large. Of course, those skilled in the art can make adjustments according to the actual situation, which will not be elaborated here.

[0034] The sub-target area is a further division of the target area, with the purpose of observing the highway in segments. In practice, the sub-target area can be further divided into 5 sub-target areas, which will not be elaborated further here.

[0035] It is understandable that, due to the fixed inspection route and fixed shooting parameters, the division of the target area and sub-template area can remain unchanged in subsequent inspections. Thus, the correspondence between each part in the panoramic image and the target area and sub-target area can be determined, which will not be elaborated further.

[0036] Specifically, after the drone collects the inspection image data stream, it can upload it to the cloud for analysis. The inspection image data stream includes the coded data corresponding to the inspection image.

[0037] Specifically, the cloud can simultaneously collect inspection image data streams uploaded by multiple drones to cover the inspection of large areas of highways, which will not be elaborated further.

[0038] Specifically, the process of determining sub-windows based on road contour features and determining the type of the resulting sub-windows includes: Determine the road contour features within the visual window of the panoramic image, including guardrail contours and road line contours; A strip-shaped area of ​​predetermined width is determined on both sides of the guardrail outline to obtain the first type of sub-window; A strip-shaped area that does not include the road line outline is determined between the guardrail outlines to obtain a second type of sub-window.

[0039] Specifically, no specific limit is made on the predetermined width. Those skilled in the art can predetermine the scale of the panoramic image captured by the UAV when performing panoramic feature acquisition. The 50cm in the actual space is converted to the size in the panoramic image through the scale to obtain the predetermined width. The reason for using 50cm is that it can include the guardrail and introduce less background area.

[0040] Specifically, there are no restrictions on the specific methods for identifying the outlines of guardrails and road markings in panoramic images. For example, existing image segmentation algorithms can be used, or an image processing model that can identify the corresponding targets can be trained independently. These are existing technologies and will not be elaborated further.

[0041] It is understandable that highway road markings are usually used to divide lanes, and the area of ​​non-road markings accounts for a relatively large proportion. Therefore, it is necessary to construct a second type of sub-window to observe the main area of ​​the road and reduce the interference caused by road markings. For example, a rectangular area can be constructed within the lane to form a second type of sub-window, which will not be elaborated further.

[0042] This invention acquires panoramic images of the target area by performing panoramic feature acquisition, providing data support for subsequent extended interferometry analysis. It constructs continuously arranged visual windows to select road areas and builds sub-windows based on road contour features. In reality, the overall image parameters of highway road areas in the extension direction are highly stable, but the repeatability and stability of different areas vary. Therefore, this invention uses road contour features to determine and classify sub-windows. The first type of sub-window reflects the strip-shaped areas on both sides of the road, mainly reflecting the guardrail situation. The second type of sub-window reflects the non-marking areas on the road. The construction of sub-windows aims to observe the stability of image parameters in continuous space for the corresponding areas and determine the visual interference type of sub-windows corresponding to different sub-target areas. This provides data support for subsequent dynamic inspection using UAVs and the selection of different analysis methods for the inspection image data stream.

[0043] Specifically, the process of determining the variation characteristics of the image parameters of the sub-view window in the direction of highway extension includes: Visual windows are determined one by one along the direction of the highway in the panoramic image; Determine the rate of change of the first image parameter of the first type of sub-window and the rate of change of the second image parameter of the second type of sub-window among several adjacent visual windows contained in the sub-target region; The mean rate of change of the first image parameters and the mean rate of change of the second image parameters are calculated and used as the change features.

[0044] Specifically, the first image parameter change rate and the second image parameter change rate are calculated in the same way, which is the ratio of the absolute difference of the image parameters of the sub-windows in two adjacent visual windows to the mean of the image parameters of the sub-windows.

[0045] Specifically, image parameters can be selected from contrast, chroma, and brightness, with chroma being the preferred option. The reason for choosing chroma is that highway inspection and maintenance usually focus on relatively serious defects in the road surface and guardrails, such as road surface cracks, deep potholes, severe deformation of guardrails, or guardrail breakage. Such anomalies can be reflected in chroma, and chroma is easy to extract, thereby improving analysis efficiency.

[0046] Specifically, please refer to Figure 2 The diagram shown is a logic block diagram for determining the visual interference type of sub-windows within each sub-target region according to an embodiment of the invention. The process of determining the visual interference type of sub-windows within each sub-target region based on the change characteristics includes: Determine the mean rate of change of the first image parameter and the mean rate of change of the second image parameter corresponding to the sub-target region; If the average rate of change of the first image parameter of the sub-target region is greater than the predetermined first rate of change threshold, the first type of sub-window of the sub-target region is determined to be a strong visual interference type. If the average rate of change of the second image parameter of the sub-target region is greater than the predetermined second rate of change threshold, the second type of sub-window of the sub-target region is determined to be a strong visual interference type.

[0047] Specifically, when setting the first rate of change threshold and the second rate of change threshold, several panoramic image samples are acquired in advance. Those skilled in the art select panoramic image samples that show no road damage, no debris accumulation, no shadow coverage, and no vegetation obstruction on the guardrails. Then, extended interferometry analysis is performed to statistically analyze the first rate of change and the second rate of change. The normal distributions of the first rate of change and the second rate of change are determined respectively. The upper limit of the 95% confidence interval of the normal distribution of the first rate of change is used as the first rate of change threshold, and the upper limit of the 95% confidence interval of the normal distribution of the second rate of change is used as the second rate of change threshold.

[0048] Specifically, please refer to Figure 3 As shown, it is a logic block diagram of an embodiment of the invention for analyzing inspection image data streams based on the visual interference type corresponding to sub-windows. The analysis of inspection image data streams based on the visual interference type corresponding to sub-windows includes, If the interference stability condition is met, the mapping relationship between the sub-window and the coding block is determined, and the stability characteristics of the coding data corresponding to the coding block in the time domain dimension are analyzed to lock the abnormal time domain. The inspection image data stream in the abnormal time domain is decoded to perform anomaly identification and early warning. If the interference stability condition is not met, decode all inspection image data streams and perform anomaly identification and early warning. The interference stability condition is that there are no sub-windows with strong visual interference.

[0049] It is understandable that when decoding the inspection image data stream in the abnormal time domain, the obtained decoded frames may contain vehicles. Therefore, in some possible implementations, the decoded frames obtained after decoding the inspection image data stream in the abnormal time domain can identify whether there are abnormal targets. Decoded frames containing abnormal targets are marked as abnormal frames, and warning information is issued. This operation can still significantly reduce the amount of data processing, and only some decoded frames are processed using image processing models.

[0050] This invention analyzes inspection image data streams based on the visual interference types of sub-windows. In practice, UAVs possess dynamic shooting characteristics, and the acquired inspection images exhibit spatial continuity. Furthermore, combined with the mechanism of high image parameter stability along the highway extension direction, different analysis methods can be employed. When the sub-target region meets the interference stability condition, due to the high stability of image parameters, the corresponding encoded data also exhibits highly stable properties. Moreover, the construction of sub-windows reduces background interference, making the data representation of encoded data stronger. Utilizing the pre-constructed mapping relationship between sub-windows and encoded blocks, only the encoding parameters of the corresponding encoded blocks are extracted for analysis. By analyzing the anomalies of encoding parameters in the temporal domain, abnormal temporal domains are identified before decoding. Such decoded frames have a higher probability of potential anomalies. Therefore, selective decoding is possible when dealing with large volumes of highway inspection image data streams, and anomalies can be identified, demonstrating reliability, reducing computational consumption, and improving analysis efficiency.

[0051] The failure of the sub-target region to meet the interference stability condition reflects the poor stability of image parameters along the highway extension direction. For example, there may be vegetation encroachment on the highway guardrail, light and shadow on the road, and other situations. In such cases, a large amount of background noise will be introduced, resulting in poor data representation of the coded data corresponding to the inspection image. Therefore, in this case, after full decoding, an image processing model is used to identify abnormal targets, avoid feature omissions, and improve reliability. Furthermore, by adaptively selecting the analysis method of the inspection image data stream, the efficiency of highway inspection can be improved.

[0052] Specifically, the process of determining the mapping relationship between sub-windows and coded blocks includes, The inspection images collected by the UAV when it flies at a reference altitude are predetermined, and sub-windows are labeled on the inspection images, including a first type of sub-window and a second type of sub-window; Determine the position of the sub-window in the inspection image, determine the corresponding coding block, and construct the mapping relationship between the sub-window and the coding block.

[0053] It is understandable that, since the drone flies along the center line of the highway and maintains its altitude, the shooting angle is fixed. Therefore, the position of the sub-window in each inspection image is relatively fixed. Thus, the correspondence between the corresponding position and the coding block can be determined.

[0054] Specifically, the process of parsing the stable characteristics of the encoded data corresponding to the coded block in the time domain includes, Determine the coded block mapped by the sub-window and extract the coded parameters; The temporal variation curve of the coding parameters is constructed based on the temporal sequence of the inspection image data stream; Extract the stability characteristics of several curve segments corresponding to the time-domain variation curve, including amplitude variance and average amplitude.

[0055] Specifically, the encoding parameters can be selected from one or more of the following: quantization parameter QP, bit rate, and energy of the change coefficient. When multiple parameters are selected, corresponding time-domain change curves are constructed respectively.

[0056] The quantization parameter QP is the index of the quantization step, which is usually an integer from 0 to 51. It is used to control the quantization precision of the frequency domain coefficients during image compression. The smaller the value, the smaller the quantization step, the more details are retained, and the larger the bitstream. The larger the value, the larger the quantization step, the higher the compression degree, and the smaller the bitstream. Bitrate is the amount of video encoded data transmitted or stored per unit of time. Commonly used units are kilobits per second (kbps) or megabits per second (Mbps). It is dynamically allocated by the encoder according to the complexity of the picture and preset quality control parameters, and directly reflects the bandwidth usage and detail retention of the compressed stream. The energy of the variation coefficient refers to the sum of the squares (or the sum of the absolute values) of all frequency domain coefficients after an image patch undergoes discrete cosine transform or similar orthogonal transform. It is used to measure the signal strength and texture richness contained in the image patch. The higher the energy value, the more edges, details, or noise components there are in the region, and the larger the prediction residual.

[0057] Extraction of encoded data does not require pixel reconstruction; it can be extracted from the encoded data stream. This is existing technology and will not be elaborated further.

[0058] It is understandable that the drone continuously collects inspection images in the time domain. Different times correspond to different inspection images, and different inspection images will have corresponding coded data. When constructing the time domain change curve, the coding parameters corresponding to the coding blocks at the same position in different inspection images are determined, and the time domain change curve is constructed with time as the horizontal axis and the coding parameters as the vertical axis.

[0059] It is understandable that the temporal variation curve refers to the encoded data corresponding to the same coding block. Therefore, there are multiple temporal variation curves when there are multiple coding blocks.

[0060] Specifically, please refer to Figure 4 As shown, it is a logic block diagram of locking the abnormal time domain according to an embodiment of the invention. The process of locking the abnormal time domain includes, Identify the time domain segment corresponding to the curve segment that meets the anomaly condition, and define the time domain segment as the abnormal time domain; Among them, the abnormal conditions include the amplitude variance being greater than the predetermined variance threshold or / and the average amplitude not belonging to the predetermined stable amplitude range.

[0061] Specifically, the process of determining the variance threshold and the stable amplitude range includes: pre-determining a number of image samples that meet the interference stability conditions; having a person skilled in the art select image samples in which there are no abnormalities in the road and guardrail; constructing a time-domain variation curve; statistically analyzing the amplitude variance corresponding to a number of curve segments; and solving for the maximum amplitude variance as the predetermined variance threshold. Understandably, inspections are usually conducted in sunny weather and during periods of low traffic volume. The applicable image samples need to be manually screened in advance to remove image samples collected on cloudy days or in special weather conditions, in order to ensure the generalization ability to obtain variance threshold values ​​and stable amplitude ranges as much as possible.

[0062] By statistically analyzing the average amplitude corresponding to several curve segments and solving for each average amplitude, the 95% confidence interval is determined as the stable amplitude range.

[0063] Specifically, the warning information includes the location and time of the collection of potentially abnormal frames by the drone.

[0064] Understandably, when sending early warning information, potential anomaly frames can be sent simultaneously for staff reference. Based on the early warning information, the time and location of the anomaly can be confirmed, so that staff can arrange relevant maintenance matters.

[0065] If the highway facility anomaly identification and real-time early warning method of the present invention is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0066] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for identifying and providing real-time early warning of anomalies in highway facilities, characterized in that, include: Control the drone to pass through several target areas of the highway and collect inspection image data streams in real time and upload them to the cloud; In response to the drone moving to the edge of the target area, a panoramic feature acquisition action is performed, including increasing the drone's flight altitude, acquiring a panoramic image of the adjacent target area, and then restoring the reference altitude; Several visual windows are constructed within the panoramic image based on the direction of the highway's extension. Extended interferometry analysis is performed based on several sub-target regions in a panoramic image, including determining the variation characteristics of image parameters of sub-windows within each sub-target region in the direction of highway extension. The sub-windows are determined based on road contour features and are classified into a first type of sub-window and a second type of sub-window. Based on the aforementioned change characteristics, determine the visual interference type of the sub-windows within each sub-target region; The sub-target area where the UAV is located is determined, and the inspection image data stream is analyzed based on the visual interference type corresponding to the sub-view window, including: The mapping relationship between sub-windows and coding blocks is determined, and the stable characteristics of the coding data corresponding to the coding block in the time domain are analyzed to lock the abnormal time domain. The inspection image data stream in the abnormal time domain is decoded to perform anomaly identification and early warning. Decode all inspection image data streams to identify and issue early warnings for anomalies; Among them, the anomaly identification and early warning includes identifying whether there are abnormal targets in the decoded frame, marking the decoded frame containing abnormal targets as a potential abnormal frame, and issuing an early warning message.

2. The method for identifying and providing real-time early warning of highway facility anomalies according to claim 1, characterized in that, The process of constructing a plurality of visual windows within the panoramic image based on the highway's extension direction includes: The centerline of the highway is determined as the baseline in the panoramic image; A plurality of visual windows are constructed with the baseline as the center line extending in the direction of the highway, so that the visual windows cover the area of ​​the highway in which there are no vehicle outlines. Each of the aforementioned visual windows is a rectangular area used to select the content of the panoramic image.

3. The method for identifying and providing real-time early warning of highway facility anomalies according to claim 2, characterized in that, The process of determining sub-windows based on road contour features and determining the type of the resulting sub-windows includes: Determine the road contour features within the visual window of the panoramic image, including guardrail contours and road line contours; A strip-shaped area of ​​predetermined width is determined on both sides of the guardrail outline to obtain the first type of sub-window; A strip-shaped area that does not include the road line outline is determined between the guardrail outlines to obtain a second type of sub-window.

4. The method for identifying and providing real-time early warning of highway facility anomalies according to claim 3, characterized in that, The process of determining the variation characteristics of the image parameters of the sub-view window in the direction of highway extension includes: Visual windows are determined one by one along the direction of the highway in the panoramic image; Determine the rate of change of the first image parameter of the first type of sub-window and the rate of change of the second image parameter of the second type of sub-window among several adjacent visual windows contained in the sub-target region; The mean rate of change of the first image parameters and the mean rate of change of the second image parameters are calculated and used as the change features.

5. The method for identifying and providing real-time early warning of highway facility anomalies according to claim 4, characterized in that, The process of determining the visual interference type of sub-windows within each sub-target region based on the aforementioned change characteristics includes, Determine the mean rate of change of the first image parameter and the mean rate of change of the second image parameter corresponding to the sub-target region; If the average rate of change of the first image parameter of the sub-target region is greater than the predetermined first rate of change threshold, the first type of sub-window of the sub-target region is determined to be a strong visual interference type. If the average rate of change of the second image parameter of the sub-target region is greater than the predetermined second rate of change threshold, the second type of sub-window of the sub-target region is determined to be a strong visual interference type.

6. The method for identifying and providing real-time early warning of highway facility anomalies according to claim 1, characterized in that, Analysis of the inspection image data stream based on the visual interference type corresponding to the sub-window includes, If the interference stability condition is met, the mapping relationship between the sub-window and the coding block is determined, and the stability characteristics of the coding data corresponding to the coding block in the time domain dimension are analyzed to lock the abnormal time domain. The inspection image data stream in the abnormal time domain is decoded to perform anomaly identification and early warning. If the interference stability condition is not met, decode all inspection image data streams and perform anomaly identification and early warning. The interference stability condition is that there are no sub-windows with strong visual interference.

7. The method for identifying and providing real-time early warning of highway facility anomalies according to claim 1, characterized in that, The process of determining the mapping relationship between sub-windows and coded blocks includes, The inspection images collected by the UAV when it flies at a reference altitude are predetermined, and sub-windows are labeled on the inspection images, including a first type of sub-window and a second type of sub-window; Determine the position of the sub-window in the inspection image, determine the corresponding coding block, and construct the mapping relationship between the sub-window and the coding block.

8. The method for identifying and providing real-time early warning of highway facility anomalies according to claim 7, characterized in that, The process of parsing the stable characteristics of the encoded data corresponding to the encoded block in the time domain includes, Determine the coded block mapped by the sub-window and extract the coded parameters; The temporal variation curve of the coding parameters is constructed based on the temporal sequence of the inspection image data stream; Extract the stability characteristics of several curve segments corresponding to the time-domain variation curve, including amplitude variance and average amplitude.

9. The method for identifying and providing real-time early warning of highway facility anomalies according to claim 8, characterized in that, The process of locking the anomalous time domain includes, Identify the time domain segment corresponding to the curve segment that meets the anomaly condition, and define the time domain segment as the abnormal time domain; Among them, the abnormal conditions include the amplitude variance being greater than the predetermined variance threshold or / and the average amplitude not belonging to the predetermined stable amplitude range.

10. The method for identifying and providing real-time early warning of highway facility anomalies according to claim 1, characterized in that, The warning information includes the location and time of the potentially abnormal frames collected by the drone.

Images