A method and system for sensing road subsidence and bumps

Abstract

The present invention relates to the fields of road engineering, highway engineering, and machine vision technology, and in particular to a method and system for sensing road subsidence and bumps. The method comprises: obtaining video captured by a driving recorder of an operating vehicle, extracting frames from the video to obtain an image sequence; performing single-frame recognition on the image sequence based on a pre-trained neural network model, extracting driving information of the operating vehicle, as well as vanishing point coordinates, target objects, and scene categories from each frame, and summarizing and sorting the images to obtain time series data; slicing the time series data, analyzing the positional changes of the vanishing points during the driving of the operating vehicle based on the sliced ​​data, and determining abnormal bumps based on the positional changes of the vanishing points; grading the abnormal bumps, and establishing associations between the relevant identification data and the abnormal bumps based on the time series data, and storing the data in a database. The present invention utilizes image analysis technology to analyze the bumps of patrol vehicles, thereby sensing road subsidence and abnormal bumps.

Description

Technical Field

[0001] The present invention relates to the fields of road engineering, highway engineering and machine vision technology, and in particular to a method and system for sensing road subsidence and bumps. Background Art

[0002] Road subsidence and bumps are common problems on urban roads, seriously affecting driving safety and comfort. Timely perception and detection of road subsidence and bumps, and early maintenance and treatment, are important tasks in current road management and maintenance. Traditional perception of road subsidence and bumps mainly relies on manual measurement methods and intelligent patrol equipment measurement methods. Although they can ensure accuracy, the degree of automation is not high and it consumes manpower and material resources. With the development of information technology, operating vehicles are generally equipped with driving recorders. These devices can collect video data in real time during driving and record information such as time and geographic location. Compared with arranging special patrol vehicles and personnel to perform operations, operating vehicles have a wide coverage, high monitoring frequency, and low cost. However, if you want to use operating vehicles to collect road bumps, subsidence and other problems, there will be the following problems:

[0003] (1) Limitations of dashcam functionality: Currently, dashcams installed on most operating vehicles generally only collect video data, not acceleration data. Furthermore, positioning, speed, and time information are typically embedded in the video as text, rather than stored as independent data. Therefore, algorithms for processing road bump data, such as laser and acceleration data, are difficult to use in this context.

[0004] (2) Challenges of hardware cost and data fusion: If new sensors (such as lasers, acceleration sensors, etc.) are installed specifically for operating vehicles, on the one hand, hardware costs will increase, and on the other hand, it will also involve the fusion, storage, and analysis of existing video data from different manufacturers.

[0005] (3) Challenges in data processing and analysis: If only acceleration sensors are installed on operating vehicles and the acceleration data is used to analyze the subsidence and abnormal bumps that the vehicle experiences while driving, in-depth analysis of the cause of the problem may be lacking, leading to misjudgment. For example, relying solely on acceleration signals may not be able to distinguish whether the bumps are caused by road problems or by the vehicle's own vibrations.

[0006] Adding video analysis to acceleration analysis requires synchronizing the acceleration data, video, latitude and longitude data, adjusting the sampling interval, and performing data slicing adjustments. For example, if the acceleration data exhibits unusual turbulence, the captured video will already be an image of the abnormal turbulence. The time difference between the two depends on the installation position of the image sensor and accelerometer, significantly increasing the complexity of the processing program.

[0007] (4) Limitations of existing video target detection and analysis: Currently, most analyses of road subsidence and bumps mainly use image detectors to detect targets in the video that may cause vehicle bumps (such as manhole covers, speed bumps, potholes, etc.). However, during the detector recognition process, there may be missed detections due to occlusion (such as occlusion by vehicles or pedestrians) or insufficient detector performance, which in turn affects the accuracy of the cause analysis.

[0008] The above analysis shows that while using commercial vehicles to collect road surface damage data has broad application prospects, it still faces many technical challenges and limitations in practical applications. Therefore, the present invention proposes a method and system for sensing road subsidence and bumps. Summary of the Invention

[0009] The purpose of the present invention is to provide a method and system for perceiving road subsidence and bumps. Without adding additional hardware sensors, the method analyzes the videos collected by the driving recorders of currently operating vehicles, that is, it uses AI image analysis technology to realize the bump analysis of patrol vehicles, and then realizes the perception of road subsidence and abnormal bump points, and classifies and structures the relevant diseases according to the bump ratio.

[0010] To achieve the above objectives, the present invention provides a method for sensing road subsidence and bumps, comprising:

[0011] Obtaining a video captured by a driving recorder of an operating vehicle, extracting frames from the video, and obtaining an image sequence;

[0012] Performing single-frame recognition on the image sequence based on a pre-trained neural network model, extracting driving information of the operating vehicle, as well as vanishing point coordinates, target objects, and scene categories in each frame, and summarizing and sorting the data to obtain time series data;

[0013] Slicing the time series data, analyzing the position change of the vanishing point during the operation of the operating vehicle based on the sliced ​​data, and determining abnormal bumps according to the position change of the vanishing point;

[0014] The abnormal turbulence is graded, and related identification data is associated with the abnormal turbulence based on the time series data and stored in a database.

[0015] Optionally, the video captured by the driving recorder of the operating vehicle is obtained if:

[0016] The video clarity is within the preset range, the road area ratio in the video meets the preset ratio, the installation method, installation position and angle of the dash cam meet the preset requirements, and the video contains clear and fixed time information, speed information, and latitude and longitude information.

[0017] Optionally, single-frame recognition is performed on the image sequence based on a pre-trained neural network model to extract driving information of the operating vehicle in each frame, as well as vanishing point coordinates, target objects, and scene categories, including:

[0018] Performing single-frame recognition on the image sequence based on a pre-trained first neural network model to extract vanishing point coordinates, target objects, and scene categories;

[0019] Based on the pre-trained second neural network model, single-frame recognition is performed on the image sequence to extract time information, speed information, and longitude and latitude information.

[0020] Optionally, the first neural network model includes: an identification backbone network and a first branch head network, a second branch head network, and a third branch head network. The identification backbone network is used to perform multi-scale feature extraction on the input image and output several feature maps; the first branch head network is used to identify the vanishing point height in the feature map, the second branch head network is used to identify the target object and target type that will cause bumps in the feature map, and the third branch head network is used to identify the scene type of image acquisition in the feature map.

[0021] Optionally, the second neural network model adopts a neural network for performing text OCR recognition.

[0022] Optionally, slicing the time series data, analyzing the position change of the vanishing point during the operation of the operating vehicle based on the sliced ​​data, and determining abnormal bumpiness according to the position change of the vanishing point includes:

[0023] Slicing the time series data at a preset first time interval to obtain a plurality of first slice data;

[0024] Extracting vanishing point height time series data and speed time series data from the first slice data, and calculating an average speed;

[0025] Fitting the vanishing point height time series data using the least square method to obtain a fitting straight line and calculate the vanishing point deviation, obtaining the vanishing point deviation time series data and determining the average bump condition by calculating the root mean square;

[0026] Slicing the vanishing point deviation time series data at a preset second time interval to obtain a plurality of second slice data, and selecting the second slice data with the largest root mean square value as the short-time maximum turbulence;

[0027] An average speed correction is introduced, a bump ratio is calculated based on the average bump condition and the short-term maximum bump, and the bump ratio is compared with a preset bump threshold to determine the abnormal bump.

[0028] Optionally, an average speed correction is introduced, and the turbulence ratio is calculated based on the average turbulence condition and the short-term maximum turbulence, including:

[0029] ;

[0030] Among them, R1 is the bump ratio, v is the average speed of the first slice data, alpha is a preset value set by the vehicle type, rms1_sub_max is the short-term maximum bump, and rms1 is the average bump condition.

[0031] On the other hand, the present invention also provides a road subsidence and bump sensing system, comprising:

[0032] The video acquisition module is used to obtain the video captured by the driving recorder of the operating vehicle, extract frames from the video, and obtain an image sequence;

[0033] A video analysis module is used to perform single-frame recognition on the image sequence based on a pre-trained neural network model, extract the driving information of the operating vehicle, as well as the vanishing point coordinates, target objects, and scene categories in each frame, and summarize and sort them to obtain time series data;

[0034] a bump analysis module, configured to slice the time series data, analyze the position change of the vanishing point during the operation of the operating vehicle based on the sliced ​​data, and determine abnormal bumps according to the position change of the vanishing point;

[0035] The hierarchical storage module is used to classify the abnormal turbulence, and to associate the relevant identification data with the abnormal turbulence based on the time series data, and store the data in a database.

[0036] The beneficial effects of the present invention are:

[0037] The present invention utilizes existing operating vehicle driving recorders and does not require the installation of additional sensors, greatly reducing the complexity and cost of data collection. Through video decoding, frame extraction and analysis of neural network models, it can efficiently identify and analyze road subsidence and bumps, timely discover road defects, and improve the real-time and accuracy of data collection. From video decoding, frame extraction, image recognition to data analysis and storage, the present invention is highly automated throughout the entire process, reducing manual intervention and improving work efficiency; through the multi-branch head design of the neural network model, it can simultaneously perform vanishing point height, target detection and scene classification, improving recognition accuracy and efficiency; and the recognition results are structured and stored to facilitate subsequent data query and analysis. Through the use of a vector database, specific types of bump and subsidence data can be quickly retrieved, improving data availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 This is a flow chart of a method for sensing road subsidence and bumps according to an embodiment of the present invention;

[0040] Figure 2 This is a flow chart of video frame analysis according to an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of vanishing point height reasoning according to an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of determining whether a recognized target intersects with a driving area according to an embodiment of the present invention;

[0043] Figure 5 This is a flowchart of time series data analysis according to an embodiment of the present invention;

[0044] Figure 6 Schematic diagram of the relationship between road bumps and image vanishing point transformation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Example 1:

[0048] This embodiment provides a method for sensing road subsidence and bumps, including:

[0049] Obtaining a video captured by a driving recorder of an operating vehicle, extracting frames from the video, and obtaining an image sequence;

[0050] Performing single-frame recognition on the image sequence based on a pre-trained neural network model, extracting driving information of the operating vehicle, as well as vanishing point coordinates, target objects, and scene categories in each frame, and summarizing and sorting the data to obtain time series data;

[0051] Slicing the time series data, analyzing the position change of the vanishing point during the operation of the operating vehicle based on the sliced ​​data, and determining abnormal bumps according to the position change of the vanishing point;

[0052] The abnormal turbulence is graded, and related identification data is associated with the abnormal turbulence based on the time series data and stored in a database.

[0053] Specifically, this embodiment utilizes existing operating vehicle driving recorders, eliminating the need for additional sensors, greatly reducing the complexity and cost of data collection. Through video decoding, frame extraction, and analysis of neural network models, it is possible to efficiently identify and analyze road subsidence and bumps, promptly detect road defects, and improve the real-time and accuracy of data collection. This embodiment is highly automated throughout the entire process, from video decoding, frame extraction, image recognition to data analysis and storage, reducing manual intervention and improving work efficiency. Through the multi-branch head design of the neural network model, it is able to simultaneously perform vanishing point height, target detection, and scene classification, improving recognition accuracy and efficiency. The recognition results are structured and stored for easy subsequent data query and analysis. Through the use of a vector database, specific types of bump and subsidence data can be quickly retrieved, improving data availability.

[0054] Furthermore, the video captured by the driving recorder of the operating vehicle is obtained if:

[0055] The video clarity is within the preset range, the road area ratio in the video meets the preset ratio, the installation method, installation position and angle of the dash cam meet the preset requirements, and the video contains clear and fixed time information, speed information, and latitude and longitude information.

[0056] Furthermore, single-frame recognition is performed on the image sequence based on a pre-trained neural network model to extract the driving information of the operating vehicle in each frame, as well as the vanishing point coordinates, target objects, and scene categories, including:

[0057] Performing single-frame recognition on the image sequence based on a pre-trained first neural network model to extract vanishing point coordinates, target objects, and scene categories;

[0058] Single-frame recognition is performed on the image sequence based on a pre-trained second neural network model to extract time information, speed information, and longitude and latitude information.

[0059] Furthermore, the first neural network model includes: an identification backbone network and a first branch head network, a second branch head network, and a third branch head network. The identification backbone network is used to perform multi-scale feature extraction on the input image and output a plurality of feature maps; the first branch head network is used to identify the vanishing point height in the feature map, the second branch head network is used to identify the target object and target type that will cause bumps in the feature map, and the third branch head network is used to identify the scene type of image acquisition in the feature map.

[0060] Furthermore, the second neural network model adopts a neural network for performing text OCR recognition.

[0061] Furthermore, slicing the time series data, analyzing the position change of the vanishing point during the operation of the operating vehicle based on the sliced ​​data, and determining abnormal bumping according to the position change of the vanishing point, including:

[0062] Slicing the time series data at a preset first time interval to obtain a plurality of first slice data;

[0063] Extracting vanishing point height time series data and speed time series data from the first slice data, and calculating an average speed;

[0064] Fitting the vanishing point height time series data using the least square method to obtain a fitting straight line and calculate the vanishing point deviation, obtaining the vanishing point deviation time series data and determining the average bump condition by calculating the root mean square;

[0065] Slicing the vanishing point deviation time series data at a preset second time interval to obtain a plurality of second slice data, and selecting the second slice data with the largest root mean square value as the short-time maximum turbulence;

[0066] An average speed correction is introduced, a bump ratio is calculated based on the average bump condition and the short-term maximum bump, and the bump ratio is compared with a preset bump threshold to determine the abnormal bump.

[0067] Furthermore, an average speed correction is introduced, and the turbulence ratio is calculated based on the average turbulence condition and the short-term maximum turbulence, including:

[0068] ;

[0069] Among them, R1 is the bump ratio, v is the average speed of the first slice data, alpha is a preset value set by the vehicle type, rms1_sub_max is the short-term maximum bump, and rms1 is the average bump condition.

[0070] The following combination Figures 1-6 The method provided in this embodiment is described in detail. Figure 1As shown, the video is first decoded and framed, and the neural network model is used to parse and identify the extracted images. The recognition results are summarized and sorted according to the frame numbers of the image frames. The recognition results are summarized to obtain time series data Q; the time series data Q is then sliced ​​and parsed, and the position changes of the vanishing points during the vehicle's driving process are analyzed from the sliced ​​data to determine whether the vehicle has abnormal bumps, and the abnormal bumps are classified. Based on the time series data Q, the relevant recognition data is associated with the abnormal bumps to complete the data storage.

[0071] 1. Video frame analysis, such as Figure 2 As shown, specifically including:

[0072] (1) Decode and extract frames of the video captured by the driving recorder of the operating vehicle to obtain the image sequence image_list. The data format of image_list is [[f_1, image_1], [f_2, image_2], [f_3,image_3], …[f_n, image_n], …], where image_1, image_2, …image_n, … are single-frame images in the video, and f_1, f_2, …f_n, … are the frame numbers corresponding to the frame images.

[0073] The requirements for dashcam video are as follows:

[0074] 1) Resolution ≥ 1 million pixels, video frame rate ≥ 24fps, clear video, the road surface area captured accounts for ≥ 20% of the image, there is no obstruction, and no frame drops or screen distortion;

[0075] 2) The dashcam used must be securely mounted and not obstructed by other objects inside the vehicle while driving. The camera's viewing angle must be in the direction of the vehicle's travel, and the image must exhibit no noticeable horizontal roll or deflection. The vanishing point in the image must be within the visible range.

[0076] 3) The video contains text such as time information, speed information, longitude and latitude information, and the text is clear and fixed in position.

[0077] (2) Use neural network model 1 to perform recognition processing on a single frame image. Take the nth frame in the image sequence as an example, the frame number is f_n, and the frame image is image_n. The processing steps are as follows:

[0078] 1) Preprocess the image image_n and input it into the neural network model 1 for inference to calculate the corresponding inference result.

[0079] The neural network model 1 is designed with reference to current mainstream object detection networks (such as Yolov11). It includes a recognition backbone network and three branching head networks: head_1, head_2, and head_3. The multiple branching heads share the inference feature maps of the backbone network, thereby reducing computational complexity and improving recognition efficiency.

[0080] A) The backbone network uses a deep convolutional neural network architecture, combined with residual connections and multi-scale feature extraction techniques to achieve efficient and powerful feature extraction capabilities. The backbone network can use pre-trained model weights or be trained with head_2 detection data.

[0081] B) head_1 is a classification head network that takes one of the feature maps output by the backbone network as input. After convolution, pooling, self-attention, and other operations, it is mapped to m categories through softmax. Here, m refers to the m equal divisions along the height H of the video image, m ≥ 512. The category with the highest probability calculated by softmax is used as the interval where the vanishing point is located. That is, the height of the midpoint of the j-th equal division interval from the top to the bottom is used as the height of the vanishing point on the image, recorded as vp_h_n, as shown in the following example: Figure 3 As shown in the figure, head_1 is trained using a supervised learning method, and the backbone network is frozen during training.

[0082] C) head_2 is a detection head network. It takes several feature maps output by the backbone network as input. After convolution, pooling, and other operations, it outputs the recognition detection results [[x1, y1, x2, y2, cls, conf], …]. It then performs an intersection-and-comparison analysis on the recognition result box with the patrol vehicle's driving area in the image. The recognition results outside the patrol vehicle's driving area are discarded, and the recognition results within the driving area are retained, denoted as detects_n. Among them, cls is the type of recognition target, including speed bumps, road markings, manhole covers, road patches, potholes, road foreign objects, expansion joints, and road subsidence. x1, y1, x2, and y2 are the coordinates of the recognition target on the image, and conf is the confidence level of the recognition target. The driving area of ​​the patrol vehicle in the image refers to the range that the wheels of the patrol vehicle can roll over. After the camera is installed and fixed, this area generally does not change. Therefore, according to the actual image perspective, a trapezoidal area T can be preset in the image as the driving area of ​​the patrol vehicle. The IOU intersection-over-union ratio of each recognition target and the trapezoidal area T is calculated, and the recognition targets with a relatively small IOU are filtered out, that is, the recognition targets outside the driving area of ​​the patrol vehicle are eliminated, such as Figure 4As shown in the figure, head_2 is trained using a supervised learning method, and the backbone network can be kept unchanged during training.

[0083] D) head_3 is a classification head network. It takes one of the feature maps output by the backbone network as input. After convolution and pooling, it is mapped to k categories using softmax. The captured image is classified into k scene types, including expressway, ground, ramp, tunnel, underpass, and bridge. The category with the highest probability calculated by softmax is the scene of the image, denoted as scene_n. Head_3 is trained using supervised learning, and the backbone network is frozen during training.

[0084] (3) Neural network model 2 uses a neural network specifically designed for text OCR recognition. Through manual pre-marking or text recognition, the position of text information such as time, longitude and latitude, and speed in image image_n is pre-marked, and the corresponding time information sub-image image_sub1_n, longitude and latitude information sub-image image_sub2_n, and speed information sub-image image_sub3_n are cut out and input into neural network model 2 respectively, thereby recognizing the corresponding time text information time_n, longitude and latitude text information lnglat_n, and speed text information v_n.

[0085] (4) The recognition results of a single frame, such as the vanishing point height, detected targets within the driving area, image scene classification, and time, are summarized and recorded as [f_n, vp_h_n, detects_n, scene_n, time_n, lnglat_n, v_n]. The analysis results of all frames of a video are summarized into a queue, which is recorded as the time series queue Q. The programs and models involved in video frame extraction and image analysis can be deployed on the edge or in the cloud.

[0086] 2. Analyze the time series data Q and analyze vehicle bumps;

[0087] The time series data Q records the changes of the image vanishing point over time, and the changes in the vanishing point height can reflect the bumpy condition of the road to a certain extent, such as Figure 6 Based on the analyzed time series data Q, the road bump condition is further analyzed. The specific steps are as follows: Figure 5 As shown, specifically including:

[0088] (1) Slice the time series data Q at a fixed interval TIME to obtain the sliced ​​time series data Q1. Separate the time series data of the vanishing point height from it and perform median filtering, denoted as h1 (including the frame number and vanishing point height). Separate the speed time series data extracted by OCR and calculate the average speed v of the slice. The interval TIME is generally a preset fixed time, ranging from 3 to 5 seconds.

[0089] (2) Using the frame number of the time series data h1 as the independent variable and the height of the vanishing point in the time series data h1 as the dependent variable, perform linear fitting using the least squares method to obtain the fitted straight line L1. Then, calculate the deviation of each point on the time series data h1 to L1 and summarize them to obtain the vanishing point deviation time series data h1'. The deviation refers to the difference between the actual height value of each point and the height value of the fitted straight line L1. For example, the frame number f_i corresponds to the vanishing point height h1_i. Substitute x=f_1 into the straight line L1 to calculate L1_h_i. The calculated deviation is h1'_i = h1_i - L1_h_i.

[0090] (3) Calculate the mean h1'_mean of the vanishing point deviation time series data h1', and then use the calculation formula , calculate the root mean square of the vanishing point deviation time series data h1' as the average bumping condition of the inspection car during the slice TIME. i is the i-th frame of the vanishing point deviation time series data h1 ′, and N is the number of valid frames of the vanishing point deviation time series data h1 ′.

[0091] (4) According to the fixed time TIME', the vanishing point deviation time series data h1' is further sliced ​​to obtain s sub-slices of the same length, and the vanishing point deviation mean h1'_sub_mean of each sub-slice is calculated respectively, and then according to the calculation formula Calculate the RMS value for each slice and take the maximum value as the maximum short-term bump of the patrol vehicle within the slice TIME. Record this value as rms1_sub_max. Also record the frame number corresponding to this maximum short-term bump, f_sub_max. N' is the number of valid frames in the sub-slice data, and TIME' is a preset value, typically a short time, typically ranging from 0.2 to 1.0 seconds.

[0092] (5) According to the calculation formula Calculate the ratio of the maximum short-term jolt to the average jolt within the slice data, after speed correction. This ratio represents the jolt experienced by the patrol vehicle during the time period of the slice TIME. Where v is the average speed of slice Q1, and alpha is a preset value, typically set by vehicle type and obtained through experimental statistics.

[0093] (6) Determine whether R1 exceeds a certain threshold THREHOLD. If R1 is smaller than THREHOLD, it indicates that the short-term bump intensity of the slice is low and the vehicle has not experienced obvious abnormal bumps. If R1 is larger than THREHOLD, it indicates that the short-term bump intensity of the slice data is high and the vehicle has experienced obvious bumps.

[0094] (7) Rating the abnormal bumpiness of the vehicle is performed based on the size of R1. According to the frame number (f_sub_max-beta1) to the frame number (f_sub_max-beta2), the corresponding identification information, speed information, time information, and longitude and latitude information are found from the time series data Q, and the relevant data are sorted and stored in the vector database.

[0095] Beta1 and beta2 refer to the time difference between the time when the image captures the road target and the time when the road target causes the vehicle to bump. They can be fixed values ​​or calculated by multiplying the average speed v by a fixed time.

[0096] Considering that some road surface defects may be difficult to identify or missed due to vehicle obstruction or insufficient detector performance, this data is considered to be classified as "severe bumps with unknown causes." When organizing the identification information, the cause of the bumps is recorded as res, with the type as a string, and the cause is classified according to the following conditions:

[0097] 1) If there is only one identification target A in the corresponding frame number range, then res is recorded as "A";

[0098] 2) If there are multiple types of identification targets A, B, C, and D within the corresponding frame number range, then res is recorded as "A\B\C\D";

[0099] 3) If no target object is identified within the corresponding frame number range and R1 exceeds a specific threshold THREHOLD2, res is recorded as "unexplained severe turbulence".

[0100] After the recognition information is organized, a data record is generated: [f_sub_max, res, lnglat, time, v, scene, R1, rms1, rms1_sub_max, vec, image], and stored in the vector database. Vec is a high-dimensional vector obtained by processing the string res using neural network model 3. In this embodiment, neural network model 3 uses NetEase's BECmbedding model.

[0101] (8) When searching the vector database, you can query the bump and sink data within a specified time, space, or scene range based on time, lnglat, and scene. You can also search for specific types of bump and sink data based on vector similarity distance. For example, if you search for road bumps and sinks caused by two diseases A and B in the query system, you can use neural network model 3 to process the string "A\B" to obtain vector vec1, and then perform similarity search or hybrid search in the vector database to obtain the corresponding results.

[0102] (9) The slicing, parsing and storage of time series data can be deployed at the edge or in the cloud.

[0103] Example 2:

[0104] This embodiment further provides a road subsidence and bump sensing system, including:

[0105] The video acquisition module is used to obtain the video captured by the driving recorder of the operating vehicle, extract frames from the video, and obtain an image sequence;

[0106] A video analysis module is used to perform single-frame recognition on the image sequence based on a pre-trained neural network model, extract the driving information of the operating vehicle, as well as the vanishing point coordinates, target objects, and scene categories in each frame, and summarize and sort them to obtain time series data;

[0107] a bump analysis module, configured to slice the time series data, analyze the position change of the vanishing point during the operation of the operating vehicle based on the sliced ​​data, and determine abnormal bumps according to the position change of the vanishing point;

[0108] The hierarchical storage module is used to classify the abnormal turbulence, and to associate the relevant identification data with the abnormal turbulence based on the time series data, and store the data in a database.

[0109] The processing flow of the system proposed in this embodiment is as follows: after exporting the video collected by the bus, it is uploaded to the cloud, and the cloud system performs frame analysis and time series data analysis on the collected video, and completes the data storage.

[0110] (1) Video collection: The requirements are as follows:

[0111] 1) Collect video data from the driving recorder.

[0112] 2) Resolution ≥ 1 million pixels, video frame rate ≥ 24fps.

[0113] 3) The photographed road area accounts for ≥ 20% of the image.

[0114] 4) The video is unobstructed and there will be no frame drops or screen distortion.

[0115] 5) The dashcam is mounted securely, with the camera facing the vehicle's direction of travel. The image should not show any noticeable horizontal roll or deflection, and the vanishing point should be within the visible range.

[0116] 6) The video contains text such as time information, speed information, longitude and latitude information, and the text is clear and fixed in position.

[0117] (2) Video frame analysis:

[0118] (1) Decode the captured video to obtain the image sequence image_list in the format of [[f_1, image_1], [f_2, image_2], [f_3, image_3], …[f_n, image_n], …], where image_1, image_2, …image_n, … are single-frame images in the video, and f_1, f_2, …f_n, … are the frame numbers corresponding to the frame images.

[0119] (2) Frame image analysis:

[0120] 1) Use a neural network model to recognize and process a single frame image. The neural network model consists of a backbone network and multiple branch head networks head_1, head_2, and head_3:

[0121] Backbone network: It adopts a deep convolutional neural network structure, combined with residual connections, multi-scale feature extraction and other technologies to achieve efficient and powerful feature extraction capabilities.

[0122] head_1: classification head network, used to identify the vanishing point height.

[0123] head_2: detection head network, used to detect targets that may cause vehicle bumps.

[0124] head_3: classification head network, used to identify image scene categories.

[0125] 2) Preset the text information area in the image and crop the sub-image according to the pre-defined coordinates. Use the neural network model to recognize the sub-image and extract information such as time, longitude and latitude, and speed.

[0126] 3) Summarize the recognition results of the single-frame image image_n by neural network model 1 and neural network model 2, record them as [f_n, vp_h_n, detects_n, scene_n, time_n, lnglat_n, v_n], and summarize them into the time series queue Q, where:

[0127] f_n is the frame number corresponding to image image_n;

[0128] vp_h_n is the height of the vanishing point of the image, which is calculated by the neural network model 1 through the head_1 branch network;

[0129] detects_n is the identified objects in image_n that may cause road bumps, including speed bumps, road markings, manhole covers, road repairs, potholes, foreign objects on the road, expansion joints, road subsidence, and other objects that may cause vehicle bumps. These objects are calculated by neural network model 1 through the head_2 branch network.

[0130] scene_n is the scene category of image_n identified by neural network model 1, including expressway, ground, ramp, tunnel, underpass, bridge, etc., which is calculated by neural network model 1 through the head_3 branch network;

[0131] time_n is the time information of image_n, which is recognized by neural network model 2;

[0132] lnglat_n is the longitude and latitude information of image_n, which is recognized by the neural network model 2;

[0133] v_n is the velocity information of image_n, which is recognized by neural network model 2.

[0134] Neural network model 2: used for text OCR recognition and extraction of time, longitude, latitude, speed and other information.

[0135] Neural network model 3: used to convert recognition results into high-dimensional vectors to facilitate vector database storage and query.

[0136] (III) Q-slice analysis of time series data:

[0137] (1) Time series data slicing:

[0138] 1) Slice the time series data Q according to the fixed interval time TIME to obtain the sliced ​​time series data Q1, with the interval time TIME = 5s.

[0139] 2) Separate the time series data of the vanishing point height from Q1 and perform median filtering to obtain h1.

[0140] 3) Separate the velocity time series data and calculate the average velocity v of the slice.

[0141] (2) Linear fitting and calculation of deviation and root mean square:

[0142] 1) Using the frame number of the time series data h1 as the independent variable and the height of the vanishing point as the dependent variable, the least squares method is used to perform straight line fitting to obtain the fitted straight line L1.

[0143] 2) Calculate the deviation of each point on the time series data h1 to L1 and obtain the vanishing point deviation time series data h1'.

[0144] 3) Calculate the mean h1'_mean of the vanishing point deviation time series data h1', according to the calculation formula , calculate the root mean square of the vanishing point deviation time series data h1' as the average bumpy condition of the inspection vehicle during the slice TIME time.

[0145] 4) According to the fixed time TIME'=0.2s, the vanishing point deviation time series data h1' is further sliced ​​to obtain 5÷0.2=25 sub-slices of the same length. The vanishing point deviation mean h1'_sub_mean of each sub-slice is calculated respectively, and then according to the calculation formula The root mean square of each slice data is calculated, and the maximum value is taken as the short-term maximum bump of the patrol vehicle during the slice TIME time, recorded as rms1_sub_max, and the frame number corresponding to the maximum short-term bump is recorded as f_sub_max.

[0146] 5) Take out the maximum value rms1_sub_max and record the frame number f_sub_max corresponding to the maximum short-term turbulence.

[0147] (3) Assessment of turbulence conditions:

[0148] 1) Calculation , evaluate the bumpy condition of the patrol car during the slice TIME time.

[0149] 2) Determine whether R1 exceeds the threshold THREHOLD and evaluate whether the vehicle has experienced abnormal bumps: If R1 is greater than the threshold THREHOLD, the vehicle's abnormal bumps are rated based on the size of R1 and subsequent calculations are performed; if R1 is less than THREHOLD, skip the slice data and take the next slice data from the time series data Q for calculation.

[0150] 3) According to the frame number (f_sub_max-beta1) to the frame number (f_sub_max-beta2), find the corresponding identification information, speed information, time information, longitude and latitude information from the time series data Q, organize the relevant data, and store it in the vector database.

[0151] Among them, beta1 and beta2 refer to the time difference between the image capturing the road target and the road target causing the vehicle to bump, which are calculated by multiplying the average speed v by the fixed time t'.

[0152] Considering that some road surface defects may be difficult to identify or missed due to vehicle obstruction or insufficient detector performance, this data is considered to be classified as "severe bumps with unknown causes." When organizing the identification information, the cause of the bumps is recorded as res, with the type as a string, and the cause is classified according to the following conditions:

[0153] If there is only one identification target A in the corresponding frame number range, then res is recorded as "A";

[0154] If there are multiple identification targets A, B, C, and D within the corresponding frame number range, the record res is "A\B\C\D";

[0155] If no target object is identified within the corresponding frame number range and R1 exceeds a specific threshold THREHOLD2, res is recorded as "severe turbulence with unknown cause".

[0156] After the recognition information is organized, a data record is formed: [f_sub_max, res, lnglat, time, v, scene, R1, rms1, rms1_sub_max, vec, image], and is stored in the vector database MILVUS. Vec is a high-dimensional vector obtained by processing the string res using neural network model 3. In this embodiment, neural network model 3 uses NetEase's BECmbedding model.

[0157] (4) Subsequent data query usage:

[0158] Subsequent searches within the vector database can be performed based on time, lnglat, and scene, targeting bump and subsidence data within a specified time, space, or scene range. Vector similarity distance can also be used to search for specific types of bump and subsidence data. For example, if the query system searches for road bumps and subsidence caused by two defects, A and B, neural network model 3 can be used to process the string "A\B" to generate vector vec1. Similarity searches or hybrid searches can then be performed within the vector database to obtain the corresponding results.

[0159] This embodiment utilizes existing driving recorders, eliminating the need for additional sensors and significantly reducing hardware costs. Compared to traditional intelligent patrol equipment, it reduces equipment procurement and maintenance costs. By leveraging the wide coverage and high-frequency travel of operating vehicles, comprehensive monitoring of urban roads can be achieved, reducing the investment in specialized patrol vehicles and personnel. Structured storage and rapid retrieval of data enable maintenance personnel to quickly locate problematic road sections, improving the efficiency and effectiveness of maintenance work. By promptly discovering and recording road surface defects, maintenance and repairs can be carried out in advance to prevent further deterioration of the defects and reduce maintenance costs. It can also reduce traffic accidents caused by road surface problems, thereby reducing the economic losses and social costs caused by traffic accidents.

[0160] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A method for sensing road subsidence and bumps, characterized in that: include: Obtaining a video captured by a driving recorder of an operating vehicle, extracting frames from the video, and obtaining an image sequence; Performing single-frame recognition on the image sequence based on a pre-trained neural network model, extracting driving information of the operating vehicle, as well as vanishing point coordinates, target objects, and scene categories in each frame, and summarizing and sorting the data to obtain time series data; Slicing the time series data, analyzing the position change of the vanishing point during the operation of the operating vehicle based on the sliced ​​data, and determining abnormal bumps according to the position change of the vanishing point; Classifying the abnormal turbulence, and establishing associations between relevant identification data and the abnormal turbulence based on the time series data, and storing the associations in a database; The method of slicing the time series data, analyzing the position change of the vanishing point during the operation of the operating vehicle based on the sliced ​​data, and determining abnormal bumpiness according to the position change of the vanishing point includes: Slicing the time series data at a preset first time interval to obtain a plurality of first slice data; Extracting vanishing point height time series data and speed time series data from the first slice data, and calculating an average speed; Fitting the vanishing point height time series data using the least square method to obtain a fitting straight line and calculate the vanishing point deviation, obtaining the vanishing point deviation time series data and determining the average bump condition by calculating the root mean square; Slicing the vanishing point deviation time series data at a preset second time interval to obtain a plurality of second slice data, and selecting the second slice data with the largest root mean square as the short-time maximum turbulence; An average speed correction is introduced, a bump ratio is calculated based on the average bump condition and the short-term maximum bump, and the bump ratio is compared with a preset bump threshold to determine the abnormal bump.

2. The method for sensing road subsidence and bumps according to claim 1, characterized in that: The conditions for obtaining the video captured by the operating vehicle's driving recorder are: The video clarity is within the preset range, the road area ratio in the video meets the preset ratio, the installation method, installation position and angle of the dash cam meet the preset requirements, and the video contains clear and fixed time information, speed information, and latitude and longitude information.

3. The method for sensing road subsidence and bumps according to claim 1, characterized in that: Perform single-frame recognition on the image sequence based on a pre-trained neural network model, extracting the driving information of the operating vehicle in each frame, as well as the vanishing point coordinates, target objects, and scene categories, including: Performing single-frame recognition on the image sequence based on a pre-trained first neural network model to extract vanishing point coordinates, target objects, and scene categories; Single-frame recognition is performed on the image sequence based on a pre-trained second neural network model to extract time information, speed information, and longitude and latitude information.

4. The method for sensing road subsidence and bumps according to claim 3, characterized in that: The first neural network model includes: an identification backbone network and a first branch head network, a second branch head network, and a third branch head network. The identification backbone network is used to perform multi-scale feature extraction on the input image and output a plurality of feature maps; the first branch head network is used to identify the vanishing point height in the feature map, the second branch head network is used to identify the target object and target object type that may cause bumps in the feature map, and the third branch head network is used to identify the scene type of image acquisition in the feature map.

5. The method for sensing road subsidence and bumps according to claim 3, characterized in that: The second neural network model adopts a neural network for performing text OCR recognition.

6. The method for sensing road subsidence and bumps according to claim 1, characterized in that: An average speed correction is introduced, and the turbulence ratio is calculated based on the average turbulence condition and the short-term maximum turbulence, including: ； Among them, R1 is the bump ratio, v is the average speed of the first slice data, alpha is a preset value set by the vehicle type, rms1_sub_max is the short-term maximum bump, and rms1 is the average bump condition.

7. A road subsidence and bump sensing system, characterized in that: include: The video acquisition module is used to obtain the video captured by the driving recorder of the operating vehicle, extract frames from the video, and obtain an image sequence; A video analysis module is used to perform single-frame recognition on the image sequence based on a pre-trained neural network model, extract the driving information of the operating vehicle, as well as the vanishing point coordinates, target objects, and scene categories in each frame, and summarize and sort them to obtain time series data; a bump analysis module, configured to slice the time series data, analyze the position change of the vanishing point during the operation of the operating vehicle based on the sliced ​​data, and determine abnormal bumps according to the position change of the vanishing point; a hierarchical storage module, configured to classify the abnormal turbulence, and associate relevant identification data with the abnormal turbulence based on the time series data, and store the data in a database; The method of slicing the time series data, analyzing the position change of the vanishing point during the operation of the operating vehicle based on the sliced ​​data, and determining abnormal bumpiness according to the position change of the vanishing point includes: Slicing the time series data at a preset first time interval to obtain a plurality of first slice data; Extracting vanishing point height time series data and speed time series data from the first slice data, and calculating an average speed; Fitting the vanishing point height time series data using the least square method to obtain a fitting straight line and calculate the vanishing point deviation, obtaining the vanishing point deviation time series data and determining the average bump condition by calculating the root mean square; Slicing the vanishing point deviation time series data at a preset second time interval to obtain a plurality of second slice data, and selecting the second slice data with the largest root mean square as the short-time maximum turbulence; An average speed correction is introduced, a bump ratio is calculated based on the average bump condition and the short-term maximum bump, and the bump ratio is compared with a preset bump threshold to determine the abnormal bump.

Images