Stud state recognition method and device, computer device and storage medium

Abstract

The application relates to a stud state recognition method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: extracting a key frame from a road video frame; the key frame is used to represent the change difference of a stud; inputting the key frame into a pre-trained stud detection model to obtain stud type and stud position information of the stud in the key frame; inputting a stud image of the stud in the key frame into a stud image reconstruction model matched with the stud type to obtain a reconstruction image corresponding to the stud image; the stud image is obtained by cutting the key frame based on the stud position information; and obtaining state information of the stud corresponding to the stud image according to a difference value between the stud image and the reconstruction image. The method can improve the recognition efficiency of the stud state.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation technology, and in particular to a method, apparatus, computer equipment, storage medium and computer program product for identifying road stud status. Background Technology

[0002] Road studs, also known as raised pavement markers, are a type of traffic safety facility installed on highways or other types of roads to mark center lines, edge lines, etc., guiding drivers through their reflective properties. However, road studs are easily damaged by vehicles or natural disasters; therefore, timely inspection and replacement of damaged studs are crucial for safe driving.

[0003] Due to the high length of highways and the dense deployment of road spikes, relying on manual inspection of the road spike status requires a huge amount of manpower and resources, resulting in low efficiency in identifying the status of road spikes. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for identifying road spikes that can improve the efficiency of identifying road spike status, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for identifying the status of road studs. The method includes:

[0006] Keyframes are extracted from road video frames; these keyframes are used to represent the differences in road stud changes.

[0007] The keyframes are input into a pre-trained road spike detection model to obtain the road spike type and position information in the keyframes.

[0008] The rail spike image in the keyframe is input into a rail spike image reconstruction model that matches the rail spike type to obtain the reconstructed image corresponding to the rail spike image; the rail spike image is obtained by cropping the keyframe based on the rail spike position information;

[0009] Based on the difference between the spike image and the reconstructed image, the status information of the spike corresponding to the spike image is obtained.

[0010] In one embodiment, keyframes are extracted from road video frames, including:

[0011] Based on the target positioning information of the road video frame, the first key frame is extracted from the road video frame according to a preset positioning interval;

[0012] Optical flow analysis is performed on the road video frame to obtain the second keyframe in the road video frame;

[0013] The first keyframe and the second keyframe are used as the keyframes.

[0014] In one embodiment, before extracting the first keyframe from the road video frame according to the target positioning information of the road video frame at a preset positioning interval, the method further includes:

[0015] Based on the initial positioning information of the road video frame and the timestamp information corresponding to the initial positioning information, the target video frame corresponding to the initial positioning information is determined from the road video frame;

[0016] Based on the initial positioning information, linear interpolation is performed on the intermediate video frames in the road video frames to obtain the interpolated positioning information of the intermediate video frames; the intermediate video frames are the video frames in the road video frames other than the target video frame.

[0017] The initial positioning information of the target video frame and the interpolated positioning information of the intermediate video frames are used as the target positioning information of the road video frame.

[0018] In one embodiment, the pre-trained road spike detection model is trained in the following manner:

[0019] Acquire sample road video frames; the sample road video frames include road studs, and the road studs are labeled with the target road stud type and the target road stud location information;

[0020] The sample road video frames are subjected to image enhancement processing to obtain enhanced video frames;

[0021] The enhanced video frame is input into the road spike detection model to be trained to obtain the predicted road spike type and predicted road spike location information in the enhanced video frame.

[0022] Based on the difference between the target road stud type and the predicted road stud type, and the difference between the target road stud location information and the predicted road stud location information, the road stud detection model to be trained is iteratively trained to obtain the pre-trained road stud detection model.

[0023] In one embodiment, the rail spike image in the keyframe is input into a rail spike image reconstruction model that matches the rail spike type to obtain a reconstructed image corresponding to the rail spike image, including:

[0024] Based on the road spike image reconstruction model that matches the road spike type, the road spike image in the key frame is convolved to obtain the feature vector of the road spike image.

[0025] The feature vectors are deconvolutionally processed according to the road spike image reconstruction model that matches the road spike type to obtain the reconstructed image.

[0026] In one embodiment, after obtaining the state information of the rail spike corresponding to the rail spike image based on the difference between the rail spike image and the reconstructed image, the method further includes:

[0027] Based on the feature information of the reconstructed image, the positioning information of the reconstructed image, and the unit spacing distance of the road spikes, determine the actual road spikes and missing road spikes within a preset distance range;

[0028] Based on the status information, location information, and timestamp information of the actual road studs, as well as the location information and timestamp information of the missing road studs, the detection results of the road video frames are generated.

[0029] Secondly, this application also provides a road stud status identification device. The device includes:

[0030] The keyframe acquisition module is used to extract keyframes from road video frames; the keyframes are used to represent the differences in road studs.

[0031] The road spike information acquisition module is used to input the key frame into a pre-trained road spike detection model to obtain the road spike type and road spike location information in the key frame;

[0032] The road spike image reconstruction module is used to input the road spike image in the key frame into a road spike image reconstruction model that matches the road spike type to obtain the reconstructed image corresponding to the road spike image; the road spike image is obtained by cropping the key frame based on the road spike position information;

[0033] The road spike status acquisition module is used to obtain the status information of the road spike corresponding to the road spike image based on the difference between the road spike image and the reconstructed image.

[0034] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0035] Keyframes are extracted from road video frames; these keyframes are used to represent the differences in road stud changes.

[0036] The keyframes are input into a pre-trained road spike detection model to obtain the road spike type and position information in the keyframes.

[0037] The rail spike image in the keyframe is input into a rail spike image reconstruction model that matches the rail spike type to obtain the reconstructed image corresponding to the rail spike image; the rail spike image is obtained by cropping the keyframe based on the rail spike position information;

[0038] Based on the difference between the spike image and the reconstructed image, the status information of the spike corresponding to the spike image is obtained.

[0039] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0040] Keyframes are extracted from road video frames; these keyframes are used to represent the differences in road stud changes.

[0041] The keyframes are input into a pre-trained road spike detection model to obtain the road spike type and position information in the keyframes.

[0042] The rail spike image in the keyframe is input into a rail spike image reconstruction model that matches the rail spike type to obtain the reconstructed image corresponding to the rail spike image; the rail spike image is obtained by cropping the keyframe based on the rail spike position information;

[0043] Based on the difference between the spike image and the reconstructed image, the status information of the spike corresponding to the spike image is obtained.

[0044] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0045] Keyframes are extracted from road video frames; these keyframes are used to represent the differences in road stud changes.

[0046] The keyframes are input into a pre-trained road spike detection model to obtain the road spike type and position information in the keyframes.

[0047] The rail spike image in the keyframe is input into a rail spike image reconstruction model that matches the rail spike type to obtain the reconstructed image corresponding to the rail spike image; the rail spike image is obtained by cropping the keyframe based on the rail spike position information;

[0048] Based on the difference between the spike image and the reconstructed image, the status information of the spike corresponding to the spike image is obtained.

[0049] The aforementioned road stud state recognition method, apparatus, computer equipment, storage medium, and computer program product extract keyframes from road video frames. These keyframes represent changes in road studs. The keyframes are input into a pre-trained road stud detection model to obtain the road stud type and location information. The road stud image from the keyframe is input into a road stud image reconstruction model matching the road stud type to obtain a reconstructed image. The road stud image is cropped from the keyframe based on the road stud location information. The state information of the road stud corresponding to the road stud image is obtained based on the difference between the road stud image and the reconstructed image. Using this method, the road stud type and location information of road studs in keyframes can be quickly identified using a pre-trained road stud detection model. The road stud image is then cropped, and the state information of the road stud corresponding to the road stud image is obtained from the difference between the reconstructed image obtained from the road stud image reconstruction model and the road stud image. This eliminates the need for manual detection of road stud states, improving the efficiency of road stud state recognition. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating a road stud status recognition method in one embodiment;

[0051] Figure 2 This is a flowchart illustrating the steps for generating detection results of road video frames in one embodiment;

[0052] Figure 3 This is a flowchart illustrating the road stud status recognition method in another embodiment;

[0053] Figure 4 This is a structural block diagram of a road stud status recognition device in one embodiment;

[0054] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] In one embodiment, such as Figure 1As shown, a method for identifying the status of road studs is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be an intelligent vehicle-mounted terminal installed in a vehicle, such as a private car or a special inspection vehicle; the terminal can also be various personal computers, laptops, smartphones, tablets, and IoT devices. In this embodiment, the method includes the following steps:

[0057] Step S101: Extract keyframes from the road video frames; keyframes are used to represent the differences in road stud changes.

[0058] In this context, "road video frame" refers to a video frame within a video feed of road spikes on a road. Roads can include highways, urban roads, highway bridges, and highway tunnels, among others.

[0059] Among them, a keyframe refers to a video frame that can cover the information of road spikes in multiple adjacent frames (or one adjacent frame).

[0060] It should be noted that road video frames carry corresponding timestamp information, and similarly, keyframes also carry corresponding timestamp information.

[0061] Specifically, the terminal acquires the collected road video frames and the synchronously acquired initial positioning information. Based on the inter-frame differential method and a preset interval, it extracts keyframes from the road video frames. The preset interval can be a preset positioning interval, a preset time interval, or a combination of both.

[0062] For example, the terminal can extract a video frame as a key frame every 5 meters based on the initial positioning information corresponding to the road video frame, and extract a video frame as a key frame every 10ms based on the timestamp information of the video frame. If the preset positioning interval or preset time interval is too large, such as extracting a frame every 50 meters or every 5 minutes, it may cause some important road stud information to be missed during the extraction process. Therefore, key frames with large inter-frame changes can be extracted by inter-frame difference.

[0063] Step S102: Input the keyframe into the pre-trained road spike detection model to obtain the road spike type and position information in the keyframe.

[0064] Among them, the pre-trained road spike detection model refers to the model that predicts the road spike type and position information in keyframes.

[0065] Among them, the spike position information refers to the image position information of the spike in the keyframe marked with a rectangular box.

[0066] Specifically, the terminal inputs keyframes into a pre-trained road spike detection model for detection. The pre-trained model performs road spike type prediction and bounding box regression on a preset number of candidate boxes, and removes redundant bounding boxes using non-maximum suppression (NMS) to generate prediction results, obtaining the road spike type and location information. The road spike location information includes: the x-coordinate of the center point of the road spike image, the y-coordinate of the center point of the road spike image, the length of the road spike image, and the width of the road spike image. Then, based on the road spike location information, the keyframe is cropped to obtain the road spike image within the keyframe.

[0067] Step S103: Input the road spike image in the keyframe into the road spike image reconstruction model that matches the road spike type to obtain the reconstructed image corresponding to the road spike image; the road spike image is obtained by cropping the keyframe based on the road spike position information.

[0068] The road spike image reconstruction model refers to a model for reconstructing road spike images. This model can be a variational autoencoder.

[0069] The reconstructed image refers to the image obtained by rebuilding the road spike image.

[0070] Specifically, the terminal inputs the road spike image into a road spike image reconstruction model that matches the road spike type to reconstruct the image, thereby obtaining the reconstructed image corresponding to the input road spike image.

[0071] Furthermore, the road stud image reconstruction model is trained as follows: First sample road stud images and second sample road stud images are acquired; the first sample road stud image is detected and cropped from sample road video frames, and the second sample road stud image is detected and cropped from road stud-carrying video frames crawled from the network; data cleaning and data augmentation are performed on the first and second sample road stud images to obtain target first sample road stud images and target second sample road stud images; the target first sample road stud images and target second sample road stud images are input into the road stud image reconstruction model to be trained to obtain first sample reconstructed images of the target first sample road stud images and second sample reconstructed images of the target second sample road stud images; based on the differences between the target first sample road stud image and the first sample reconstructed image, and the differences between the target second sample road stud image and the second sample reconstructed image, the road stud image reconstruction model to be trained is iteratively trained and optimized using KL divergence and binary cross-entropy loss to obtain a road stud image reconstruction model that matches the road stud types of the first and second sample road stud images.

[0072] For roads collected for the first time and road spikes of different types than those in the historical road spikes, a preset number of sampled road spike images are extracted from the target second sample road spike images. Based on the sampled road spike images, the road spike image reconstruction model is fine-tuned to obtain the fine-tuned road spike image reconstruction model. The fine-tuned road spike image reconstruction model is then used to update the road spike image reconstruction model.

[0073] For example, video frames containing road spikes crawled from the internet are input into the road spike detection model to obtain the road spike type and location information in the video frame. Based on the road spike location information, the video frame is cropped to obtain a second sample road spike image. Through interactive random sampling, 100 sample road spike images that meet the requirements are extracted from the second sample road spike images. Based on the 100 sample road spike images, the road spike image reconstruction model is fine-tuned.

[0074] Step S104: Based on the difference between the road spike image and the reconstructed image, obtain the status information of the road spike corresponding to the road spike image.

[0075] The status information of the road stud indicates its current condition, such as normal (intact) or abnormal (slightly damaged, severely damaged, covered with dirt, etc.).

[0076] Specifically, the terminal acquires the difference between the road stud image and the reconstructed image. When the difference is greater than or equal to a preset difference threshold, the status information of the road stud corresponding to the road stud image is confirmed as damaged; when the difference is less than the preset difference threshold, the status information of the road stud corresponding to the road stud image is confirmed as normal. For example, when a road stud is damaged or soiled, the difference between the road stud image and the reconstructed image is larger, and the larger the difference, the worse the condition of the road stud.

[0077] In the aforementioned road stud state recognition method, keyframes are extracted from road video frames. These keyframes represent the variations in road studs. The keyframes are input into a pre-trained road stud detection model to obtain the road stud type and location information within the keyframes. The road stud image from the keyframes is then input into a road stud image reconstruction model matching the road stud type to obtain a reconstructed image. The road stud image is cropped from the keyframes based on the road stud location information. The state information of the road stud corresponding to the road stud image is obtained based on the difference between the road stud image and the reconstructed image. This method can quickly identify the road stud type and location information in keyframes using a pre-trained road stud detection model, then crop the road stud image. The difference between the reconstructed image obtained from the road stud image reconstruction model and the original road stud image is used to obtain the state information of the road stud corresponding to the original image. This eliminates the need for manual detection of the road stud state, thus improving the efficiency of road stud state recognition.

[0078] In one embodiment, step S101, which extracts keyframes from road video frames, specifically includes the following: extracting a first keyframe from the road video frames according to the target positioning information of the road video frames at a preset positioning interval; performing optical flow analysis on the road video frames to obtain a second keyframe in the road video frames; and using the first keyframe and the second keyframe as keyframes.

[0079] Among them, target positioning information refers to the positioning information of each video frame in the road video frame. The positioning information can be GPS (Global Positioning System) information.

[0080] Specifically, in addition to carrying corresponding timestamp information, the timestamp information in the road video frame is also associated with the target positioning information. The terminal extracts the first key frame from the road video frame according to the target positioning information and according to the preset positioning interval. The road video frame is downsampled and fuzzy calculated to obtain the coarse optical flow between video frames in the road video frame. The second key frame in the road video frame is obtained according to the difference in coarse optical flow between video frames in the road video frame. The first key frame and the second key frame are used as key frames.

[0081] Furthermore, based on the timestamp information of the road video frames, a third key frame can be extracted from the road video frames at preset time intervals; the first key frame, the second key frame, and the third key frame are then used as key frames.

[0082] In this embodiment, the terminal can extract the first key frame according to the target positioning information of the road video frame at a preset positioning interval, perform optical flow analysis on the road video frame to obtain the second key frame in the road video frame, and extract the third key frame according to the timestamp information of the road video frame at a preset time interval, thereby realizing the reasonable extraction of the target video frame in the road video frame.

[0083] In one embodiment, before extracting the first keyframe from the road video frame according to the target positioning information of the road video frame at a preset positioning interval, the method further includes: determining the target video frame corresponding to the initial positioning information from the road video frame according to the initial positioning information of the road video frame and the timestamp information corresponding to the initial positioning information; performing linear interpolation processing on the intermediate video frames in the road video frame according to the initial positioning information to obtain the interpolated positioning information of the intermediate video frames; the intermediate video frames are video frames in the road video frame other than the target video frame; and using the initial positioning information of the target video frame and the interpolated positioning information of the intermediate video frames as the target positioning information of the road video frame.

[0084] The initial positioning information refers to the positioning information collected synchronously when collecting road video frames.

[0085] It should be noted that when collecting initial positioning information, the positioning information of each video frame is not collected. Instead, the positioning information of the current moment is obtained at a fixed frequency. This positioning information carries corresponding timestamp information. Therefore, by querying the timestamp information of the road video frame based on the timestamp information of the initial positioning information, the target video frame in the road video frame that corresponds to the initial positioning information can be obtained.

[0086] Specifically, video frames in the road video frame other than the target video frame are marked as intermediate video frames. Since the positioning information of the intermediate video frames is not collected, bilinear interpolation is performed on the positioning information of the intermediate video frames in the road video frame according to the initial positioning information to obtain the interpolated positioning information of the intermediate video frames. The target positioning information of all video frames in the road video frame is obtained from the initial positioning information of the target video frame and the interpolated positioning information of the intermediate video frames.

[0087] In this embodiment, the target video frame corresponding to the initial positioning information in the road video frame is determined based on the initial positioning information and the timestamp information corresponding to the initial positioning information. Based on the initial positioning information, linear interpolation processing is performed on the intermediate video frames in the road video frame to obtain the interpolated positioning information of the intermediate video frames, and then the target positioning information of the road video frame is obtained, thus realizing the reasonable determination of the positioning information of all video frames in the road video frame.

[0088] In one embodiment, the pre-trained road stud detection model is trained as follows: sample road video frames are acquired; the sample road video frames include road studs, each labeled with the target road stud type and target road stud location information; image enhancement processing is performed on the sample road video frames to obtain enhanced video frames; the enhanced video frames are input into the road stud detection model to be trained to obtain the predicted road stud type and predicted road stud location information in the enhanced video frames; based on the differences between the target road stud type and the predicted road stud type, and the differences between the target road stud location information and the predicted road stud location information, the road stud detection model to be trained is iteratively trained to obtain the pre-trained road stud detection model.

[0089] Among them, the sample road video frames refer to the video frame samples used to train the road stud detection model to be trained.

[0090] Among them, the pre-trained road spike detection model is a lightweight model, which can be a one-stage fast detection network.

[0091] Specifically, a large number of sample video frames from different provinces and regions are collected, including road spikes. The terminal extracts sample video frames from the terminal according to preset positioning intervals and / or preset time intervals. These sample video frames can be keyframes, or other types of video frames. The location information and type of road spikes in the sampled video frames are marked with rectangular boxes to obtain sample road video frames. Image enhancement processing is performed on the sample road video frames, including at least one of the following methods: random brightness adjustment, contrast adjustment, random pixel jitter adjustment, random overall image cropping, filling, and flipping. Enhanced video frames are input into the feature extraction network of the road spike detection model to be trained, resulting in feature vectors for the enhanced video frames. These feature vectors are then input into the detection network of the road spike detection model to predict road spike types and regress bounding boxes on a predetermined number of candidate boxes. Redundant bounding boxes are removed using non-maximum suppression to obtain the predicted road spike type and predicted road spike location information in the enhanced video frames. Based on the differences between the target road spike type and the predicted road spike type, as well as the differences between the target road spike location information and the predicted road spike location information, a loss function is constructed. The road spike detection model to be trained is iteratively trained based on the loss function to obtain a pre-trained road spike detection model.

[0092] Furthermore, the road spike detection model can be composed of a Residual Network (ResNet) and a Single Shot Detector (SSD). The feature extraction network of the road spike detection model can be a ResNet. ResNet adds skip connections to the network, allowing the input of the convolutional layers to be a combination of the outputs of previous network layers. Thus, the feature extraction network of the road spike detection model can learn the differences between the inputs and outputs of multiple network layers through residual learning, thereby effectively extracting the feature vectors of the enhanced video frames. The detection network of the road spike detection model can be an SSD. SSD predicts [x, y, w, h, c] (where x represents the x-coordinate of the center point of the road spike image, y represents the y-coordinate of the center point of the road spike image, w represents the length of the road spike image, h represents the width of the road spike image, and c represents the type of road spike) through sliding convolutions. For example, SSD can perform sliding prediction using 3x3 convolutions.

[0093] In this embodiment, image enhancement processing is performed on the collected sample road video frames to obtain enhanced video frames. The enhanced video frames are then input into the road stud detection model to be trained to obtain the predicted road stud type and predicted road stud location information in the enhanced video frames. Based on the differences between the target road stud type and the predicted road stud type, as well as the differences between the target road stud location information and the predicted road stud location information, the road stud detection model to be trained is iteratively trained to obtain a pre-trained road stud detection model. This model can then be used to obtain the road stud type and road stud location information. While maintaining high detection accuracy, it also achieves real-time detection in a lightweight manner, thereby improving the detection efficiency of road stud location information and road stud type, and reducing the power consumption of the detection process.

[0094] In one embodiment, step S103, which involves inputting the road spike image in the keyframe into a road spike image reconstruction model that matches the road spike type, to obtain a reconstructed image corresponding to the road spike image, specifically includes the following: performing convolution processing on the road spike image in the keyframe according to the road spike image reconstruction model that matches the road spike type to obtain the feature vector of the road spike image; and performing deconvolution processing on the feature vector according to the road spike image reconstruction model that matches the road spike type to obtain the reconstructed image.

[0095] The road spike image reconstruction model is trained using road spike images of a single type. The model includes an encoder and a decoder. The encoder maps high-resolution features from the top image of the road spike to low-resolution features in the top image. The decoder receives the low-resolution features from the top image and converts them into high-resolution features in the top image. These low-resolution features can also be called latent vectors.

[0096] Specifically, the terminal obtains a road spike image reconstruction model that matches the road spike type based on the road spike type in the road spike image; the encoder of the road spike image reconstruction model can be a convolution device, which performs convolution processing on the road spike image in the key frame to obtain the feature vector of the road spike image; the encoder of the road spike image reconstruction model can be a deconvolution device, which performs deconvolution processing on the feature vector to obtain the reconstructed image.

[0097] It should be noted that the feature vector of the road spike image is the latent vector in the road spike image. Since the latent vector contains relevant information about the road spike image, compared with the traditional technique of randomly selecting random noise for decoding, using the latent vector in the road spike image for deconvolution processing can obtain a reconstructed image with better reconstruction effect.

[0098] In this embodiment, the terminal reconstructs the image based on the road spike image reconstruction model corresponding to the road spike type. This allows the terminal to obtain the status information of the road spike corresponding to the road spike image based on the difference between the reconstructed image and the road spike image. This eliminates the need for manual detection of the road spike status and improves the efficiency of road spike status recognition.

[0099] In one embodiment, such as Figure 2 As shown, after obtaining the state information of the road spike corresponding to the road spike image based on the difference between the road spike image and the reconstructed image, the process also includes:

[0100] Step S201: Based on the feature information of the reconstructed image, the positioning information of the reconstructed image, and the unit interval distance of the road spikes, determine the actual road spikes and missing road spikes within the preset distance range.

[0101] In this context, "actual road studs" refers to the actual road studs that exist and their quantity. "Missing road studs" refers to the road studs that are missing from the road and their quantity.

[0102] The unit spacing refers to the distance at which road studs are placed on the road at unit intervals. For example, 1 meter, 2 meters, 3 meters, etc.

[0103] It should be noted that the above embodiments are for detecting and identifying road spikes in keyframes, that is, detecting and identifying road spikes that exist in keyframes, and cannot detect and identify missing road spikes.

[0104] For example, if the shooting angle of a keyframe is 0-20 meters, and the unit spacing of road spikes in the keyframe is 5 meters, assuming that road spikes are set starting from 0 meters in the keyframe, then the theoretical number of road spikes in the keyframe is 5, namely road spike A, road spike B, road spike C, road spike D, and road spike E. However, the actual road spikes in the keyframe are road spikes A, B, and C. Therefore, steps S101 to S104 can only detect and identify road spikes A, B, and C, and cannot detect and identify the missing road spikes D and E. Therefore, it is also necessary to determine the missing road spikes on the road and their number. In addition, since multiple keyframes may contain multiple images of the same road spike, i.e., multiple detections and identifications of the same road spike, it is necessary to determine the actual number of road spikes on the road.

[0105] Specifically, the terminal generates a sliding window of a preset length as a preset distance range; based on the feature information, positioning information, and timestamp information of the reconstructed image, it determines the actual road spikes and their quantity within the preset distance range; based on the unit interval distance of the road spikes, it determines the theoretical number of road spikes within the preset distance range; based on the difference between the theoretical and actual number of road spikes, it determines the number of missing road spikes; and based on the positioning information and unit interval distance of the actual road spikes, it determines the positioning information of the missing road spikes within the preset distance range.

[0106] For example, by combining timestamps, GPS information, and other data, a dynamically sliding fixed-distance window can be constructed. By using timestamps and feature information from reconstructed images, the actual number of road spikes and the number of missing road spikes within the fixed-distance window can be calculated. It is also possible to calculate the number of road spikes in good condition for different types of road sections per 100 meters.

[0107] Step S202: Based on the actual status information, location information, and timestamp information of the road studs, as well as the location information and timestamp information of the missing road studs, generate the detection results of the road video frames.

[0108] Specifically, based on the actual status information, location information, and timestamp information of the road studs, as well as the location information and timestamp information of the missing road studs, the detection results of the road studs for the entire road section are generated. The detection results are stored in the form of a time series, and then the detection results of each road stud can be re-verified based on the time series.

[0109] For example, it can generate electronic data of road studs for the entire road segment, displaying the location and status of road studs for the entire road segment. It can also manage road studs for the entire road segment by combining timestamp information. Furthermore, it can verify the changes in road stud status by combining time series. For example, if the road stud at the current location is intact on the first day, and also intact on the second and third days, the electronic data of the road studs shows that the road stud is damaged on the fourth day, indicating that the road stud at that location was in an abnormal state on the fourth day.

[0110] In this embodiment, the terminal generates digital detection results of road video frames based on the actual status information, location information, and timestamp information of the road studs, as well as the location information and timestamp information of the missing road studs. This enables temporal and spatial correlation of the road stud status, allowing relevant personnel to view and manage road studs with abnormal status in a timely manner, so as to repair road sections with damaged or missing road studs and improve driving safety.

[0111] In one embodiment, such as Figure 3 As shown, another method for identifying road stud status is provided. Taking the application of this method to a terminal as an example, the method includes the following steps:

[0112] Step S301: Based on the initial positioning information of the road video frame and the timestamp information corresponding to the initial positioning information, determine the target video frame corresponding to the initial positioning information from the road video frame.

[0113] Step S302: Based on the initial positioning information, perform linear interpolation on the intermediate video frames in the road video frames to obtain the interpolated positioning information of the intermediate video frames; the intermediate video frames are the video frames in the road video frames other than the target video frames.

[0114] Step S303: Use the initial positioning information of the target video frame and the interpolated positioning information of the intermediate video frames as the target positioning information of the road video frame.

[0115] Step S304: Based on the target positioning information of the road video frame, extract the first key frame from the road video frame according to the preset positioning interval.

[0116] Step S305: Perform optical flow analysis on the road video frame to obtain the second keyframe in the road video frame.

[0117] Step S306: Use the first keyframe and the second keyframe as keyframes.

[0118] Step S307: Input the keyframe into the pre-trained road spike detection model to obtain the road spike type and position information in the keyframe.

[0119] Step S308: Based on the road spike image reconstruction model that matches the road spike type, perform convolution processing on the road spike image in the key frame to obtain the feature vector of the road spike image; based on the road spike image reconstruction model that matches the road spike type, perform deconvolution processing on the feature vector to obtain the reconstructed image.

[0120] Step S309: Obtain the status information of the road spike corresponding to the road spike image based on the difference between the road spike image and the reconstructed image.

[0121] In this embodiment, a pre-trained road spike detection model can quickly identify the road spike type and location information in a keyframe, and then crop out a road spike image. The reconstructed image obtained by the road spike image reconstruction model corresponding to the road spike type is obtained by the difference between the reconstructed image and the road spike image, thus obtaining the road spike status information corresponding to the road spike image. There is no need for manual detection of the road spike status, which improves the efficiency of road spike status recognition.

[0122] To more clearly illustrate the road spike status recognition method provided in the embodiments of this disclosure, the following specific embodiment will be used to describe the road spike status recognition method in detail. In one embodiment, another method for identifying road stud status is provided, which can be applied to an in-vehicle terminal. Specifically, it includes the following: The in-vehicle terminal continuously extracts frames from the collected roadside video to obtain extracted video frames. Combined with synchronously collected GPS information, the location information of the extracted video frames is obtained. Key frames are determined using an inter-frame difference method combined with vehicle speed changes. Based on a large amount of collected road stud data of different types, data augmentation and cleaning are performed to train a one-stage fast detection network. The key frames are input into the one-stage fast detection network to quickly detect the road stud location information and road stud type. The key frames are then cropped based on the road stud location information to obtain road stud images. A road stud image reconstruction model is trained based on a variational autoencoder model. The road stud images are input into the road stud image reconstruction model for image reconstruction to obtain reconstructed images. The difference between the input road stud image and the output reconstructed image is calculated, and the road stud status is determined based on the difference. Based on GPS information and relevant information from the reconstructed images, the number of effective non-repeating road studs within an equidistant window is calculated to determine the overall status of road studs across the entire road segment.

[0123] The above-mentioned method for identifying the status of road spikes has the following beneficial effects:

[0124] (1) The vehicle-mounted terminal can be installed on various maintenance and social vehicles and special inspection vehicles;

[0125] (2) The location and status of road studs can be quickly detected and identified through the vehicle terminal without the need for manual detection and identification, which greatly reduces manpower and material costs. The location and status of road studs can be automatically detected and identified during the daily maintenance and inspection of vehicles.

[0126] (3) The lightweight network structure design enables edge computing and real-time computing of the vehicle terminal, while reducing the energy consumption of the vehicle terminal.

[0127] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0128] Based on the same inventive concept, this application also provides a road spike status identification device for implementing the road spike status identification method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more road spike status identification device embodiments provided below can be found in the limitations of the road spike status identification method described above, and will not be repeated here.

[0129] In one embodiment, such as Figure 4 As shown, a road stud status recognition device 400 is provided, including: a keyframe acquisition module 401, a road stud information acquisition module 402, a road stud image reconstruction module 403, and a road stud status acquisition module 404, wherein:

[0130] The keyframe acquisition module 401 is used to extract keyframes from road video frames; keyframes are used to represent the differences in road studs.

[0131] The road spike information acquisition module 402 is used to input key frames into a pre-trained road spike detection model to obtain the road spike type and position information of the road spikes in the key frames.

[0132] The road spike image reconstruction module 403 is used to input the road spike image in the key frame into the road spike image reconstruction model that matches the road spike type to obtain the reconstructed image corresponding to the road spike image; the road spike image is obtained by cropping the key frame based on the road spike position information.

[0133] The road spike status acquisition module 404 is used to obtain the status information of the road spike corresponding to the road spike image based on the difference between the road spike image and the reconstructed image.

[0134] In one embodiment, the keyframe acquisition module 401 is further configured to extract a first keyframe from the road video frame according to the target positioning information of the road video frame and at a preset positioning interval; perform optical flow analysis on the road video frame to obtain a second keyframe in the road video frame; and use the first keyframe and the second keyframe as keyframes.

[0135] In one embodiment, the road stud status recognition device 400 further includes a video frame positioning module, configured to determine the target video frame corresponding to the initial positioning information from the road video frame based on the initial positioning information and the timestamp information corresponding to the initial positioning information; perform linear interpolation processing on the intermediate video frames in the road video frame based on the initial positioning information to obtain the interpolated positioning information of the intermediate video frames; the intermediate video frames are the video frames in the road video frame other than the target video frame; and use the initial positioning information of the target video frame and the interpolated positioning information of the intermediate video frames as the target positioning information of the road video frame.

[0136] In one embodiment, the road stud state recognition device 400 further includes a detection model training module for acquiring sample road video frames; the sample road video frames include road studs, and the road studs are labeled with target road stud type and target road stud location information; the sample road video frames are subjected to image enhancement processing to obtain enhanced video frames; the enhanced video frames are input into the road stud detection model to be trained to obtain the predicted road stud type and predicted road stud location information of the road studs in the enhanced video frames; based on the difference between the target road stud type and the predicted road stud type, and the difference between the target road stud location information and the predicted road stud location information, the road stud detection model to be trained is iteratively trained to obtain a pre-trained road stud detection model.

[0137] In one embodiment, the road spike image reconstruction module 403 is further configured to perform convolution processing on the road spike image of the road spike in the key frame according to the road spike image reconstruction model that matches the road spike type, to obtain the feature vector of the road spike image; and perform deconvolution processing on the feature vector according to the road spike image reconstruction model that matches the road spike type, to obtain the reconstructed image.

[0138] In one embodiment, the road stud status recognition device 400 further includes a detection result acquisition module, which is used to determine the actual road studs and missing road studs within a preset distance range based on the feature information of the reconstructed image, the positioning information of the reconstructed image, and the unit interval distance of the road studs; and to generate the detection result of the road video frame based on the status information, positioning information, and timestamp information of the actual road studs, as well as the positioning information and timestamp information of the missing road studs.

[0139] Each module in the aforementioned road stud status recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0140] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a road stud status recognition method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0141] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0142] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0143] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0144] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0145] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for identifying the status of road studs, characterized in that, The method includes: Based on the target positioning information of the road video frame, key frames are extracted from the road video frame at preset positioning intervals; the key frames are used to represent the differences in road stud changes. The keyframes are input into a pre-trained road spike detection model to obtain the road spike type and position information in the keyframes. The rail spike image in the keyframe is input into a rail spike image reconstruction model that matches the rail spike type to obtain the reconstructed image corresponding to the rail spike image; the rail spike image is obtained by cropping the keyframe based on the rail spike position information; Based on the difference between the spike image and the reconstructed image, the state information of the spike corresponding to the spike image is obtained; Based on the feature information of the reconstructed image, the target positioning information of the reconstructed image, and the unit spacing distance of the road spikes, the actual road spikes within a preset distance range are determined; Based on the unit spacing of the rail spikes and the positioning information of the actual rail spikes, the missing rail spikes within the preset distance range are determined; Based on the status information, location information, and timestamp information of the actual road studs, as well as the location information and timestamp information of the missing road studs, the detection results of the road video frames are generated.

2. The method according to claim 1, characterized in that, The step of extracting keyframes from the road video frames includes: Based on the target positioning information of the road video frame, the first key frame is extracted from the road video frame according to a preset positioning interval; Optical flow analysis is performed on the road video frame to obtain the second keyframe in the road video frame; The first keyframe and the second keyframe are used as the keyframes.

3. The method according to claim 2, characterized in that, Before extracting the first keyframe from the road video frame according to the target positioning information of the road video frame at a preset positioning interval, the method further includes: Based on the initial positioning information of the road video frame and the timestamp information corresponding to the initial positioning information, the target video frame corresponding to the initial positioning information is determined from the road video frame; Based on the initial positioning information, linear interpolation is performed on the intermediate video frames in the road video frames to obtain the interpolated positioning information of the intermediate video frames; the intermediate video frames are the video frames in the road video frames other than the target video frame. The initial positioning information of the target video frame and the interpolated positioning information of the intermediate video frames are used as the target positioning information of the road video frame.

4. The method according to claim 1, characterized in that, The pre-trained road spike detection model is trained in the following manner: Acquire sample road video frames; the sample road video frames include road studs, and the road studs are labeled with the target road stud type and the target road stud location information; The sample road video frames are subjected to image enhancement processing to obtain enhanced video frames; The enhanced video frame is input into the road spike detection model to be trained to obtain the predicted road spike type and predicted road spike location information in the enhanced video frame. Based on the difference between the target road stud type and the predicted road stud type, and the difference between the target road stud location information and the predicted road stud location information, the road stud detection model to be trained is iteratively trained to obtain the pre-trained road stud detection model.

5. The method according to claim 1, characterized in that, The step of inputting the rail spike image from the keyframe into a rail spike image reconstruction model that matches the rail spike type to obtain the reconstructed image corresponding to the rail spike image includes: Based on the road spike image reconstruction model that matches the road spike type, the road spike image in the key frame is convolved to obtain the feature vector of the road spike image. The feature vectors are deconvolutionally processed according to the road spike image reconstruction model that matches the road spike type to obtain the reconstructed image.

6. A rail spike status recognition device, characterized in that, The device includes: The keyframe acquisition module is used to extract keyframes from the road video frame according to the target positioning information of the road video frame and at a preset positioning interval; the keyframes are used to represent the changes in road studs. The road spike information acquisition module is used to input the key frame into a pre-trained road spike detection model to obtain the road spike type and road spike location information in the key frame; The road spike image reconstruction module is used to input the road spike image in the key frame into a road spike image reconstruction model that matches the road spike type to obtain the reconstructed image corresponding to the road spike image; the road spike image is obtained by cropping the key frame based on the road spike position information; The road spike status acquisition module is used to obtain the status information of the road spike corresponding to the road spike image based on the difference between the road spike image and the reconstructed image; The detection result acquisition module is used to determine the actual road spikes within a preset distance range based on the feature information of the reconstructed image, the positioning information of the reconstructed image, and the unit interval distance of the road spikes; determine the missing road spikes within the preset distance range based on the unit interval distance of the road spikes and the positioning information of the actual road spikes; and generate the detection result of the road video frame based on the status information, positioning information, and timestamp information of the actual road spikes, as well as the positioning information and timestamp information of the missing road spikes.

7. The apparatus according to claim 6, characterized in that, The road spike status recognition device also includes a video frame positioning module; The video frame positioning module is configured to: determine the target video frame corresponding to the initial positioning information from the road video frame based on the initial positioning information and the timestamp information corresponding to the initial positioning information; perform linear interpolation processing on the intermediate video frames in the road video frame based on the initial positioning information to obtain the interpolated positioning information of the intermediate video frames; the intermediate video frames are the video frames in the road video frame other than the target video frame; and use the initial positioning information of the target video frame and the interpolated positioning information of the intermediate video frames as the target positioning information of the road video frame.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images