Road condition recognition method and device, vehicle end control equipment, storage medium and program product

Abstract

The invention relates to a road condition recognition method and device, vehicle end control equipment, a storage medium and a program product. The method comprises the following steps: dividing a road condition video into video segments, and obtaining a visual feature vector, an optical flow feature vector and a brightness feature vector of each video frame in the video segments; obtaining the complexity of each video frame according to the optical flow feature vector and the brightness feature vector of each video frame; performing key frame extraction processing on the video clip according to the complexity of each video frame in the video clip to obtain a key frame in the video clip; obtaining global features of the video clip according to the visual feature vectors, the optical flow feature vectors and the brightness feature vectors of all the key frames of the video clip; and inputting the global feature of the video clip into a pre-trained road condition classification model to obtain a road condition classification result of the video clip. By adopting the method, the road condition recognition precision and recognition efficiency can be effectively balanced.

Description

Technical Field

[0001] This application relates to the field of intelligent vehicle technology, and in particular to a road condition recognition method, device, vehicle-side control equipment, storage medium, and program product. Background Technology

[0002] With the rapid development of intelligent driving technology, real-time and accurate road condition recognition has become a key requirement for ensuring vehicle driving safety and optimizing vehicle control.

[0003] In related technologies, a fixed frame extraction strategy is usually used to extract video frames for road condition recognition. This fixed frame extraction strategy is easy to waste processor computing power when applied to simple road sections, and may lose key information when applied to complex road sections, making it difficult to balance the accuracy and efficiency of road condition recognition. Summary of the Invention

[0004] Based on this, this application addresses the aforementioned technical problems by providing a road condition recognition method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can at least effectively balance road condition recognition accuracy and recognition efficiency.

[0005] Firstly, this application provides a road condition recognition method, including:

[0006] The road condition video is divided into video segments, and the visual feature vector, optical flow feature vector, and luminance feature vector of each video frame in the video segment are obtained.

[0007] The complexity of each video frame is obtained based on the optical flow feature vector and luminance feature vector of each video frame;

[0008] Based on the complexity of each video frame in the video segment, keyframe extraction processing is performed on the video segment to obtain the keyframes in the video segment;

[0009] The global features of the video segment are obtained based on the visual feature vectors, optical flow feature vectors, and luminance feature vectors of all the keyframes of the video segment.

[0010] The global features of the video clip are input into a pre-trained traffic condition classification model to obtain the traffic condition classification result of the video clip.

[0011] The aforementioned road condition recognition method has at least the following beneficial effects: it divides road condition videos into video segments and comprehensively extracts the visual feature vector, optical flow feature vector, and brightness feature vector of each video frame, realizing multi-dimensional representation of static content, dynamic motion, and lighting conditions; then, it intelligently evaluates the complexity of each video frame based on optical flow and brightness feature vectors, and accurately selects the most representative key frames accordingly, which not only preserves the core semantic information of the video segments but also significantly reduces data redundancy; subsequently, it fuses the multi-dimensional feature vectors of all key frames to generate global features of the video segments, which are finally input into the road condition classification model to output highly reliable road condition classification results. By selecting key frames through complexity, it not only effectively compresses the amount of computation and improves the processing speed of vehicle-side control equipment, but also ensures the comprehensiveness and accuracy of road condition classification results through multi-modal feature fusion, effectively balancing road condition recognition accuracy and recognition efficiency. It is particularly suitable for complex and ever-changing driving scenarios, providing accurate and efficient road condition understanding support for vehicle-side applications such as intelligent driving assistance, traffic flow monitoring, and autonomous driving decision-making.

[0012] In an optional embodiment of the first aspect, global features of the video segment are obtained based on the visual feature vectors, optical flow feature vectors, and luminance feature vectors of all the keyframes of the video segment, including:

[0013] The visual feature vector, optical flow feature vector, and brightness feature vector of the key frame are concatenated to obtain the multimodal feature vector of the key frame.

[0014] Based on the multimodal feature vectors of all the keyframes of the video segment, a feature vector sequence of the video segment is constructed;

[0015] The feature vector sequence is input into a pre-trained feature fusion model to obtain the global features of the video segment.

[0016] In this embodiment, at least the following beneficial effects are achieved: by concatenating the visual feature vectors, optical flow feature vectors, and brightness feature vectors of key frames to construct multimodal feature vectors, it is possible to fully integrate image semantics, inter-frame motion, and illumination distribution information, thereby improving the ability of multimodal feature vectors to represent complex road conditions; furthermore, by constructing feature sequences based on the multimodal feature vectors of all key frames, the temporal dynamic information of video segments can be preserved; and by inputting this sequence into a pre-trained feature fusion model, deep interaction and global aggregation of multimodal and temporal information can be achieved, ultimately obtaining more robust and semantically complete global features of video segments, effectively improving the accuracy and scene generalization ability of vehicle road condition recognition.

[0017] In an optional embodiment of the first aspect, keyframe extraction processing is performed on the video segment according to the complexity of each video frame in the video segment to obtain keyframes in the video segment, including:

[0018] Based on the complexity of each video frame in the video segment, candidate probability information for each video frame is determined; the candidate probability information is used to characterize the probability distribution of the video frame being selected as the keyframe.

[0019] Based on the candidate probability information of each video frame, differentiable sampling processing is performed on the video frames in the video segment to obtain the keyframes in the video segment.

[0020] In this embodiment, at least the following beneficial effects are achieved: based on the complexity of each video frame, the probability distribution of its selection as a key frame is dynamically calculated, so that video frames with high complexity and rich information have a higher chance of being selected, ensuring that key frames can accurately capture the core dynamics and key scenes in the video segment; furthermore, through differentiable sampling processing, the discrete frame selection process is transformed into a continuous and differentiable mathematical operation, so that the entire key frame extraction process can be seamlessly embedded into the traffic condition classification model for joint optimization training. This not only ensures the rationality and representativeness of key frame selection and effectively compresses the amount of video data, but also breaks through the limitation of non-differentiability of traditional key frame extraction methods, providing strong support for end-to-end training and global optimization of the model, and significantly improving the performance and efficiency of traffic condition classification tasks.

[0021] In an optional embodiment of the first aspect, the complexity of each video frame is obtained based on the optical flow feature vector and the luminance feature vector of each video frame, including:

[0022] The optical flow feature vector and the luminance feature vector of each video frame are concatenated to obtain the concatenated vector of each video frame.

[0023] The concatenated vector is input into a pre-trained complexity evaluation model to obtain the complexity of each video frame.

[0024] In this embodiment, at least the following beneficial effects are achieved: by concatenating the optical flow feature vector and the brightness feature vector of each video frame, a deep fusion of motion information and illumination information is realized, comprehensively integrating the intensity of motion and illumination change characteristics in dynamic scenes. Based on this, the concatenated vector is input into a pre-trained complexity evaluation model, making the complexity evaluation results closer to the visual perception requirements of real driving environments. Furthermore, end-to-end model inference improves the evaluation efficiency and accuracy.

[0025] In an optional embodiment of the first aspect, the pre-trained complexity evaluation model, the differentiable sampling process, the pre-trained feature fusion model, and the pre-trained road condition classification model are trained in the following manner:

[0026] The sample video segments are labeled to obtain the road condition tags for the sample video segments;

[0027] The global features of the sample video clip are input into the road condition classification model to be trained to obtain the sample classification result of the sample video clip;

[0028] Based on the sample classification results and the road condition labels, the classification loss value is obtained;

[0029] Based on the keyframes of the sample video segment, a sparse loss value is obtained;

[0030] Based on the classification loss value and the sparse loss value, the complexity evaluation model to be trained, the differentiable sampling process to be trained, the feature fusion model to be trained, and the road condition classification model to be trained are collaboratively updated to obtain the pre-trained complexity evaluation model, the differentiable sampling process, the pre-trained feature fusion model, and the pre-trained road condition classification model.

[0031] In this embodiment, at least the following beneficial effects are achieved: Accurate annotation of sample video segments yields real-world road condition labels, which are then forward-propagated through complexity assessment, differentiable sampling, feature fusion, and road condition classification modules to obtain sample classification results. Furthermore, a comprehensive loss function is obtained by combining the classification loss value with a sparse loss value constructed based on keyframes, merging classification accuracy and feature selection rationality. Finally, the target loss value calculated using this comprehensive loss function is used to collaboratively update the parameters of the complexity assessment model, differentiable sampling module, feature fusion model, and road condition classification model. This not only ensures performance optimization of each module in its respective task but also achieves deep coupling and joint optimization between modules through gradient backpropagation. The sparse loss guides the complexity assessment model to focus on information-rich frames, while the classification loss drives the feature fusion and classification models to extract more discriminative global representations. The resulting model achieves global optimum in terms of keyframe selection rationality, feature expression discriminability, and final classification accuracy, significantly improving generalization ability and robustness under complex road conditions.

[0032] In an alternative embodiment of the first aspect, the optical flow feature vector of each video frame in the video segment is obtained in the following manner:

[0033] Based on the motion vector field of the video frame, the motion amplitude of each pixel in each video frame is obtained;

[0034] Based on the motion amplitude of each pixel in each video frame, the first optical flow statistical feature of each video frame is obtained;

[0035] Based on the first optical flow statistical features and motion amplitude of each video frame, the second optical flow statistical features of each video frame are obtained;

[0036] The optical flow feature vector of each video frame is obtained based on the first optical flow statistical feature and the second optical flow statistical feature of each video frame.

[0037] In this embodiment, at least the following beneficial effects are achieved: First, the motion amplitude of each pixel is calculated based on the motion vector field to capture the motion intensity at the microscopic level; then, a first optical flow statistical feature is generated through statistical aggregation to characterize the overall motion distribution; next, the original motion amplitude and the first optical flow statistical feature are combined to extract a more discriminative second optical flow statistical feature, revealing the deep-seated patterns of motion between consecutive video frames; finally, the first and second optical flow statistical features are fused into an optical flow feature vector, improving the ability to understand and describe dynamic scenes in video frames.

[0038] In an optional embodiment of the first aspect, the luminance feature vector of each video frame in the video segment is obtained in the following manner:

[0039] The luminance entropy of each video frame is obtained based on the quantization level of the luminance component of each video frame.

[0040] The brightness contrast of each video frame is obtained based on the brightness component of each video frame.

[0041] The luminance feature vector of each video frame is obtained based on the luminance entropy and luminance contrast of each video frame.

[0042] In this embodiment, at least the following beneficial effects are achieved: Brightness entropy, based on the quantization level of brightness components, effectively characterizes the complexity and richness of illumination distribution in an image, reflecting the texture details and illumination levels of the scene; brightness contrast accurately measures the intensity of difference between bright and dark areas, revealing the visual impact and clarity of the image. The brightness feature vector formed by fusing the two not only takes into account the statistical characteristics of illumination distribution but also includes key dimensions of visual perception, significantly improving the accuracy and descriptive ability of perceiving illumination conditions in visual scenes, providing more discriminative feature inputs for subsequent road condition recognition tasks. It is particularly suitable for driving scenarios with drastic changes in illumination, effectively supporting the vehicle-side control equipment in accurately identifying and quickly responding to complex illumination conditions such as tunnel entry and exit and nighttime driving.

[0043] In an optional embodiment of the first aspect, the visual feature vector of each video frame in the video segment is obtained in the following manner:

[0044] Each video frame in the video segment is sequentially scaled and normalized to obtain the processed video frame for each video frame.

[0045] Global pooling is performed on each of the processed video frames to obtain the visual feature vector of each video frame.

[0046] In this embodiment, at least the following beneficial effects are achieved: by scaling each video frame in the video segment, the problem of inconsistent feature dimensions caused by differences in video size can be reduced; by normalizing the video frames, the distribution of pixel values ​​under different lighting and contrast conditions can be balanced, enhancing the robustness and generalization ability of visual features; by performing global pooling on the video frames, high-dimensional image data can be compressed into fixed-length visual feature vectors, which not only preserves the key semantic information of the video frames but also significantly reduces the data dimension, reducing computational and storage overhead, providing high-quality, low-redundancy feature input for subsequent video analysis, and helping to improve the efficiency and accuracy of vehicle road condition recognition.

[0047] In an optional embodiment of the first aspect, dividing the traffic video into video segments includes:

[0048] The target number of frames for each video segment is obtained based on the preset frame rate and preset segment duration;

[0049] The road condition video is divided into multiple non-overlapping video segments, each containing several video frames of the target frame.

[0050] In this embodiment, at least the following beneficial effects are achieved: the target number of frames for each video segment is calculated by setting a preset frame rate and a preset segment duration, and the traffic video is divided into multiple non-overlapping video segments of equal length. This realizes the standardization and structured processing of traffic data, ensures the integrity and consistency of each video segment in the time dimension, provides a regular data unit for subsequent video analysis and feature extraction, and effectively improves processing efficiency. At the same time, the non-overlapping division strategy avoids duplicate storage and computational redundancy of data, optimizes storage space utilization and computational resource consumption, and lays a solid data foundation for the analysis and processing of traffic video.

[0051] Secondly, this application also provides a road condition recognition device, comprising:

[0052] The vector acquisition module is used to divide the road condition video into video segments and acquire the visual feature vector, optical flow feature vector and brightness feature vector of each video frame in the video segment;

[0053] The complexity acquisition module is used to obtain the complexity of each video frame based on the optical flow feature vector and the luminance feature vector of each video frame;

[0054] The keyframe extraction module is used to perform keyframe extraction processing on the video segment according to the complexity of each video frame in the video segment, so as to obtain the keyframes in the video segment.

[0055] The feature fusion module is used to obtain the global features of the video segment based on the visual feature vectors, optical flow feature vectors, and luminance feature vectors of all the key frames of the video segment;

[0056] The traffic condition classification module is used to input the global features of the video clip into a pre-trained traffic condition classification model to obtain the traffic condition classification result of the video clip.

[0057] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0058] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0059] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the above aspects.

[0060] Regarding the beneficial effects of any of the technical solutions in the second to fifth aspects mentioned above, refer to the beneficial effects of the corresponding technical solutions in the first aspect; repeated examples will not be listed here. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a schematic diagram of an optional flow of the road condition recognition method in one embodiment;

[0063] Figure 2 This is an optional flowchart illustrating the steps for obtaining global features of a video segment in one embodiment;

[0064] Figure 3 This is a schematic diagram of an optional process from acquiring global features to outputting road condition classification results in one embodiment;

[0065] Figure 4 This is a schematic diagram of an optional process for an end-to-end joint training model in one embodiment;

[0066] Figure 5 This is a schematic diagram of another optional flow of the road condition recognition method in one embodiment;

[0067] Figure 6 This is a schematic diagram of another optional flow of the road condition recognition method in one embodiment;

[0068] Figure 7 This is a schematic diagram of an optional structure of the road condition recognition device in one embodiment;

[0069] Figure 8 This is a schematic diagram of an optional internal structure of a computer device in one embodiment. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0071] The terms "first," "second," etc., used in this application may be used to describe various elements, but these elements are not limited by these terms. These terms are used only to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0072] In one embodiment, such as Figure 1 As shown, a road condition recognition method is provided. This embodiment illustrates the application of this method to a vehicle-side control device. It is understood that this method can also be applied to a server, or to a system including both a vehicle-side control device and a server, and is implemented through the interaction between the vehicle-side control device and the server. In this embodiment, the method includes the following steps:

[0073] Step S101: Divide the road condition video into video segments and obtain the visual feature vector, optical flow feature vector, and brightness feature vector of each video frame in the video segment.

[0074] Road condition video refers to video frames that record the process of a vehicle driving (including the road surface, vehicle, and environment). For example, road condition video can be video captured by a vehicle dashcam.

[0075] In this context, visual feature vectors refer to feature vectors that reflect the image content of a video frame. For example, visual feature vectors can reflect visual features such as road surface texture and smoothness.

[0076] Among them, optical flow feature vector refers to the feature vector that reflects the motion changes of a target (such as an object, road surface, etc.) in a video frame, such as its speed, direction, and rate of change.

[0077] Among them, the brightness feature vector refers to the feature vector that reflects the brightness, darkness and illumination of the video frame.

[0078] For example, the vehicle-mounted control device can read road condition videos captured by a dashcam, then divide the video into video segments of a target number of frames (or a fixed duration), and perform road condition recognition on a segment-by-segment basis. From each video frame in each video segment, the vehicle-mounted control device extracts visual feature vectors, optical flow feature vectors, and luminance feature vectors. The optical flow and luminance feature vectors can be used as "pseudo-multimodal" feature vectors to simulate multi-source modalities, enhancing the model's adaptability to complex environments such as lighting changes, shadows, rain and fog, and uneven camera exposure, ensuring the model's stability and accuracy in road condition recognition across various scenarios.

[0079] Step S102: Based on the optical flow feature vector and luminance feature vector of each video frame, obtain the complexity of each video frame.

[0080] For example, a lightweight complexity evaluation model can be pre-built. For instance, it can be based on a multilayer perceptron (MLP) or a lightweight neural network. The sample optical flow vector and sample brightness feature vector of the sample video frame are used as input data and fed into the lightweight complexity evaluation model. The output is the complexity of the sample video frame. The model parameters of the lightweight complexity evaluation model are iteratively updated using the difference between the output complexity and the complexity label of the sample video frame, and finally the pre-trained complexity evaluation model is obtained.

[0081] Compared to traditional complex models, lightweight complexity evaluation models are better suited for in-vehicle environments where computational performance is paramount and storage space is limited. Understandably, model training can be completed on other devices, servers, or cloud servers, and the pre-trained complexity evaluation model can then be deployed on the vehicle's control equipment.

[0082] The vehicle-mounted control equipment inputs the optical flow feature vector and luminance feature vector of the video frame into a pre-trained complexity evaluation model. The pre-trained complexity evaluation model performs complexity evaluation on the optical flow feature vector and luminance feature vector, and outputs the complexity of each video frame.

[0083] Step S103: Based on the complexity of each video frame in the video segment, perform keyframe extraction processing on the video segment to obtain the keyframes in the video segment.

[0084] For example, the vehicle-side control device can use the complexity of each video frame in the video segment as the basis for frame extraction, and use a frame extraction algorithm to perform frame extraction processing on the video segment to obtain one or more key frames in the video segment.

[0085] The number of keyframes extracted is not a fixed value, but dynamically determined by the complexity of each video frame in the video clip. For example, a reasonable range for the number of keyframes can be determined experimentally (e.g., 2-6 frames), and then the extraction algorithm and complexity can be dynamically adjusted within this range. For instance, in a video clip of a flat road, there are fewer complex video frames, so 2 keyframes can be selected; in a video clip of a complex road, there are more complex video frames, so 5 keyframes can be selected.

[0086] Step S104: Obtain the global features of the video segment based on the visual feature vectors, optical flow feature vectors, and luminance feature vectors of all keyframes of the video segment.

[0087] Global features refer to feature vectors that represent the semantic, structural, and distributional information of a video clip as a whole. Global features are used to reflect the overall road conditions of a video clip.

[0088] For example, the vehicle-mounted control device concatenates the visual feature vector, optical flow feature vector, and luminance feature vector of each key frame to obtain the multimodal feature vector of each key frame, and then performs feature fusion processing on the multimodal feature vectors of all key frames to form the global features of the video segment.

[0089] Step S105: Input the global features of the video clip into the pre-trained traffic condition classification model to obtain the traffic condition classification result of the video clip.

[0090] For example, the global features of the video clip are input into a pre-trained traffic condition classification model. The pre-trained traffic condition classification model is used to perform traffic condition classification processing, and the traffic condition category probability distribution of the video clip is output. The confidence of the traffic condition probability category distribution is calculated. Finally, the traffic condition category probability distribution and its confidence can be set as the traffic condition classification result of the video clip.

[0091] The road condition classification model can be a classifier built using a simple fully connected layer (Linear Layer).

[0092] In practical applications, the global features (h) of a video segment can be obtained by using a feature fusion model built based on the Transformer model. cls The road condition classification model built based on the fully connected layer receives the global features (h) output by the Transformer. cls ), and the global features (h) are processed through the Softmax function of the fully connected layer. cls ) is mapped to a road condition category probability distribution ( The process of obtaining the probability distribution of road condition categories using a road condition classification model can be represented by the following formula:

[0093]

[0094] In the formula, W cls and b cls This represents the model parameters of the road condition classification model.

[0095] Among them, W cls W represents the weight matrix of the road condition classification model. cls It is a trainable parameter matrix of shape [D, C], where D is the global feature (h cls The dimension of W is C, where C is the total number of road condition categories. cls Its function is to: transform high-dimensional global features (h) cls It is linearly mapped to the class score (logits) space.

[0096] Among them, b cls This represents the bias vector of the classifier for the road condition classification model, with a length equal to the total number of classes, C. cls Its function is to provide a basic offset for each category, which is added to the result of the linear mapping to form the final category score.

[0097] In the aforementioned road condition recognition method, road condition videos are divided into video segments, and the visual feature vector, optical flow feature vector, and luminance feature vector of each video frame are comprehensively extracted, realizing multi-dimensional representation of static content, dynamic motion, and lighting conditions. Then, based on the optical flow and luminance feature vectors, the complexity of each video frame is intelligently evaluated, and the most representative key frames are accurately selected accordingly. This preserves the core semantic information of the video segments while significantly reducing data redundancy. Subsequently, the multi-dimensional feature vectors of all key frames are fused to generate global features of the video segments, which are finally input into the road condition classification model, outputting highly reliable road condition classification results. By selecting key frames based on complexity, not only is the computational load effectively reduced and the processing speed of vehicle-side control equipment improved, but the multi-modal feature fusion also ensures the comprehensiveness and accuracy of the road condition classification results, effectively balancing road condition recognition accuracy and efficiency. This method is particularly suitable for complex and ever-changing driving scenarios, providing accurate and efficient road condition understanding support for vehicle-side applications such as intelligent driving assistance, traffic flow monitoring, and autonomous driving decision-making.

[0098] In one exemplary embodiment, such as Figure 2 As shown, in step S104 above, based on the visual feature vectors, optical flow feature vectors, and luminance feature vectors of all keyframes of the video clip, the global features of the video clip are obtained, including the following:

[0099] Step S201: Perform vector concatenation on the visual feature vector, optical flow feature vector, and brightness feature vector of the key frame to obtain the multimodal feature vector of the key frame.

[0100] For example, Figure 3 This is a flowchart illustrating the process from acquiring global features to outputting road condition classification results. For the sampled keyframes, based on modal embedding vectors and temporal location encoding, their visual feature vectors, optical flow feature vectors, and brightness feature vectors are concatenated to form the multimodal feature vector of the keyframe.

[0101] In practical applications, the vector concatenation process of multimodal feature vectors can be represented by the following formula:

[0102]

[0103] In the formula, Represents the multimodal feature vector of the t-th keyframe; Represents a linear projection matrix; Represents visual feature vectors; Represents the optical flow eigenvector; Represents the brightness feature vector; Represents the modal embedding vector; This indicates the time and location encoding.

[0104] The modality embedding vector is used to indicate whether the current feature comes from a visual feature vector, an optical flow feature vector, or a brightness feature vector, so that the subsequent feature fusion model can distinguish the semantic differences of different data sources and improve the fusion effect of multimodal feature vectors.

[0105] Among them, temporal location encoding is used to indicate which frame (or keyframe) the current feature belongs to, so that the feature fusion model can understand the sequence of frames and model the dynamic changes in road conditions.

[0106] Step S202: Based on the multimodal feature vectors of all keyframes of the video segment, construct the feature vector sequence of the video segment.

[0107] For example, the multimodal feature vectors of all keyframes sampled from a video segment are used to form a feature vector sequence, which can be represented as {x1, x2, ..., x...} t , ..., x T}; where T represents the total number of keyframes.

[0108] Step S203: Input the feature vector sequence into the pre-trained feature fusion model to obtain the global features of the video segment.

[0109] Feature fusion models, in particular, are intelligent models used to fuse input data (such as sequences of feature vectors) and extract global features from them. Feature fusion models can be built based on deep learning models; for example, they can be built based on Transformer networks.

[0110] For example, the vehicle-side control device can input a sequence of feature vectors into a Transformer-based feature fusion model. Each layer of the Transformer-based feature fusion model contains a multi-head self-attention mechanism and a feed-forward network. Through its multi-head self-attention mechanism, the Transformer interacts with the multi-modal feature vectors of each keyframe in the feature vector sequence with the multi-modal feature vectors of all other keyframes, based on the modal embedding vectors and temporal position encodings implicit in the feature vector sequence, thus achieving deep information fusion and temporal modeling across keyframes. After Transformer encoding, the feature vector sequence is converted into an output representation containing rich contextual information, ultimately outputting a token vector h. cls It is used to aggregate global information from the entire feature vector sequence; therefore, h cls Essentially, it is a compressed global representation of the multimodal feature vectors of all keyframes after deep fusion and refinement.

[0111] In this embodiment, a multimodal feature vector is constructed by concatenating the visual feature vector, optical flow feature vector, and brightness feature vector of key frames. This fully integrates image semantics, inter-frame motion, and illumination distribution information, enhancing the multimodal feature vector's ability to represent complex road conditions. Furthermore, a feature sequence is constructed based on the multimodal feature vectors of all key frames, preserving the temporal dynamic information of the video clips. This sequence is then input into a pre-trained feature fusion model, enabling deep interaction and global aggregation of multimodal and temporal information. Ultimately, this yields a more robust and semantically complete global feature of the video clips, effectively improving the accuracy and scene generalization ability of vehicle road condition recognition.

[0112] In one embodiment, step S103 above, which involves extracting keyframes from a video segment based on the complexity of each video frame, includes the following: determining candidate probability information for each video frame based on its complexity; the candidate probability information is used to characterize the probability distribution of a video frame being selected as a keyframe; and performing differentiable sampling on the video frames in the video segment based on the candidate probability information of each video frame to obtain the keyframes in the video segment.

[0113] For example, the vehicle-side control device employs a differentiable sampling technique, such as Gumbel-Softmax (a reparameterization trick), to generate a probability distribution for each video frame to be selected as a keyframe, based on the complexity of that frame, thus obtaining candidate probability information for each video frame. This probability distribution is perturbed with Gumbel noise, and then an approximately discrete distribution is generated using the Softmax function. For example, the candidate probability information for video frame 1 is 0.95, for video frame 2 it is 0.01, for video frame 3 it is 0.94, and so on. Sampling is then performed based on the candidate probability information to obtain keyframes in the video segment; for example, video frames 1 and 3 are selected as keyframes.

[0114] It should be noted that the differentiable sampling technique ensures that the frame extraction strategy in this embodiment can be jointly trained end-to-end with the subsequent road condition classification model.

[0115] In practical applications, the formula for implementing differentiable abstraction using Gumbel-Softmax is as follows:

[0116]

[0117]

[0118] In the formula, This represents the candidate probability information for the t-th video frame; Let represent the complexity of the t-th video frame; This indicates that the t-th video frame is selected as a keyframe; Softmax() represents the Softmax function; Indicates Gumbel noise; This represents the temperature parameter, during training. In reasoning, Approaching 0.

[0119] In this embodiment, the probability distribution of each video frame being selected as a keyframe is dynamically calculated based on its complexity. This ensures that video frames with high complexity and rich information have a higher chance of being selected, guaranteeing that keyframes can accurately capture the core dynamics and key scenes in the video clip. Furthermore, through differentiable sampling processing, the discrete frame selection process is transformed into a continuous and differentiable mathematical operation. This allows the entire keyframe extraction process to be seamlessly embedded into the traffic condition classification model for joint optimization training. This not only ensures the rationality and representativeness of keyframe selection and effectively compresses the amount of video data, but also overcomes the limitation of non-differentiability in traditional keyframe extraction methods. It provides strong support for end-to-end training and global optimization of the model, significantly improving the performance and efficiency of the traffic condition classification task.

[0120] In one embodiment, step S102 above, which obtains the complexity of each video frame based on the optical flow feature vector and the luminance feature vector of each video frame, includes the following: performing vector concatenation processing on the optical flow feature vector and the luminance feature vector of each video frame to obtain the concatenated vector of each video frame; and inputting the concatenated vector into a pre-trained complexity evaluation model to obtain the complexity of each video frame.

[0121] Complexity can be expressed as a fraction. Complexity is positively correlated with the information content of a video frame; that is, the more information a video frame contains, the higher its complexity score.

[0122] For example, the vehicle-mounted control device performs vector concatenation on the optical flow feature vector and luminance feature vector of the video frame to obtain the concatenated vector of the video frame; then, the concatenated vector is input into a pre-trained complexity evaluation model to evaluate the complexity of the concatenated vector through the pre-trained complexity evaluation model, and outputs the complexity of the video frame.

[0123] In practical applications, the complexity of a video frame, obtained from the output of a pre-trained complexity evaluation model, can be expressed by the following formula:

[0124]

[0125] In the formula, Let represent the complexity of the t-th video frame; , , and The model parameters represent the complexity evaluation model; σ() represents the Sigmoid activation function, which has the mathematical form: σ(x)=1 / (1+e^(-x)). The Sigmoid activation function is used to map the input real scalar (such as concatenated vectors) to the interval (0, 1) to normalize the final linear output of the complexity evaluation model, resulting in a complexity score between 0 and 1. RELU() represents the linear rectified function; This indicates vector concatenation processing.

[0126] In practical applications, complexity The closer the value is to 1, the greater the information content of the video frame (e.g., the more complex the scene, the more varied the scenes, or the more critical the lighting conditions), and the more worthwhile the video frame is to be extracted for subsequent analysis; complexity The closer to 0, the less information the video frame contains (e.g., a static, simple, or uniformly lit scene), and the more likely the video frame is to be skipped.

[0127] In this embodiment, by concatenating the optical flow feature vector and the brightness feature vector of each video frame, a deep fusion of motion information and illumination information is achieved, comprehensively integrating the intensity of motion and illumination change characteristics in dynamic scenes. Based on this, the concatenated vector is input into a pre-trained complexity evaluation model, making the complexity evaluation results closer to the visual perception requirements of real driving environments. Furthermore, end-to-end model inference improves the evaluation efficiency and accuracy.

[0128] In one embodiment, the pre-trained complexity evaluation model, differentiable sampling processing, pre-trained feature fusion model, and pre-trained traffic classification model can be trained as follows: Annotate sample video segments to obtain traffic labels for the sample video segments; input the global features of the sample video segments into the traffic classification model to be trained to obtain the sample classification result of the sample video segments; obtain the classification loss value based on the sample classification result and traffic labels; obtain the sparse loss value based on the keyframes of the sample video segments; and perform a collaborative update process on the complexity evaluation model, differentiable sampling processing, feature fusion model, and traffic classification model to be trained based on the classification loss value and the sparse loss value to obtain the pre-trained complexity evaluation model, differentiable sampling processing, pre-trained feature fusion model, and pre-trained traffic classification model.

[0129] The sample classification result refers to the road condition category predicted for the sample video clip.

[0130] The classification loss value is used to measure the degree of difference between the sample classification result and the road condition label.

[0131] The sparse loss value is used to control the sparse intensity of the extracted keyframes.

[0132] For example, a large number of road condition video samples can be collected from the dashcam of the vehicle control device. These samples cover various weather conditions, lighting, and road environments. The road condition video samples are divided into video clips of fixed duration and target frame number. Each video clip can be manually labeled with a unique road condition tag. For example, road condition tags include, but are not limited to, Belgian roads, pothole roads, gravel roads, resonant roads, uphill roads, downhill roads, and figure-eight plazas. All video clips and their road condition tags are organized into a dataset, and the dataset is divided into training, validation, and test sets according to a preset ratio (e.g., 8:1:1) for subsequent model training, model inference, and model validation.

[0133] A feature fusion model is built based on the Transformer, and a traffic condition classification model is built based on the fully connected layer. The feature fusion model and the traffic condition classification model are deployed on a server with GPU (Graphics Processing Unit) acceleration capabilities to support end-to-end optimization of the feature fusion model and the traffic condition classification model.

[0134] Figure 4 This is a flowchart illustrating the end-to-end joint training model. Visual feature vectors, optical flow feature vectors, and brightness feature vectors are extracted from the sample video frames of each sample video segment. The optical flow and brightness feature vectors of each sample video frame are input into the complexity evaluation model to be trained, yielding the complexity of each sample video frame. Based on the complexity of each sample video frame, differentiable frame extraction technology is used to extract keyframes from the sample video segments, obtaining the keyframes within each segment. Multimodal feature vectors of the keyframes for each sample video segment are generated. These multimodal feature vectors are then input into the feature fusion model to be trained for cross-modal feature fusion processing, resulting in the global features of each sample video segment. Finally, the global features are input into the road condition classification model to be trained for road condition recognition, yielding the classification results for each sample video segment.

[0135] The cross-entropy is calculated based on the sample classification results and road condition labels, and set as the classification loss value. The sparse loss value is calculated based on the keyframes. The classification loss value and the sparse loss value are fused to obtain the target loss value. Based on the target loss value, the parameters of the complexity evaluation model to be trained, the differentiable frame extraction technique, the feature fusion model to be trained, and the road condition classification model to be trained are all iteratively updated to achieve collaborative optimization of all modules.

[0136] The formulas for calculating the classification loss, sparse loss, and target loss are as follows:

[0137]

[0138]

[0139]

[0140] In the formula, This represents the classification loss value; This represents the actual road condition label for the c-th category; This represents the classification result of the sample in the c-th category; Indicates the sparse loss value; Indicates the target loss value; This represents a hyperparameter used to control the sparsity intensity.

[0141] In practical applications, such as Figure 4 As shown, during the backpropagation process, gradient descent (or a variant thereof) can be used to collaboratively optimize all modules of the model to be trained: the complexity evaluation model, the differentiable frame extraction technique, the feature fusion model, and the road condition classification model. Alternatively, a phased training strategy can be adopted: the first phase acquires the sample visual feature vectors, sample optical flow feature vectors, and sample brightness feature vectors, training the remaining modules; the second phase fine-tunes all modules and the model end-to-end with a lower learning rate to further improve performance.

[0142] In this embodiment, sample video segments are accurately labeled to obtain real road condition tags. The samples are then forward-propagated through complexity assessment, differentiable sampling, feature fusion, and road condition classification modules to obtain classification results. A comprehensive loss function, combining classification loss and sparse loss based on keyframes, is then derived to balance classification accuracy and feature selection rationality. Finally, the target loss value calculated using this comprehensive loss function is used to collaboratively update the parameters of the complexity assessment model, differentiable sampling module, feature fusion model, and road condition classification model. This not only ensures performance optimization for each module in its respective task but also achieves deep coupling and joint tuning between modules through gradient backpropagation. The sparse loss guides the complexity assessment model to focus on information-rich frames, while the classification loss drives the feature fusion and classification models to extract more discriminative global representations. The resulting model achieves global optimization in keyframe selection rationality, feature representation discriminability, and final classification accuracy, significantly improving generalization ability and robustness under complex road conditions.

[0143] In one embodiment, in step S101 above, the optical flow feature vector of each video frame in the video segment is obtained as follows: based on the motion vector field of each video frame, the motion amplitude of each pixel in each video frame is obtained; based on the motion amplitude of each pixel in each video frame, the first optical flow statistical feature of each video frame is obtained; based on the first optical flow statistical feature and the motion amplitude of each video frame, the second optical flow statistical feature of each video frame is obtained; based on the first optical flow statistical feature and the second optical flow statistical feature of each video frame, the optical flow feature vector of each video frame is obtained.

[0144] Among them, the first and second optical flow statistical features can be regarded as statistical features for the optical flow amplitude.

[0145] Among them, the optical flow feature vector encodes the motion intensity and motion consistency information between frames.

[0146] For example, the vehicle-mounted control device can calculate the motion vector field between consecutive video frames using an optical flow estimation algorithm; wherein, the motion vector field includes the motion vector of a pixel in the x-axis direction and the motion vector in the y-axis direction; the motion vector of a pixel in the x-axis direction is used to quantify the magnitude of the pixel's displacement in the horizontal direction (usually the horizontal to the right is positive) from the current frame to the next frame (the displacement unit can be pixels, for example, a horizontal displacement of 2 pixels to the right); the motion vector of a pixel in the y-axis direction is used to quantify the magnitude of the pixel's displacement in the vertical direction (usually the vertical downward direction is positive) from the current frame to the next frame (the displacement unit can be pixels, for example, a vertical displacement of 3 pixels downward).

[0147] As shown in the formula below, the motion amplitude of each pixel in a video frame can be calculated based on the motion vector field:

[0148]

[0149] In the formula, This represents the motion amplitude of the pixel with pixel coordinates (i, j) in a video frame. Represents the horizontal component of the optical flow of pixel (i, j), which is the motion vector of pixel (i, j) in the x-axis direction; This represents the vertical component of the optical flow of pixel (i, j), which is the motion vector of pixel (i, j) in the y-axis direction.

[0150] Then, based on the motion amplitude of each pixel in the video frame, the first optical flow statistical feature of the video frame is calculated. The calculation process of the first optical flow statistical feature is as follows:

[0151]

[0152] In the formula, The first optical flow statistical characteristic is represented by W; the width of the video frame is represented by H; and the height of the video frame is represented by H.

[0153] The vehicle-side control equipment can further calculate the second optical flow statistical characteristics of each video frame based on the first optical flow statistical characteristics and motion amplitude of each video frame; the second optical flow statistical characteristics ( The calculation process for ) is shown below;

[0154]

[0155] The first and second optical flow statistical features of each video frame are converted into two-dimensional vectors, thus obtaining the optical flow feature vector of each video frame. The optical flow feature vector of the t-th video frame can then be expressed as: .

[0156] In this embodiment, the motion amplitude of each pixel is first calculated based on the motion vector field to capture the motion intensity at the microscopic level; then, a first optical flow statistical feature is generated through statistical aggregation to characterize the overall motion distribution; next, the original motion amplitude and the first optical flow statistical feature are combined to extract a more discriminative second optical flow statistical feature, revealing the deep-seated patterns of motion between consecutive video frames; finally, the first and second optical flow statistical features are fused into an optical flow feature vector, improving the ability to understand and describe dynamic scenes in video frames.

[0157] In one embodiment, in step S101 above, the luminance feature vector of each video frame in the video segment is obtained by: obtaining the luminance entropy of each video frame based on the quantization level of the luminance component of each video frame; obtaining the luminance contrast of each video frame based on the luminance component of each video frame; and obtaining the luminance feature vector of each video frame based on the luminance entropy and luminance contrast of each video frame.

[0158] Among them, the luminance component refers to the luminance value of a single pixel in a video frame.

[0159] Among them, luminance entropy is used to characterize the uncertainty and disorder of the luminance distribution of video frames. The higher the value of luminance entropy, the more uniform and random the luminance distribution of the video frames; the lower the value of luminance entropy, the more concentrated and orderly the luminance distribution of the video frames.

[0160] In this context, luminance contrast ratio refers to the standard deviation of the luminance component. Luminance contrast ratio measures the dispersion of a pixel's luminance value relative to the average luminance value of the video frame. A higher luminance contrast ratio indicates a greater difference in brightness within the video frame, while a lower value indicates more uniform overall brightness.

[0161] For example, the vehicle-side control device extracts the luminance component from the video frame and then determines the quantization level of the luminance component. This can be done by discretizing the luminance value of the video frame (e.g., the pixel value range of a single-channel luminance image is 0-255) or treating it as a discrete random variable, and using k to represent a possible value of this random variable. For example, if the luminance value is quantized into 256 levels (0 to 255), then k is a variable that takes integer values ​​in the range [0, 1, 2, ..., 255], and k is the quantization level of the luminance component. In practical applications, if grayscale processing is used to convert the luminance value into a random variable, then k can also be regarded as the grayscale index of the pixel. The probability of occurrence of pixels in the video frame whose luminance value is equal to or approximately k is then calculated. For example, the probability of occurrence can be obtained by calculating the luminance histogram of the video frame and performing normalization processing. Using this probability of occurrence, the luminance entropy of the video frame is calculated. The formula for calculating the luminance entropy is as follows:

[0162]

[0163] In the formula, E bright p(k) represents the luminance entropy; p(k) represents the probability of a pixel in a video frame having a luminance value equal to or approximately equal to quantization level k.

[0164] The vehicle-mounted control equipment calculates the standard deviation of the luminance component of each video frame to obtain the luminance contrast ratio of each video frame; the formula for calculating the luminance contrast ratio is shown below:

[0165]

[0166] In the formula, C bright Indicates brightness contrast; L t (i, j) represents the luminance component of pixel (i, j) in the t-th video frame; This represents the average value of the global luminance component in a video frame. Used to reflect the overall brightness level or average light intensity of a video frame.

[0167] Finally, the luminance entropy and luminance contrast of each video frame are converted into two-dimensional vectors, thus obtaining the luminance feature vector of each video frame. The luminance feature vector of the t-th video frame can then be expressed as: .

[0168] In this embodiment, luminance entropy, based on the quantization level of luminance components, effectively characterizes the complexity and richness of illumination distribution in an image, reflecting the texture details and illumination levels of the scene. Luminance contrast, on the other hand, accurately measures the intensity of the difference between bright and dark areas, revealing the visual impact and clarity of the image. The luminance feature vector formed by fusing these two metrics takes into account both the statistical characteristics of illumination distribution and key dimensions of visual perception, significantly improving the accuracy and descriptive ability of perceiving illumination conditions in visual scenes. This provides more discriminative feature input for subsequent road condition recognition tasks. It is particularly suitable for driving scenarios with drastic changes in illumination, effectively supporting vehicle-side control equipment in accurately identifying and rapidly responding to complex illumination conditions such as tunnel entry / exit and nighttime driving.

[0169] In one embodiment, in step S101 above, the visual feature vector of each video frame in the video segment is obtained by performing scale scaling and normalization processing on each video frame in the video segment in sequence to obtain the processed video frame of each video frame; and performing global pooling processing on each processed video frame to obtain the visual feature vector of each video frame.

[0170] The video frames are transformed into high-dimensional visual feature vectors. For example, the vehicle-mounted control device scales each video frame of the video segment to obtain scaled video frames; it then normalizes each scaled video frame to obtain processed video frames; and finally, it inputs each processed video frame into a feature extractor, which performs global average pooling on each frame. This means averaging the two-dimensional feature map output by the feature extractor in the spatial dimension, compressing all pixel values ​​of each channel into a scalar, and ultimately outputting a one-dimensional feature vector of length C, i.e., a visual feature vector.

[0171] In practical applications, the feature extractor can be optimized based on MobileNetV3 (an efficient and lightweight convolutional neural network architecture). The original classification head (fully connected layer) of MobileNetV3 can be removed, retaining only its backbone network to extract rich two-dimensional feature maps from the input video frames (e.g., processed video frames). The process of using the feature extractor to perform global pooling on the processed video frames can be represented by the following formula:

[0172] f t =GlobalAveragePool(MobileNetV3(I t ))∈R 576

[0173] In the formula, GlobalAveragePool() represents the global average pooling operation; R represents the real number field; 576 represents the dimension of the visual feature vector; and f t ∈R 576 Represents the visual feature vector f t It is a 576-dimensional real vector.

[0174] The 576-dimensional feature map is determined by the MobileNetV3 backbone network in conjunction with the GlobalAveragePool operation. For example, the 2D feature map output by the MobileNetV3 backbone network has 576 channels. The GlobalAveragePool() operation averages these 576 channels of the 2D feature map into 576 scalars. Finally, each video frame is converted into a feature vector f of length 576. t .

[0175] In this embodiment, scaling each video frame in the video segment reduces the inconsistency in feature dimensions caused by differences in video size. Normalizing the video frames balances the pixel value distribution under different lighting and contrast conditions, enhancing the robustness and generalization ability of visual features. Global pooling of the video frames compresses high-dimensional image data into fixed-length visual feature vectors, preserving key semantic information of the video frames while significantly reducing data dimensionality, computational and storage overhead, and providing high-quality, low-redundancy feature input for subsequent video analysis, thus improving the efficiency and accuracy of vehicle road condition recognition.

[0176] In one embodiment, step S101, which divides the traffic video into video segments, includes the following: obtaining the target number of frames for each video segment based on a preset frame rate and a preset segment duration; and dividing the traffic video into multiple non-overlapping video segments containing the target number of video frames.

[0177] The preset frame rate refers to the number of frames per second in a video clip. The preset clip duration refers to the fixed duration of a single video clip; the unit of duration can be seconds.

[0178] For example, the vehicle-mounted control device calculates the target number of each video segment based on the product of the preset frame rate and the preset segment duration; then, according to the target number of frames, the video stream of the road condition video is divided into multiple non-overlapping video segments, each containing a fixed number of target video frames.

[0179] In practical applications, the target frame count can be calculated using the following formula:

[0180] N frames =round(fps*T)

[0181] In the formula, N frames The target frame rate is indicated by fps; the preset frame rate is indicated by fps; and the preset segment duration is indicated by T.

[0182] In this embodiment, the target number of frames for each video segment is calculated using a preset frame rate and a preset segment duration. The traffic video is then divided into multiple non-overlapping video segments of equal length, achieving standardized and structured processing of traffic data. This ensures the integrity and consistency of each video segment in the temporal dimension, providing a regular data unit for subsequent video analysis and feature extraction, effectively improving processing efficiency. At the same time, the non-overlapping partitioning strategy avoids redundant data storage and computation, optimizing storage space utilization and computational resource consumption, thus laying a solid data foundation for the analysis and processing of traffic videos.

[0183] In one embodiment, such as Figure 5 As shown, a road condition recognition method is provided. Taking the application of this method to vehicle-side control equipment as an example, the method includes the following steps:

[0184] Step S501: Based on the preset frame rate and preset segment duration, obtain the target number of frames for each video segment; divide the road condition video into multiple non-overlapping video segments containing the target number of video frames.

[0185] Step S502: Obtain the visual feature vector, optical flow feature vector, and luminance feature vector of each video frame in the video segment.

[0186] Step S503: Perform vector concatenation on the optical flow feature vector and luminance feature vector of each video frame to obtain the concatenated vector of each video frame; input the concatenated vector into the pre-trained complexity evaluation model to obtain the complexity of each video frame.

[0187] Step S504: Determine the candidate probability information for each video frame based on the complexity of each video frame in the video segment; the candidate probability information is used to characterize the probability distribution of a video frame being selected as a keyframe.

[0188] Step S505: Based on the candidate probability information of each video frame, perform differentiable sampling processing on the video frames in the video segment to obtain the key frames in the video segment.

[0189] Step S506: Perform vector concatenation on the visual feature vector, optical flow feature vector, and brightness feature vector of the key frame to obtain the multimodal feature vector of the key frame.

[0190] Step S507: Construct a sequence of feature vectors for the video segment based on the multimodal feature vectors of all keyframes of the video segment.

[0191] Step S508: Input the feature vector sequence into the pre-trained feature fusion model to obtain the global features of the video segment.

[0192] Step S509: Input the global features of the video clip into the pre-trained traffic condition classification model to obtain the traffic condition classification result of the video clip.

[0193] The aforementioned road condition recognition method achieves the following beneficial effects: it divides road condition videos into video segments and comprehensively extracts the visual feature vector, optical flow feature vector, and brightness feature vector of each video frame, realizing multi-dimensional representation of static content, dynamic motion, and lighting conditions; then, based on the optical flow and brightness feature vectors, it intelligently evaluates the complexity of each video frame and accurately selects the most representative key frames, preserving the core semantic information of the video segments while significantly reducing data redundancy; subsequently, it fuses the multi-dimensional feature vectors of all key frames to generate global features of the video segments, which are finally input into the road condition classification model to output highly reliable road condition classification results. By selecting key frames based on complexity, it not only effectively compresses the computational load and improves the processing speed of vehicle-side control equipment, but also ensures the comprehensiveness and accuracy of road condition classification results through multi-modal feature fusion. It is particularly suitable for complex and ever-changing driving scenarios, providing accurate and efficient road condition understanding support for vehicle-side applications such as intelligent driving assistance, traffic flow monitoring, and autonomous driving decision-making.

[0194] To more clearly illustrate the road condition recognition method provided in this disclosure, a specific embodiment will be used to describe the method below. For example... Figure 6 As shown, a road condition recognition method is provided, which can be applied to vehicle-side control equipment, and specifically includes the following:

[0195] 1. Input data

[0196] The system uses the vehicle's dashcam to obtain video sources of road conditions; it then segments and reads the video frame by frame to obtain video segments with the target number of frames (or a fixed duration).

[0197] 2. Feature Generation

[0198] By using a feature extractor based on MobileNetV3, visual feature vectors are extracted from each frame of the video clip. These visual feature vectors can represent visual semantic information such as road texture and smoothness. Optical flow feature vectors are extracted from each frame of the video clip through optical flow calculation. These optical flow feature vectors represent dynamic road condition information such as inter-frame motion, road bumps, and vehicle swaying. Brightness feature vectors are extracted from each frame of the video clip through brightness analysis. These brightness feature vectors represent light intensity, contrast, and the degree of distribution disorder, enhancing robustness under complex lighting conditions.

[0199] 3. Dynamic frame skipping

[0200] The optical flow feature vector and luminance feature vector of each video frame are input into a lightweight complexity evaluation model, which outputs the complexity of each video frame. Differentiable sampling techniques are used to generate candidate probability information for each video frame in a video segment based on the complexity, and one or more keyframes are extracted from the video segment.

[0201] 4. Multimodal fusion

[0202] The visual feature vector, optical flow feature vector, and brightness feature vector of the keyframe are concatenated, and modal embedding and temporal position encoding are added to distinguish modal type and temporal position, so as to obtain the multimodal feature vector of the keyframe. The multimodal feature vector is then input into the feature fusion model to output the global features of the video segment.

[0203] 5. Road condition recognition

[0204] The global features of the video clip are input into a pre-trained traffic condition classification model, which outputs the traffic condition category probability distribution of the video clip and calculates the confidence level of the traffic condition category probability distribution. Finally, the traffic condition category probability distribution and its confidence level can be set as the traffic condition classification result of the video clip.

[0205] In this embodiment, the following beneficial effects can be achieved: (1) A complexity-based adaptive frame extraction mechanism is proposed, which can dynamically control the frame sampling rate according to the content of the video frame, thereby achieving a balance between computational efficiency and recognition accuracy. (2) A pseudo-multimodal feature vector containing optical flow feature vector and brightness feature vector is constructed as input data to characterize motion changes and lighting conditions, thereby improving the robustness of the model under different road conditions. (3) A differentiable sampling method is used to achieve key frame selection, so that the frame extraction strategy can participate in end-to-end training and be jointly optimized with the road condition classification model. (4) The visual feature vector and the pseudo-multimodal feature vector are concatenated and input into a lightweight feature fusion model to achieve efficient fusion of multimodal information.

[0206] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0207] Based on the same inventive concept, this application also provides a road condition recognition device for implementing the road condition recognition method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more road condition recognition device embodiments provided below can be found in the limitations of the road condition recognition method described above, and will not be repeated here.

[0208] In one exemplary embodiment, such as Figure 7 As shown, a road condition recognition device 700 is provided, comprising:

[0209] The vector acquisition module 701 is used to divide the road condition video into video segments and acquire the visual feature vector, optical flow feature vector and brightness feature vector of each video frame in the video segment.

[0210] The complexity acquisition module 702 is used to obtain the complexity of each video frame based on the optical flow feature vector and the brightness feature vector of each video frame.

[0211] The keyframe extraction module 703 is used to extract keyframes from a video segment based on the complexity of each video frame in the video segment, thereby obtaining the keyframes in the video segment.

[0212] The feature fusion module 704 is used to obtain the global features of the video segment based on the visual feature vectors, optical flow feature vectors, and luminance feature vectors of all key frames of the video segment.

[0213] The road condition classification module 705 is used to input the global features of the video clip into the pre-trained road condition classification model to obtain the road condition classification result of the video clip.

[0214] In one embodiment, the feature fusion module 704 is further configured to perform vector concatenation processing on the visual feature vector, optical flow feature vector, and brightness feature vector of the key frame to obtain the multimodal feature vector of the key frame; construct the feature vector sequence of the video segment based on the multimodal feature vectors of all key frames of the video segment; and input the feature vector sequence into a pre-trained feature fusion model to obtain the global features of the video segment.

[0215] In one embodiment, the keyframe extraction module 703 is further configured to determine the candidate probability information of each video frame based on the complexity of each video frame in the video segment; the candidate probability information is used to characterize the probability distribution of a video frame being selected as a keyframe; and based on the candidate probability information of each video frame, perform differentiable sampling processing on the video frames in the video segment to obtain the keyframes in the video segment.

[0216] In one embodiment, the complexity acquisition module 702 is further configured to perform vector concatenation processing on the optical flow feature vector and luminance feature vector of each video frame to obtain the concatenated vector of each video frame; and input the concatenated vector into a pre-trained complexity evaluation model to obtain the complexity of each video frame.

[0217] In one embodiment, the traffic condition recognition device 700 further includes a collaborative training module, which is used to annotate sample video segments to obtain traffic condition labels for the sample video segments; input the global features of the sample video segments into the traffic condition classification model to be trained to obtain the sample classification result of the sample video segments; obtain a classification loss value based on the sample classification result and the traffic condition label; obtain a sparse loss value based on the keyframes of the sample video segments; and perform collaborative update processing on the complexity evaluation model, the differentiable sampling process, the feature fusion model, and the traffic condition classification model to be trained based on the classification loss value and the sparse loss value to obtain a pre-trained complexity evaluation model, a differentiable sampling process, a pre-trained feature fusion model, and a pre-trained traffic condition classification model.

[0218] In one embodiment, the vector acquisition module 701 is further configured to: obtain the motion amplitude of each pixel in each video frame based on the motion vector field of the video frame; obtain a first optical flow statistical feature of each video frame based on the motion amplitude of each pixel in each video frame; obtain a second optical flow statistical feature of each video frame based on the first optical flow statistical feature and the motion amplitude; and obtain an optical flow feature vector of each video frame based on the first optical flow statistical feature and the second optical flow statistical feature.

[0219] In one embodiment, the vector acquisition module 701 is further configured to obtain the luminance entropy of each video frame based on the quantization level of the luminance component of each video frame; obtain the luminance contrast of each video frame based on the luminance component of each video frame; and obtain the luminance feature vector of each video frame based on the luminance entropy and luminance contrast of each video frame.

[0220] In one embodiment, the vector acquisition module 701 is further configured to perform scale scaling and normalization processing on each video frame in the video segment in sequence to obtain the processed video frame of each video frame; and to perform global pooling processing on each processed video frame to obtain the visual feature vector of each video frame.

[0221] In one embodiment, the traffic condition recognition device 700 further includes a video segmentation module, which is used to obtain the target number of each video segment according to a preset frame rate and a preset segment duration; and to divide the traffic condition video into multiple non-overlapping video segments containing the target number of video frames.

[0222] Each module in the aforementioned road condition recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0223] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a road condition recognition method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0224] Those skilled in the art will understand that Figure 8 The structure shown is a block diagram of a partial structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0225] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0226] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0227] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0228] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0229] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program mentioned can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0230] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0231] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A road condition recognition method, characterized in that, The method includes: The road condition video is divided into video segments, and the visual feature vector, optical flow feature vector, and luminance feature vector of each video frame in the video segment are obtained. The complexity of each video frame is obtained based on the optical flow feature vector and luminance feature vector of each video frame; Based on the complexity of each video frame in the video segment, keyframe extraction processing is performed on the video segment to obtain the keyframes in the video segment; The global features of the video segment are obtained based on the visual feature vectors, optical flow feature vectors, and luminance feature vectors of all the keyframes of the video segment. The global features of the video clip are input into a pre-trained traffic condition classification model to obtain the traffic condition classification result of the video clip.

2. The method according to claim 1, characterized in that, The step of obtaining the global features of the video segment based on the visual feature vectors, optical flow feature vectors, and luminance feature vectors of all keyframes of the video segment includes: The visual feature vector, optical flow feature vector, and brightness feature vector of the key frame are concatenated to obtain the multimodal feature vector of the key frame. Based on the multimodal feature vectors of all the keyframes of the video segment, a feature vector sequence of the video segment is constructed; The feature vector sequence is input into a pre-trained feature fusion model to obtain the global features of the video segment.

3. The method according to claim 1, characterized in that, The step of extracting keyframes from the video segment based on the complexity of each video frame in the video segment, to obtain the keyframes in the video segment, includes: Based on the complexity of each video frame in the video segment, candidate probability information for each video frame is determined; the candidate probability information is used to characterize the probability distribution of the video frame being selected as the keyframe. Based on the candidate probability information of each video frame, differentiable sampling processing is performed on the video frames in the video segment to obtain the keyframes in the video segment.

4. The method according to claim 1, characterized in that, The process of obtaining the complexity of each video frame based on the optical flow feature vector and luminance feature vector of each video frame includes: The optical flow feature vector and the luminance feature vector of each video frame are concatenated to obtain the concatenated vector of each video frame. The concatenated vector is input into a pre-trained complexity evaluation model to obtain the complexity of each video frame.

5. The method according to any one of claims 2 to 4, characterized in that, The pre-trained complexity evaluation model, the differentiable sampling process, the pre-trained feature fusion model, and the pre-trained road condition classification model are trained in the following manner: The sample video segments are labeled to obtain the road condition tags for the sample video segments; The global features of the sample video clip are input into the road condition classification model to be trained to obtain the sample classification result of the sample video clip; Based on the sample classification results and the road condition labels, the classification loss value is obtained; Based on the keyframes of the sample video segment, a sparse loss value is obtained; Based on the classification loss value and the sparse loss value, the complexity evaluation model to be trained, the differentiable sampling process to be trained, the feature fusion model to be trained, and the road condition classification model to be trained are collaboratively updated to obtain the pre-trained complexity evaluation model, the differentiable sampling process, the pre-trained feature fusion model, and the pre-trained road condition classification model.

6. The method according to claim 1, characterized in that, The optical flow feature vector of each video frame in the video segment is obtained in the following way: Based on the motion vector field of the video frame, the motion amplitude of each pixel in each video frame is obtained; Based on the motion amplitude of each pixel in each video frame, the first optical flow statistical feature of each video frame is obtained; Based on the first optical flow statistical features and motion amplitude of each video frame, the second optical flow statistical features of each video frame are obtained; The optical flow feature vector of each video frame is obtained based on the first optical flow statistical feature and the second optical flow statistical feature of each video frame.

7. The method according to claim 1, characterized in that, The luminance feature vector of each video frame in the video segment is obtained in the following way: The luminance entropy of each video frame is obtained based on the quantization level of the luminance component of each video frame. The brightness contrast of each video frame is obtained based on the brightness component of each video frame. The luminance feature vector of each video frame is obtained based on the luminance entropy and luminance contrast of each video frame.

8. The method according to claim 1, characterized in that, The visual feature vector of each video frame in the video segment is obtained in the following way: Each video frame in the video segment is sequentially scaled and normalized to obtain the processed video frame for each video frame. Global pooling is performed on each of the processed video frames to obtain the visual feature vector of each video frame.

9. The method according to claim 1, characterized in that, The process of dividing road condition video into video segments includes: The target number of frames for each video segment is obtained based on the preset frame rate and preset segment duration; The road condition video is divided into multiple non-overlapping video segments, each containing several video frames of the target frame.

10. A road condition recognition device, characterized in that, The device includes: The vector acquisition module is used to divide the road condition video into video segments and acquire the visual feature vector, optical flow feature vector and brightness feature vector of each video frame in the video segment; The complexity acquisition module is used to obtain the complexity of each video frame based on the optical flow feature vector and the luminance feature vector of each video frame; The keyframe extraction module is used to perform keyframe extraction processing on the video segment according to the complexity of each video frame in the video segment, so as to obtain the keyframes in the video segment. The feature fusion module is used to obtain the global features of the video segment based on the visual feature vectors, optical flow feature vectors, and luminance feature vectors of all the key frames of the video segment; The traffic condition classification module is used to input the global features of the video clip into a pre-trained traffic condition classification model to obtain the traffic condition classification result of the video clip.

11. A vehicle-end control device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images