Roadside digital video monitoring method and system based on deep learning

Abstract

The present invention discloses a roadside digital video monitoring method and system based on deep learning, comprising: first acquiring and analyzing roadside monitoring images to identify abnormal vehicle behavior. Once an abnormal classification result is found, the corresponding target vehicle image data is determined, and the face data of the target driving object is extracted therefrom. These data are then input into a pre-trained driving fatigue detection model to determine the fatigue state of the driving object, thereby obtaining monitoring results. With such a design, automatic analysis of traffic monitoring images is realized, the accuracy and efficiency of abnormal behavior detection are improved, and strong support is provided for intelligent traffic management.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a roadside digital video monitoring method and system based on deep learning. Background Art

[0002] With the development of intelligent transportation systems, roadside digital video surveillance plays an increasingly important role in urban traffic management, and the monitoring and analysis of abnormal vehicle behavior has become crucial. Traditional video surveillance systems can only provide simple tracking of vehicle movements, but cannot effectively identify abnormal vehicle behavior and related driver status. Summary of the invention

[0003] The purpose of the present invention is to provide a roadside digital video monitoring method and system based on deep learning.

[0004] In a first aspect, an embodiment of the present invention provides a roadside digital video monitoring method based on deep learning, characterized in that it includes:

[0005] Acquire a roadside monitoring image to be analyzed and a vehicle abnormal behavior classification indication corresponding to the roadside monitoring image, and analyze the roadside monitoring image and the vehicle abnormal behavior classification indication to obtain a classification result of the vehicle abnormal behavior classification indication;

[0006] When the classification result is characterized as an abnormal classification result, determining the target vehicle image data corresponding to the abnormal classification result;

[0007] Target facial data of a target driving object included in the target vehicle image data is obtained, and the target facial data is input into a pre-trained driving object fatigue driving detection model to obtain a driving object fatigue state of the target driving object as a monitoring result.

[0008] In a second aspect, an embodiment of the present invention provides a server system, including a server, wherein the server is used to execute the method described in the first aspect.

[0009] Compared with the prior art, the beneficial effects provided by the present invention include: using a roadside digital video monitoring method and system based on deep learning disclosed by the present invention, by acquiring and analyzing roadside monitoring images, abnormal vehicle behavior is identified. Once an abnormal classification result is found, the corresponding target vehicle image data is determined, and the face data of the target driving object is extracted therefrom. These data are then input into a pre-trained driving fatigue detection model to determine the fatigue state of the driving object, thereby obtaining monitoring results. Such a design realizes the automatic analysis of traffic monitoring images, improves the accuracy and efficiency of abnormal behavior detection, and provides strong support for intelligent traffic management. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other strongly related drawings can also be obtained based on these drawings without creative work.

[0011] Figure 1 A schematic diagram of the steps of a roadside digital video monitoring method based on deep learning provided by an embodiment of the present invention;

[0012] Figure 2 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0014] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0015] In order to solve the technical problems in the aforementioned background technology, Figure 1 A schematic flow chart of a roadside digital video surveillance method based on deep learning provided in an embodiment of the present disclosure is provided below. The roadside digital video surveillance method based on deep learning is introduced in detail.

[0016] Step S201, obtaining a roadside monitoring image to be analyzed and a vehicle abnormal behavior classification indication corresponding to the roadside monitoring image, and analyzing the roadside monitoring image and the vehicle abnormal behavior classification indication to obtain a classification result of the vehicle abnormal behavior classification indication;

[0017] Step S202, when the classification result is characterized as an abnormal classification result, determining the target vehicle image data corresponding to the abnormal classification result;

[0018] Step S203, obtaining target facial data of the target driving object included in the target vehicle image data, and inputting the target facial data into a pre-trained driving object fatigue driving detection model to obtain the driving object fatigue state of the target driving object as a monitoring result.

[0019] In an embodiment of the present invention, illustratively, the server receives real-time monitoring images from a digital camera installed on the roadside. These images cover multiple key areas of road traffic to ensure that all vehicles in motion can be captured. At the same time, the server also receives vehicle abnormal behavior classification instructions related to these images, which may include multiple possible abnormal situations such as vehicle reversing, illegal lane change, speeding, etc. For example, at the main road intersection of a certain city, a digital camera captured an image of a car changing lanes quickly and transmitted this image to the server in real time. After receiving the image, the server analyzes the image according to the preset vehicle abnormal behavior classification instructions. In this example, the server recognizes that the vehicle's lane change behavior did not turn on the turn signal, which is inconsistent with traffic regulations, so it is classified as abnormal behavior. After identifying the abnormal behavior, the server will further process these abnormal classification results to determine the specific target vehicle image data. This includes extracting the vehicle image involving abnormal behavior from the original monitoring image for more detailed analysis. After determining that the vehicle has abnormal lane change behavior, the server will intercept the specific image data of the vehicle from the monitoring image. These data include the appearance of the vehicle, the license plate number, and a clear picture of the lane change moment. The server will then use image recognition technology to extract the facial data of the driver from the target vehicle image data. This data will be input into a pre-trained driver fatigue detection model to determine whether the driver is in a fatigued state. In the above example, the server successfully extracted the facial data of the driver from the target vehicle image. Then, this data was input into the fatigue detection model. After analysis, the model outputs a result indicating the degree of fatigue of the driver. If the result shows that the driver is indeed in a fatigued state, the server will record this information and may use it as one of the evidences of traffic violations. Through deep learning technology, the server can automatically process a large number of roadside monitoring images, accurately identify abnormal vehicle behavior, and further determine whether the driver is driving fatigued, thereby improving road traffic safety.

[0020] In the embodiment of the present invention, the aforementioned step S201 can be implemented through the following example execution.

[0021] Acquire a roadside surveillance image to be analyzed and a vehicle abnormal behavior classification indication corresponding to the roadside surveillance image;

[0022] Performing a feature extraction operation on the roadside monitoring image to obtain a first roadside monitoring image feature that is weakly associated with the vehicle abnormal behavior classification indication and a second roadside monitoring image feature that is strongly associated with the vehicle abnormal behavior classification indication;

[0023] Acquire a first semantic description feature that is strongly associated with the vehicle abnormal behavior classification indication and a second semantic description feature that is weakly associated with the vehicle abnormal behavior classification indication;

[0024] The classification result of the abnormal behavior classification indication of the vehicle is inferred based on the feature matching degree between the roadside surveillance image feature and the semantic description feature, and the feature matching degree includes the matching degree between the first roadside surveillance image feature and the first semantic description feature and the matching degree between the second roadside surveillance image feature and the second semantic description feature.

[0025] In an embodiment of the present invention, illustratively, the server receives real-time surveillance image data streams from roadside cameras at various key intersections in the city. At the same time, it also obtains a series of vehicle abnormal behavior classification indicators corresponding to these surveillance images. These classification indicators are predefined and are used to guide the server on how to identify and analyze possible vehicle abnormal behaviors. For example, at a busy intersection, the server receives a surveillance image from a high-definition camera, which contains multiple moving cars. At the same time, the server also loads vehicle abnormal behavior classification indicators related to this intersection, such as illegal lane changes, running red lights, speeding, etc. The server will then extract features from the received roadside surveillance images. In this process, the server will use deep learning algorithms, such as convolutional neural networks (CNNs), to extract features from the images. These features are divided into two categories: first roadside surveillance image features that are weakly associated with vehicle abnormal behavior classification indicators, and second roadside surveillance image features that are strongly associated. In the above example, the first roadside surveillance image features that can be extracted by the server include basic information such as the color and model of the vehicle, which are not closely related to the specific abnormal behavior, but are helpful for subsequent analysis. The strongly associated second roadside surveillance image features may include the vehicle's speed, driving trajectory, whether there is a sudden lane change or parking, and other behavioral features. In addition to extracting features from the surveillance image, the server also obtains corresponding semantic description features from the vehicle abnormal behavior classification indication. These semantic description features are text descriptions of abnormal behaviors, and are also divided into two categories: strong association and weak association with abnormal behaviors. For example, for the abnormal behavior of "illegal lane change", the strongly associated semantic description features may include "no turn signal", "crossing multiple lanes", etc., while the weakly associated semantic description features may include more general descriptions such as "the vehicle is driving". Finally, the server will use the extracted roadside surveillance image features and semantic description features for matching analysis. By calculating the matching degree between the first roadside surveillance image feature and the first semantic description feature, and the matching degree between the second roadside surveillance image feature and the second semantic description feature, the server can infer whether the vehicle in the surveillance image has abnormal behavior and determine the specific type of abnormal behavior. In the above example, if the server finds that the vehicle's driving trajectory features extracted from the surveillance image are highly matched with the semantic description features of "illegal lane change" (such as the vehicle suddenly changes lanes without turning on the turn signal), then it will infer that the vehicle has abnormal behavior of "illegal lane change" and record this result for subsequent processing and analysis.

[0026] In the embodiment of the present invention, the classification result of the abnormal behavior classification indication of the vehicle is inferred according to the degree of feature matching between the roadside monitoring image features and the semantic description features, which can be implemented through the following examples.

[0027] Comparing the first roadside monitoring image feature with the first semantic description feature to obtain a first feature matching degree;

[0028] Comparing the second roadside monitoring image feature with the second semantic description feature to obtain a second feature matching degree;

[0029] Performing a linear superposition operation on the first feature matching degree and the second feature matching degree according to a preset importance coefficient to obtain an inferred confidence level of the vehicle abnormal behavior classification indication;

[0030] A classification result of the vehicle abnormal behavior classification indication is determined according to the inference confidence.

[0031] In the embodiment of the present invention, illustratively, after receiving the roadside monitoring image, the server will first extract the first roadside monitoring image features that are weakly associated with the classification indication of abnormal vehicle behavior, such as the color and model of the vehicle. At the same time, the server will also obtain the first semantic description features corresponding to these features, which are usually broad and do not involve specific abnormal behavior details. For example, the server can extract the features of a white car from the monitoring image and compare them with the semantic description feature of "white vehicle". By comparing the similarity between the two, the server can obtain the first feature matching degree, which reflects the degree of conformity between the vehicle in the monitoring image and the vehicle in the semantic description in terms of general attributes. Next, the server will extract the second roadside monitoring image features that are strongly associated with the classification indication of abnormal vehicle behavior, such as the speed and driving trajectory of the vehicle. At the same time, the server will obtain the second semantic description features corresponding to these features, which are more specific and directly associated with possible abnormal behavior. For example, if the server extracts the driving trajectory feature of a car suddenly changing lanes from the monitoring image, it will compare it with the specific semantic description feature of "suddenly changing lanes without turning on the turn signal". By comparing the similarity between the two, the server can obtain the second feature matching degree, which reflects the degree of conformity between the vehicle behavior in the monitoring image and the semantic description of the specific abnormal behavior. After obtaining the first feature matching degree and the second feature matching degree, the server will perform a linear superposition operation on the two matching degrees according to the preset importance coefficient. This importance coefficient reflects the importance of different types of features in inferring abnormal behavior. For example, the server may believe that the second feature matching degree directly related to the abnormal behavior is more important, so it will assign it a higher importance coefficient. Through the linear superposition operation, the server can obtain a comprehensive inference confidence, which reflects the possibility that the vehicle in the monitoring image has abnormal behavior. Finally, the server will determine the classification result of the vehicle's abnormal behavior based on the inference confidence. If the inference confidence exceeds the preset threshold, the server will consider that the vehicle in the monitoring image has abnormal behavior and classify it into the corresponding abnormal behavior type. For example, if the inference confidence calculated by the server exceeds 0.8 (assuming 1 is a complete match), then it will consider that the vehicle in the monitoring image has an abnormal behavior of "suddenly changing lanes without turning on the turn signal" and record this result. In this way, through deep learning technology and feature matching analysis, the server can automatically identify and process abnormal vehicle behavior in roadside surveillance images.

[0032] In an embodiment of the present invention, the classification result includes a vehicle behavior feature domain composed of multiple inferred vehicle behavior types, and the inference confidence includes a vehicle behavior type confidence corresponding to each of the inferred vehicle behavior types; the classification result of the vehicle abnormal behavior classification indication is determined based on the inference confidence, which can be implemented through the following example.

[0033] Normalizing each vehicle behavior type confidence in the inferred confidence to obtain a normalized score of each inferred vehicle behavior type in the vehicle behavior feature domain;

[0034] The inferred vehicle behavior type with the highest normalized score is used as the classification result of the vehicle abnormal behavior classification indication.

[0035] In an embodiment of the present invention, exemplarily, after the server calculates the vehicle behavior type confidence of each inferred vehicle behavior type according to the feature matching degree between the roadside monitoring image feature and the semantic description feature, in order to be able to compare the confidence of different behavior types on a unified scale, the server will normalize the confidence of each vehicle behavior type. For example, the server can infer that there are three possible behavior types of vehicles in a certain monitoring image: illegal lane change, speeding and running a red light, and calculate the confidence for these three behavior types respectively, such as the confidence of illegal lane change is 0.8, the confidence of speeding is 0.6, and the confidence of running a red light is 0.5. In order to eliminate the influence of the dimensions and numerical ranges between different confidences, the server will normalize these confidences and convert them into values ​​between 0 and 1, that is, normalized scores. After obtaining the normalized scores of each inferred vehicle behavior type, the server will compare these scores to find the behavior type with the highest score. This behavior type with the highest score is considered to be the most likely abnormal behavior of the vehicle in the monitoring image. Assuming that after normalization, the normalized score of illegal lane change is the highest, the server will determine the illegal lane change as the abnormal behavior classification result of the vehicle in the monitoring image. Finally, the server will output this classification result, which can be recorded in a log or sent to the relevant traffic management system for further processing. This classification result not only includes the type of abnormal vehicle behavior (such as illegal lane change), but also includes the confidence level or other relevant information of the type, so as to facilitate subsequent analysis and processing. In this way, the server can accurately identify the abnormal behavior type of the vehicle in the roadside monitoring image and provide strong data support for traffic management and safety monitoring.

[0036] In an embodiment of the present invention, a feature extraction operation is performed on the roadside surveillance image to obtain a first roadside surveillance image feature weakly associated with the vehicle abnormal behavior classification indication and a second roadside surveillance image feature strongly associated with the vehicle abnormal behavior classification indication, which can be implemented through the following example.

[0037] Acquire a roadside monitoring image feature extraction network that is weakly associated with the vehicle abnormal behavior classification indication and a roadside monitoring image feature mapping network that is strongly associated with the vehicle abnormal behavior classification indication, wherein the roadside monitoring image feature extraction network is used to extract roadside monitoring image features, and the roadside monitoring image feature mapping network is a neural network obtained through pre-training based on the vehicle abnormal behavior classification indication;

[0038] Performing a feature extraction operation on the roadside monitoring image through the roadside monitoring image feature extraction network to obtain a first roadside monitoring image feature that is weakly associated with the vehicle abnormal behavior classification indication;

[0039] The first roadside monitoring image feature is feature mapped through the roadside monitoring image feature mapping network to obtain a second roadside monitoring image feature that is strongly associated with the vehicle abnormal behavior classification indication.

[0040] In an embodiment of the present invention, exemplarily, the server first obtains two neural network models: one is a roadside monitoring image feature extraction network that is weakly associated with the vehicle abnormal behavior classification indication, and the other is a roadside monitoring image feature mapping network that is strongly associated with the vehicle abnormal behavior classification indication. The former is used to extract general features from the roadside monitoring image, while the latter is obtained through prior training based on specific vehicle abnormal behavior classification indications, and is used to map general features to features that are strongly associated with specific abnormal behaviors. For example, the server can load pre-trained roadside monitoring image feature extraction networks and feature mapping networks from its repository. These networks can be built based on deep learning frameworks (such as TensorFlow or PyTorch). Next, the server will use the roadside monitoring image feature extraction network to perform feature extraction operations on the input roadside monitoring image. In this process, the network will identify basic elements such as vehicles, roads, traffic signs, etc. in the image, and extract features related to these elements, which are weakly associated with specific vehicle abnormal behavior classification indications. For example, the server can input a section of roadside surveillance image into the feature extraction network, and the network will output a series of feature vectors that describe general features such as color, shape, size, etc. of the vehicle in the image, but do not involve specific abnormal behaviors. Finally, the server will use the roadside surveillance image feature mapping network to further process the first roadside surveillance image features extracted in the first step. This network will map the first roadside surveillance image features to the second roadside surveillance image features that are strongly associated with specific abnormal behaviors based on the vehicle abnormal behavior classification instructions learned during previous training. The server will input the feature vectors extracted in the first step into the feature mapping network. The network will map these general features to features that are strongly associated with specific abnormal behaviors (such as illegal lane changes, running red lights, etc.) based on its previous knowledge obtained through training with a large amount of abnormal behavior data. In this way, the server obtains the second roadside surveillance image features that contain both general vehicle information and specific abnormal behaviors. Through these steps, the server can effectively extract features related to abnormal vehicle behaviors from roadside surveillance images, providing strong data support for subsequent behavior classification and recognition.

[0041] In the embodiment of the present invention, obtaining a first semantic description feature that is strongly associated with the vehicle abnormal behavior classification indication and a second semantic description feature that is weakly associated with the vehicle abnormal behavior classification indication may be implemented through the following example.

[0042] Acquire a semantic description feature extraction network weakly associated with the vehicle abnormal behavior classification indication and a feature representation strongly associated with the vehicle abnormal behavior classification indication, wherein the semantic description feature extraction network is used to extract semantic description features, and the feature representation is a feature vector obtained through prior training based on the vehicle abnormal behavior classification indication;

[0043] Performing a feature extraction operation on the feature representation through the semantic description feature extraction network to obtain a first semantic description feature that is strongly associated with the vehicle abnormal behavior classification indication;

[0044] The semantic description feature extraction network performs a feature extraction operation on the preset reference description to obtain a second semantic description feature that is weakly associated with the vehicle abnormal behavior classification indication.

[0045] In an embodiment of the present invention, illustratively, the server first obtains two key components: one is a semantic description feature extraction network that is weakly associated with the classification indication of abnormal vehicle behavior, and the other is a feature representation that is strongly associated with abnormal behavior of a specific vehicle. This feature representation is a feature vector obtained after prior training for specific abnormal behaviors (such as illegal lane change, speeding, etc.). For example, the server can load a pre-trained semantic description feature extraction network that can extract semantic features of text descriptions. At the same time, the server will also load a set of feature vectors that are strongly associated with the abnormal behavior of "illegal lane change". Next, the server will use the semantic description feature extraction network to perform feature extraction operations on the previously acquired feature representation. Since the feature representation is already strongly associated with the specific abnormal behavior of the vehicle, the semantic description feature (i.e., the first semantic description feature) extracted through this step will also be closely related to the abnormal behavior. Taking "illegal lane change" as an example, the server inputs the feature vector strongly associated with this behavior into the semantic description feature extraction network, and the network will output a series of semantic description features that are closely related to the "illegal lane change" behavior, such as "no turn signal", "sudden lane change", etc. Finally, the server will use the same semantic description feature extraction network to extract features from the preset benchmark descriptions. These benchmark descriptions are usually some text descriptions related to vehicle behavior but relatively broad, and they are weakly associated with specific vehicle abnormal behavior classification indications. The semantic description features extracted through this step (i.e., the second semantic description features) will be used for comparison and matching with the first semantic description features. For example, the server can extract features from some benchmark descriptions such as "the vehicle is moving" and "the vehicle has changed lanes" to obtain semantic features related to these descriptions. Although these features have a certain correlation with "illegal lane change", they are not as close as the first semantic description features. Through these steps, the server can obtain two types of semantic description features that are strongly and weakly associated with the vehicle abnormal behavior classification indications, providing basic data for subsequent feature matching and abnormal behavior identification.

[0046] In the embodiment of the present invention, before inferring the classification result of the vehicle abnormal behavior classification indication, the following implementation is also provided.

[0047] Acquire a vehicle abnormal behavior classification model for executing the vehicle abnormal behavior classification indication, wherein the vehicle abnormal behavior classification model includes a roadside monitoring image feature extraction network, a roadside monitoring image feature mapping network, and a semantic description feature extraction network; the roadside monitoring image feature extraction network is used to extract roadside monitoring image features, the roadside monitoring image feature mapping network is used to perform feature mapping on the roadside monitoring image features, and the semantic description feature extraction network is used to extract semantic description features;

[0048] Acquire training instance data corresponding to the vehicle abnormal behavior classification model, wherein the training instance data includes a roadside monitoring image instance and a training instance target value bound to the roadside monitoring image instance;

[0049] Performing vehicle abnormal behavior classification processing on the roadside monitoring image instance by using the vehicle abnormal behavior classification model to obtain an inferred vehicle behavior type corresponding to the roadside monitoring image instance;

[0050] The adjustable parameters of the abnormal vehicle behavior classification model are optimized according to the inferred vehicle behavior type and the training instance target value.

[0051] In an embodiment of the present invention, exemplarily, the server first obtains a vehicle abnormal behavior classification model for executing vehicle abnormal behavior classification indication. This model is a composite model, including a roadside monitoring image feature extraction network, a roadside monitoring image feature mapping network, and a semantic description feature extraction network. These networks are used to extract features of roadside monitoring images, map the extracted features, and extract features of semantic descriptions, respectively. For example, the server can load a pre-built vehicle abnormal behavior classification model from a model library. This model can be a deep learning model, including multiple neural network layers, for processing and analyzing roadside monitoring images and related semantic descriptions. Next, the server will obtain training instance data corresponding to the vehicle abnormal behavior classification model. These data include multiple roadside monitoring image instances and training instance target values ​​bound to these image instances. The training instance target value represents the true classification of vehicle behavior in each image instance. For example, the server can load training instance data from a labeled data set. This data set can contain a large number of roadside monitoring images, each of which corresponds to one or more labels of abnormal vehicle behavior (such as illegal lane change, running a red light, etc.), and these labels are the training instance target values. The server uses the acquired vehicle abnormal behavior classification model to classify the roadside surveillance image instances. The model first extracts the features of the image through the roadside surveillance image feature extraction network, then maps these features through the roadside surveillance image feature mapping network, and finally extracts the semantic description features related to the vehicle behavior through the semantic description feature extraction network. Based on these features, the model infers the vehicle behavior type corresponding to each roadside surveillance image instance. For example, for a roadside surveillance image instance, the model infers that the vehicle in the image has the behavior of "illegal lane change". Finally, the server optimizes the adjustable parameters of the vehicle abnormal behavior classification model based on the vehicle behavior type inferred by the model and the target value of the training instance. This process is usually implemented through the back propagation algorithm, which reduces the difference between the model inference result and the true target value by adjusting the model parameters, thereby improving the accuracy of the model. If the vehicle behavior type inferred by the model is inconsistent with the target value of the training instance (for example, the model infers "illegal lane change", while the actual label is "normal driving"), the server will adjust the model parameters so that the model can make more accurate inferences the next time it encounters a similar situation. Through these steps, the server can continuously optimize and improve the vehicle abnormal behavior classification model to improve its accuracy and reliability in practical applications.

[0052] In an embodiment of the present invention, optimizing the adjustable parameters of the abnormal vehicle behavior classification model according to the inferred vehicle behavior type and the training instance target value may be implemented through the following example.

[0053] Determining a cost parameter of the vehicle abnormal behavior classification model according to the inferred vehicle behavior type and the training instance target value;

[0054] Returning the cost parameter in the vehicle abnormal behavior classification model to obtain a parameter optimization direction corresponding to an adjustable parameter of the vehicle abnormal behavior classification model;

[0055] The adjustable parameters of the vehicle abnormal behavior classification model are optimized according to the parameter optimization direction.

[0056] In an embodiment of the present invention, illustratively, the server first determines the cost parameter of the vehicle abnormal behavior classification model based on the inferred vehicle behavior type and the training instance target value. This cost parameter reflects the gap between the model inference result and the actual result, and is usually calculated using a loss function. For example, suppose the model infers that a certain vehicle behavior is "illegal lane change", but the training instance target value shows that the vehicle is actually "normal driving". The server uses a loss function, such as a cross entropy loss function, to calculate the difference between the two and obtain a specific cost parameter value. Next, the server will return this cost parameter in the vehicle abnormal behavior classification model. This means that the server will use the back propagation algorithm to adjust the adjustable parameters such as the weights and biases of each neuron in the model according to the cost parameter to obtain a parameter optimization direction. This direction indicates how to adjust the parameters of the model so as to reduce the cost parameter in the next inference, that is, to improve the accuracy of the inference. The server will use the back propagation algorithm to determine which neurons need to increase their weights and which need to be reduced, so that the model can more accurately infer "normal driving" instead of "illegal lane change" when encountering a similar situation next time. Finally, the server will actually optimize the adjustable parameters of the vehicle abnormal behavior classification model according to the parameter optimization direction. This usually involves using optimization algorithms (such as gradient descent) to update parameters such as weights and biases of the model. In the above example, the server will use optimization algorithms such as gradient descent to gradually adjust the parameters of the model in the direction of parameter optimization. Through multiple iterations and optimizations, the inference ability of the model will gradually improve, and it will eventually be able to more accurately identify the normal driving behavior of the vehicle and reduce the misjudgment of illegal lane changes. Through these steps, the server can continuously improve and perfect the vehicle abnormal behavior classification model, improve its accuracy and reliability in practical applications, and better serve fields such as intelligent transportation systems and autonomous driving.

[0057] In an embodiment of the present invention, the cost parameter includes a first cost parameter determined based on the training instance data of the current training cycle and a second cost parameter determined based on the training instance data of the previous training cycle, and the vehicle abnormal behavior classification model performs different vehicle abnormal behavior classification indications in the current training cycle and the previous training cycle.

[0058] In the embodiment of the present invention, illustratively, in this round of training cycle, the server will determine the first cost parameter based on the training instance data of this round. This usually involves using the model to predict the training instance of this round and compare it with the real label in the instance, so as to calculate the difference between the predicted value and the real value, that is, the first cost parameter. For example, suppose that in this round of training cycle, the server uses a batch of new roadside monitoring images as training instances, which include a variety of abnormal vehicle behaviors. The server predicts these images through the vehicle abnormal behavior classification model and obtains the inference results of the vehicle behavior in each image. Then, the server compares these inference results with the real label in the image, and calculates the prediction error using the loss function (such as mean square error or cross entropy loss), which is the first round of cost parameters. In addition to the data of this round of training cycle, the server will also consider the training instance data of the previous training cycle to determine the second cost parameter. This is to retain the previously learned knowledge and prevent the model from forgetting the old knowledge when training new data, which is the so-called "catastrophic forgetting". Based on the performance on the previous data, the server will re-use these previous data to test the model and obtain the prediction results. Then, these prediction results are compared with the true labels, and the prediction error is calculated using the loss function. This error is the second cost parameter. After determining the first cost parameter and the second cost parameter, the server will combine these two cost parameters to optimize the adjustable parameters of the vehicle abnormal behavior classification model. This is usually achieved by weighted summation, that is, assigning different weights to the two cost parameters, and then summing them to obtain a total cost parameter. Finally, the server will use an optimization algorithm (such as gradient descent) to minimize this total cost parameter to update the parameters of the model. For example, the server will assign weights of 0.6 and 0.4 to the first cost parameter and the second cost parameter, respectively. Then, the two cost parameters are weighted and summed to obtain a total cost parameter. Next, the server will use optimization algorithms such as gradient descent to minimize this total cost parameter and gradually adjust the parameters of the model to improve the overall performance of the model on new and old data. In this way, the server can ensure that the vehicle abnormal behavior classification model will not forget the previously learned knowledge while learning new data, thereby maintaining the stability and generalization ability of the model.

[0059] In the embodiment of the present invention, the second cost parameter is obtained in the following manner.

[0060] Non-specifically selecting a preset number of past training instances from the training instance data of the previous training cycle, and obtaining classification results of the past training instances in the previous training cycle;

[0061] Using the vehicle abnormal behavior classification model to perform vehicle abnormal behavior classification processing on the past training examples, and obtaining classification results of the past training examples in the current training cycle;

[0062] The second cost parameter is determined according to the classification results of the past training examples in the previous training cycle and the classification results of the past training examples in the current training cycle.

[0063] In an embodiment of the present invention, illustratively, the server first non-specifically selects a preset number of past training instances from the training instance data of the previous training cycle. These past training instances are samples that have been used for training and have obtained classification results. For example, assume that the server used 1000 roadside monitoring images as training instances in the previous training cycle. Now, in order to calculate the second cost parameter, the server randomly selects 200 of these 1000 images as past training instances. Next, the server obtains the classification results of these past training instances in the previous training cycle. These results are predicted and recorded by the model in the previous training cycle. The server obtains the classification results of these 200 past training instances in the previous training cycle from the previous training records, such as which ones are classified as "illegal lane change" and which ones are classified as "normal driving". Then, the server uses the current vehicle abnormal behavior classification model to perform vehicle abnormal behavior classification processing on these 200 past training instances again to obtain the classification results of these instances in this round of training cycle. For example, the server inputs the selected 200 past training instances into the current model, and the model outputs the classification results of each instance, such as "illegal parking" and "traveling against traffic". Finally, the server determines the second cost parameter based on the classification results of the past training instances in the previous training cycle and the classification results in the current training cycle. This is usually calculated by comparing the consistency of the two rounds of classification results. In the above example, the server compares the classification results of the 200 past training instances in the previous training cycle and the current training cycle. If the classification results of an instance remain consistent in the two cycles, it means that the model's prediction on this instance is stable; if the classification results are inconsistent, it means that the model's prediction on this instance has changed. The server counts the number of these inconsistent instances and calculates a cost parameter, namely the second cost parameter, which is used to measure the prediction stability of the model on past data. In this way, the server can evaluate the prediction performance of the model on old data after a new round of training, thereby ensuring that the model does not forget the previously learned knowledge during the continuous learning process, and improving the stability and generalization ability of the model.

[0064] In an embodiment of the present invention, the classification results of the past training instances in the current training cycle include the inferred confidence of the past training instances in the current training cycle, and the classification results of the past training instances in the previous training cycle include the inferred confidence of the past training instances in the previous training cycle and the inferred vehicle behavior type of the past training instances in the previous training cycle; determining the second cost parameter based on the classification results of the past training instances in the previous training cycle and the classification results of the past training instances in the current training cycle can be implemented through the following examples.

[0065] Determining a first deviation for characterizing an inferred deviation of a preceding training cycle based on an inferred vehicle behavior type of the past training instance in a preceding training cycle and a training instance target value of the past training instance;

[0066] Determine, according to the inference confidence of the past training instance in the previous training cycle and the inference confidence of the past training instance in the current training cycle, a second deviation for characterizing the confidence fluctuation between the previous training cycle and the current training cycle;

[0067] The second cost parameter is determined according to the first deviation and the second deviation.

[0068] In an embodiment of the present invention, illustratively, the server first focuses on the inferred vehicle behavior type and inference confidence of past training instances in the previous training cycle, as well as the inference confidence of these instances in the current training cycle. This information is a key indicator for evaluating the performance change of the model. For example, suppose the server has a past training instance, which is inferred by the model as an "illegal lane change" behavior in the previous training cycle, with an inference confidence of 0.85. In this round of training cycle, the inference confidence of the same instance becomes 0.70, but the behavior type is still inferred as "illegal lane change". Next, the server calculates the first deviation, i.e., the inference deviation, based on the inferred vehicle behavior type of the past training instance in the previous training cycle and the target value of the training instance. This deviation reflects the difference between the inference result of the model in the previous training cycle and the true target value. If the target value of the training instance shows that the vehicle behavior is actually "normal driving", the server will calculate the deviation between the "illegal lane change" inferred by the model and the target value "normal driving". This deviation can be quantified by preset evaluation indicators (such as accuracy, recall, etc.). Then, the server calculates the second deviation, i.e., the confidence fluctuation, based on the inference confidence of the past training instance in the previous training cycle and the inference confidence in the current training cycle. This deviation reflects the change in the inference confidence of the model for the same instance in different training cycles. In the above example, the server calculates the difference between the inference confidence of 0.85 in the previous training cycle and the inference confidence of 0.70 in the current training cycle to obtain the confidence fluctuation. Finally, the server determines the second cost parameter based on the first deviation and the second deviation. This cost parameter comprehensively considers the performance of the model in terms of inference accuracy and confidence stability. In the above example, the server performs a weighted sum or other combination of the inference deviation and confidence fluctuation to obtain a comprehensive second cost parameter. This cost parameter will be used to subsequently optimize the model parameters to improve the overall performance of the model on new and old data. In this way, the server can more comprehensively evaluate the performance changes of the model in different training cycles, thereby more effectively optimizing and updating the model.

[0069] In an embodiment of the present invention, determining a first deviation for characterizing an inferred deviation of a previous training cycle based on the inferred vehicle behavior type of the past training instance in a previous training cycle and a training instance target value of the past training instance can be implemented through the following example.

[0070] The inferred vehicle behavior type of the past training instance in a previous training cycle and the training instance target value of the past training instance are converted and calculated according to a preset log-likelihood cost function to obtain a first deviation for characterizing the inferred deviation of the previous training cycle.

[0071] In an embodiment of the present invention, illustratively, the server first obtains the inferred vehicle behavior type and training instance target value of the past training instance in the previous training cycle. These data are the basis for calculating the first deviation (inference deviation). For example, suppose that the server has a past training instance, which is inferred by the model as "illegal lane change" behavior in the previous training cycle, and the training instance target value (i.e., the true label) of the instance is "normal driving". Next, the server will use a preset log-likelihood cost function. This function can calculate a cost value based on the inference result of the model and the true label to quantify the deviation of the model inference. The log-likelihood cost function is usually used to measure the performance of the classification model, especially when dealing with multi-classification problems. It can calculate a loss value based on the probability distribution output by the model and the true label. The larger the loss value, the greater the difference between the inference result of the model and the true label. The server inputs the inferred vehicle behavior type and the training instance target value of the past training instance in the previous training cycle into the log-likelihood cost function for conversion calculation. This process will output a specific value, i.e., the first deviation used to characterize the inference deviation of the previous training cycle. The server inputs the inference result of "illegal lane change" and the true label of "normal driving" into the log-likelihood cost function. The function calculates a cost value based on these two inputs. This cost value is the first deviation, which reflects the model's inference deviation for this instance in the previous training cycle. Finally, the server uses this first deviation to optimize the vehicle abnormal behavior classification model. By reducing the first deviation, the server can guide the model to more accurately infer the type of vehicle behavior in future training processes, thereby improving the performance of the model. Such an optimization process is usually achieved through the backpropagation algorithm and the gradient descent method. The server adjusts the parameters of the model based on the first deviation so that the model can make more accurate inferences the next time it encounters a similar situation.

[0072] In an embodiment of the present invention, a second deviation for characterizing the confidence fluctuation between a previous training cycle and a current training cycle is determined based on the inference confidence of the past training instance in a previous training cycle and the inference confidence of the past training instance in the current training cycle, which can be implemented through the following example.

[0073] The inference confidence of the past training instance in the previous training cycle and the inference confidence of the past training instance in the current training cycle are converted and calculated according to a preset average absolute cost function to obtain a second deviation for characterizing the confidence fluctuation between the previous training cycle and the current training cycle.

[0074] In an embodiment of the present invention, illustratively, the server first obtains the inference confidence of the past training instance in the previous training cycle and the inference confidence in the current training cycle. These data are the basis for calculating the second deviation (confidence fluctuation). For example, suppose the server has a past training instance, and the inference confidence of the model for its behavior in the previous training cycle is 0.8 (that is, the model is 80% sure that its predicted behavior is correct), and in the current training cycle, the inference confidence of the same instance becomes 0.6. Next, the server uses a preset average absolute cost function. This function is used to measure the difference between two confidences, that is, the fluctuation of confidence. The average absolute cost function calculates the absolute value of the difference between the predicted value and the actual value, which reflects the degree to which the predicted value deviates from the actual value, regardless of the direction of deviation. In this scenario, the inference confidence of the past training instance in the previous training cycle and the current training cycle is regarded as "predicted value" and "actual value". The server inputs the inference confidence of the past training instance in the previous training cycle and the current training cycle into the average absolute cost function for conversion calculation. This process will output a specific value, which is the second deviation used to characterize the confidence fluctuations between the previous training cycle and the current training cycle. The server will calculate the absolute value of the difference between 0.8 and 0.6, that is, |0.8-0.6|=0.2. This 0.2 is the second deviation, which reflects the fluctuation of the model's inference confidence for the same instance in two training cycles. Finally, the server will use this second deviation to optimize the vehicle abnormal behavior classification model. By reducing the second deviation, the server can guide the model to maintain a more stable inference confidence in future training processes, thereby improving the robustness and reliability of the model. The server will use the second deviation as a regularization term in the model training process, and adjust the model parameters through optimization algorithms (such as gradient descent) to minimize this deviation. Doing so helps the model give more consistent and stable inference results when faced with similar or identical inputs.

[0075] In an embodiment of the present invention, the driving subject fatigue driving detection model is obtained in the following manner.

[0076] receiving a plurality of facial data of a plurality of driving subjects, wherein the plurality of facial data includes data related to a fatigue state of the driving subjects;

[0077] extracting a plurality of driving object features from the plurality of face data;

[0078] performing at least one of a weight adjustment operation on a first driving object feature among the plurality of driving object features, or an association relationship establishment operation on a second driving object feature among the plurality of driving object features and a third driving object feature among the plurality of driving object features;

[0079] performing an integration operation on the plurality of driving object features for which at least one of a weight adjustment operation or an association relationship establishment operation is performed; and

[0080] A model training process is performed on the multiple driving object features that perform the integration operation to obtain the driving object fatigue driving detection model.

[0081] In an embodiment of the present invention, illustratively, the server first receives multiple face data from different driving objects. These data are captured by vehicle cameras or other monitoring equipment and contain information related to the fatigue state of the driving object, such as yawning, eye closure frequency, etc. For example, the server can collect a large amount of driver face video data from the in-vehicle monitoring systems of multiple taxis. These data are shot in different time periods and driving environments, ensuring the diversity and practicality of the data. Next, the server uses advanced image processing technology to extract the features of multiple driving objects from the collected face data. These features may include the opening and closing state of the eyes, the posture of the head, the facial expression, etc., which are important bases for judging whether the driver is tired. For example, the server can use a deep learning algorithm to identify the state of the driver's eyes in the video frame, such as whether they are closed, the frequency and duration of closure, etc. The server then processes the extracted driving object features. This includes two possible operations: one is to adjust the weights of certain features to emphasize or weaken their importance in fatigue detection; the other is to establish associations between different features to more comprehensively evaluate the driver's fatigue state. For example, the server may find that the frequency and duration of eye closure are particularly important in judging fatigue, so it will increase the weights of these two features. At the same time, the server may also find that there is a certain correlation between the change in head posture and the state of the eyes, so it will establish a correlation between the two features. After completing the weight adjustment and establishing the correlation, the server will integrate these features. The purpose of integration is to merge these features into a comprehensive and comprehensive indicator that can accurately reflect the driver's fatigue state. For example, the server will use the weighted average method to combine the values ​​of different features with their weights to calculate a fatigue index. The higher this index is, the more likely the driver is in a fatigue state. Finally, the server will use the integrated feature data for model training. Through a large amount of historical data and machine learning algorithms, the server can train a model that can accurately detect the driver's fatigue state. For example, the server can use algorithms such as support vector machines (SVM), random forests, or neural networks to train the model. During the training process, the server will continuously adjust the parameters of the model to minimize the prediction error. The resulting driver fatigue driving detection model will be able to accurately determine whether the driver is in a fatigue state in real time, thereby improving road driving safety.

[0082] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned roadside digital video monitoring method based on deep learning. Figure 2 As shown, Figure 2 The block diagram of the computer device 100 provided in the embodiment of the present invention. The computer device 100 includes a memory 111, a processor 112 and a communication unit 113. To achieve data transmission or interaction, the memory 111, the processor 112 and the communication unit 113 are directly or indirectly electrically connected to each other. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0083] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.

Claims

1. A roadside digital video monitoring method based on deep learning, characterized in that: include: Acquire a roadside monitoring image to be analyzed and a vehicle abnormal behavior classification indication corresponding to the roadside monitoring image, and analyze the roadside monitoring image and the vehicle abnormal behavior classification indication to obtain a classification result of the vehicle abnormal behavior classification indication; When the classification result is characterized as an abnormal classification result, determining the target vehicle image data corresponding to the abnormal classification result; Acquire target face data of a target driving object included in the target vehicle image data, and input the target face data into a pre-trained driving fatigue detection model to obtain a driving object fatigue state of the target driving object as a monitoring result; The step of acquiring a roadside monitoring image to be analyzed and a vehicle abnormal behavior classification indication corresponding to the roadside monitoring image, and analyzing the roadside monitoring image and the vehicle abnormal behavior classification indication to obtain a classification result of the vehicle abnormal behavior classification indication includes: Acquire a roadside surveillance image to be analyzed and a vehicle abnormal behavior classification indication corresponding to the roadside surveillance image; Acquire a roadside monitoring image feature extraction network that is weakly associated with the vehicle abnormal behavior classification indication and a roadside monitoring image feature mapping network that is strongly associated with the vehicle abnormal behavior classification indication, wherein the roadside monitoring image feature extraction network is used to extract roadside monitoring image features, and the roadside monitoring image feature mapping network is a neural network obtained through pre-training based on the vehicle abnormal behavior classification indication; Performing a feature extraction operation on the roadside monitoring image through the roadside monitoring image feature extraction network to obtain a first roadside monitoring image feature that is weakly associated with the vehicle abnormal behavior classification indication; Performing feature mapping on the first roadside monitoring image feature through the roadside monitoring image feature mapping network to obtain a second roadside monitoring image feature that is strongly associated with the vehicle abnormal behavior classification indication; Obtaining a semantic description feature extraction network of the vehicle abnormal behavior classification indication and a feature representation strongly associated with the vehicle abnormal behavior classification indication, wherein the semantic description feature extraction network is used to extract semantic description features, and the feature representation is a feature vector obtained through prior training based on the vehicle abnormal behavior classification indication; Performing a feature extraction operation on the feature representation through the semantic description feature extraction network to obtain a first semantic description feature that is weakly associated with the vehicle abnormal behavior classification indication; Performing a feature extraction operation on a preset reference description through the semantic description feature extraction network to obtain a second semantic description feature that is strongly associated with the vehicle abnormal behavior classification indication; Comparing the first roadside monitoring image feature with the first semantic description feature to obtain a first feature matching degree; Comparing the second roadside monitoring image feature with the second semantic description feature to obtain a second feature matching degree; Performing a linear superposition operation on the first feature matching degree and the second feature matching degree according to a preset importance coefficient to obtain an inferred confidence level of the vehicle abnormal behavior classification indication; A classification result of the vehicle abnormal behavior classification indication is determined according to the inference confidence.

2. The method according to claim 1, characterized in that The classification result includes a vehicle behavior feature domain consisting of a plurality of inferred vehicle behavior types, and the inference confidence includes a vehicle behavior type confidence corresponding to each of the inferred vehicle behavior types; Determining a classification result of the vehicle abnormal behavior classification indication according to the inference confidence includes: Normalizing each vehicle behavior type confidence in the inferred confidence to obtain a normalized score of each inferred vehicle behavior type in the vehicle behavior feature domain; The inferred vehicle behavior type with the highest normalized score is used as the classification result of the vehicle abnormal behavior classification indication.

3. The method according to claim 1, characterized in that Before determining the classification result of the vehicle abnormal behavior classification indication according to the inference confidence, the method further includes: Acquire a vehicle abnormal behavior classification model for executing the vehicle abnormal behavior classification indication, wherein the vehicle abnormal behavior classification model includes a roadside monitoring image feature extraction network, a roadside monitoring image feature mapping network, and a semantic description feature extraction network; the roadside monitoring image feature extraction network is used to extract roadside monitoring image features, the roadside monitoring image feature mapping network is used to perform feature mapping on the roadside monitoring image features, and the semantic description feature extraction network is used to extract semantic description features; Acquire training instance data corresponding to the vehicle abnormal behavior classification model, wherein the training instance data includes a roadside monitoring image instance and a training instance target value bound to the roadside monitoring image instance; Performing vehicle abnormal behavior classification processing on the roadside monitoring image instance by using the vehicle abnormal behavior classification model to obtain an inferred vehicle behavior type corresponding to the roadside monitoring image instance; Determining a cost parameter of the vehicle abnormal behavior classification model according to the inferred vehicle behavior type and the training instance target value; Returning the cost parameter in the vehicle abnormal behavior classification model to obtain a parameter optimization direction corresponding to an adjustable parameter of the vehicle abnormal behavior classification model; The adjustable parameters of the vehicle abnormal behavior classification model are optimized according to the parameter optimization direction.

4. The method according to claim 3, characterized in that The cost parameters include a first cost parameter determined based on the training instance data of the current training cycle and a second cost parameter determined based on the training instance data of the previous training cycle. The vehicle abnormal behavior classification model performs different vehicle abnormal behavior classification instructions in the current training cycle and the previous training cycle.

5. The method according to claim 4, characterized in that The second cost parameter is obtained by the following method, including: Non-specifically selecting a preset number of past training instances from the training instance data of the previous training cycle, and obtaining classification results of the past training instances in the previous training cycle; Using the vehicle abnormal behavior classification model to perform vehicle abnormal behavior classification processing on the past training examples, and obtaining classification results of the past training examples in the current training cycle; The second cost parameter is determined according to the classification results of the past training examples in the previous training cycle and the classification results of the past training examples in the current training cycle.

6. The method according to claim 5, characterized in that The classification result of the past training instance in the current training cycle includes the inference confidence of the past training instance in the current training cycle, and the classification result of the past training instance in the previous training cycle includes the inference confidence of the past training instance in the previous training cycle and the inferred vehicle behavior type of the past training instance in the previous training cycle; Determining the second cost parameter according to the classification result of the past training instance in the previous training cycle and the classification result of the past training instance in the current training cycle includes: Converting and calculating the inferred vehicle behavior type of the past training instance in a previous training cycle and the training instance target value of the past training instance according to a preset log-likelihood cost function to obtain a first deviation for characterizing the inferred deviation of the previous training cycle; Determine, according to the inference confidence of the past training instance in the previous training cycle and the inference confidence of the past training instance in the current training cycle, a second deviation for characterizing the confidence fluctuation between the previous training cycle and the current training cycle; The second cost parameter is determined according to the first deviation and the second deviation.

7. The method according to claim 6, characterized in that Determining a second deviation for characterizing the confidence fluctuation between the previous training cycle and the current training cycle according to the inference confidence of the past training instance in the previous training cycle and the inference confidence of the past training instance in the current training cycle includes: The inference confidence of the past training instance in the previous training cycle and the inference confidence of the past training instance in the current training cycle are converted and calculated according to a preset average absolute cost function to obtain a second deviation for characterizing the confidence fluctuation between the previous training cycle and the current training cycle.

8. The method according to claim 1, characterized in that The driving fatigue detection model of the driving subject is obtained by the following methods, including: receiving a plurality of facial data of a plurality of driving subjects, wherein the plurality of facial data includes data related to a fatigue state of the driving subjects; extracting a plurality of driving object features from the plurality of face data; performing at least one of a weight adjustment operation on a first driving object feature among the plurality of driving object features, or an association relationship establishment operation on a second driving object feature among the plurality of driving object features and a third driving object feature among the plurality of driving object features; performing an integration operation on the plurality of driving object features for which at least one of a weight adjustment operation or an association relationship establishment operation is performed; and A model training process is performed on the multiple driving object features that perform the integration operation to obtain the driving object fatigue driving detection model.

9. A server system, characterized in that: The method comprises a server, wherein the server is used to execute the method described in any one of claims 1 to 8.

Images