Vehicle safety officer supervision method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a vehicle safety officer supervision method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining training data, and generating a first safety officer supervision model based on initial operation state data, initial sensor sensing information and an initial monitoring video; when real-time operation state data, real-time sensor sensing information and a real-time monitoring video containing operation behaviors of a security officer are obtained, inputting the real-time operation state data, the real-time sensor sensing information and the real-time monitoring video into a first security officer supervision model; outputting a first supervision judgment result for the security officer through the first security officer supervision model, wherein the first supervision judgment result is used for judging whether behaviors of the security officer for the real-time operation state data and the real-time sensor sensing information meet preset requirements or not; according to the invention, the safety officers of the automatic driving vehicle can be supervised in real time, and the safety of the automatic driving vehicle is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle safety officer supervision, and in particular to a vehicle safety officer supervision method, a vehicle safety officer supervision device, an electronic device and a computer-readable storage medium. Background Art

[0002] Safety officers play a vital role in road testing and demonstration applications of intelligent connected vehicles. They not only need to take over the vehicle in time when problems occur in the autonomous driving system to ensure driving safety, but also have the responsibility of monitoring the vehicle status in real time, providing feedback on the subjective experience of the test, and assisting the test team in improving the system.

[0003] At present, the supervision of safety officers mainly relies on video surveillance on vehicles, which is checked regularly or irregularly by operators. This method has the problem of insufficient real-time performance and cannot monitor the behavior of safety officers in real time. Summary of the invention

[0004] The embodiments of the present invention provide a vehicle safety officer supervision method, device, electronic device and computer-readable storage medium to overcome the above problems or at least partially solve the above problems.

[0005] The embodiment of the present invention discloses a vehicle safety officer supervision method, comprising:

[0006] Acquiring training data, wherein the training data includes initial operating state data of the vehicle, initial sensor perception information, and initial monitoring video containing operating behavior of the safety officer;

[0007] Generate a first safety officer supervision model based on the initial operating state data, the initial sensor perception information and the initial monitoring video;

[0008] When real-time operation status data, real-time sensor perception information, and real-time monitoring video containing the operation behavior of the safety officer are obtained, the real-time operation status data, the real-time sensor perception information, and the real-time monitoring video are input into the first safety officer supervision model, and the first supervision judgment result for the safety officer is output through the first safety officer supervision model;

[0009] When it is determined through the first supervision judgment result that the behavior of the safety officer with respect to the real-time operating status data and the real-time sensor perception information does not meet the preset requirements, an alarm message is generated.

[0010] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0011] The memory is used to store computer programs;

[0012] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0013] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0014] The embodiments of the present invention include the following advantages:

[0015] In an embodiment of the present invention, training data is acquired, wherein the training data includes initial operating status data of the vehicle, initial sensor perception information, and an initial monitoring video containing the operating behavior of the safety officer; a first safety officer supervision model is generated based on the initial operating status data, the initial sensor perception information, and the initial monitoring video; when real-time operating status data, real-time sensor perception information, and a real-time monitoring video containing the operating behavior of the safety officer are acquired, the real-time operating status data, the real-time sensor perception information, and the real-time monitoring video are input into the first safety officer supervision model, and a first supervision judgment result for the safety officer is output through the first safety officer supervision model; when it is determined through the first supervision judgment result that the safety officer's behavior with respect to the real-time operating status data and the real-time sensor perception information does not meet the preset requirements, an alarm message is generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the steps of a vehicle safety officer supervision method provided in an embodiment of the present invention;

[0017] Figure 2 It is a structural block diagram of a vehicle safety officer monitoring device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] Reference Figure 1 , shows a flow chart of the steps of a vehicle safety officer supervision method provided in an embodiment of the present invention, which may specifically include the following steps:

[0020] Step 101, obtaining training data, wherein the training data includes initial operating state data of the vehicle, initial sensor perception information, and initial monitoring video containing the operating behavior of the safety officer;

[0021] Step 102, generating a first safety officer supervision model based on the initial operating state data, the initial sensor perception information and the initial monitoring video;

[0022] Step 103, when real-time operation status data, real-time sensor perception information, and real-time monitoring video containing the operation behavior of the safety officer are obtained, the real-time operation status data, the real-time sensor perception information, and the real-time monitoring video are input into the first safety officer supervision model, and the first supervision judgment result for the safety officer is output through the first safety officer supervision model;

[0023] Step 104 , when it is determined through the first supervision determination result that the safety officer's behavior with respect to the real-time operating status data and the real-time sensor perception information does not meet preset requirements, an alarm message is generated.

[0024] The embodiment of the present invention can be applied to an intelligent connected vehicle supervision and operation platform. The purpose of acquiring training data is to collect sufficient high-quality data for training a safety officer behavior supervision model.

[0025] In practical applications, the initial operating state data of the vehicle may include, but is not limited to, the vehicle's speed, acceleration, steering wheel angle, etc., reflecting the dynamic state of the vehicle.

[0026] The initial sensor perception information may include, but is not limited to, environmental information collected by sensors such as cameras and radars, such as lane lines, pedestrians, obstacles, etc.

[0027] The initial monitoring video may be a video containing the safety officer's operating behavior, which is used to capture the safety officer's visual and motion information.

[0028] The purpose of generating the first safety officer supervision model is to establish a model that can identify normal and abnormal behaviors of safety officers.

[0029] In a specific implementation, the present invention can extract useful features from the initial operating status data, initial sensor perception information and initial monitoring video, such as vehicle status features, road environment features, safety officer behavior features, etc. The model training process can utilize machine learning or deep learning algorithms, and use the extracted features as input to train a model that can distinguish between normal and abnormal behaviors of safety officers. Optionally, after generating the first safety officer supervision model, the trained first safety officer supervision model can be evaluated to ensure that it has high accuracy and robustness.

[0030] The embodiment of the present invention can also perform real-time monitoring and behavior judgment on the safety officer through the first safety officer supervision model, with the aim of monitoring the safety officer's behavior in real time during actual driving and judging whether his behavior meets preset requirements.

[0031] The process includes: inputting the real-time collected vehicle operation status data, sensor perception information and monitoring video into the trained model. The model outputs the first supervisory judgment result based on the input data. The first supervisory judgment result can be used by the safety officer to determine whether the safety officer's behavior in response to the real-time operation status data and real-time sensor perception information meets the preset requirements. For example, when the vehicle is determined to be speeding through the real-time operation status data, the first supervisory judgment result can be used to determine whether the safety officer performs a braking operation on the vehicle in response to the speeding behavior. For example, when the vehicle is determined to be driving on the solid line through the real-time sensor perception information, the first supervisory judgment result can be used to determine whether the safety officer performs a direction return operation on the vehicle in response to the vehicle's driving behavior on the solid line; that is, when the safety officer's behavior in response to the real-time operation status data and real-time sensor perception information is determined to be not in accordance with the preset requirements through the first supervisory judgment result, a corresponding alarm information can be given for feedback or alarm reminder.

[0032] In an embodiment of the present invention, training data is acquired, wherein the training data includes initial operating status data of the vehicle, initial sensor perception information, and an initial monitoring video containing the operating behavior of the safety officer; a first safety officer supervision model is generated based on the initial operating status data, the initial sensor perception information, and the initial monitoring video; when real-time operating status data, real-time sensor perception information, and a real-time monitoring video containing the operating behavior of the safety officer are acquired, the real-time operating status data, the real-time sensor perception information, and the real-time monitoring video are input into the first safety officer supervision model, and a first supervision judgment result for the safety officer is output through the first safety officer supervision model; when it is determined through the first supervision judgment result that the safety officer's behavior with respect to the real-time operating status data and the real-time sensor perception information does not meet the preset requirements, an alarm message is generated.

[0033] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0034] In an optional embodiment of the present invention, the first safety officer supervision model includes an input layer, and the step of generating the first safety officer supervision model based on the initial operating state data, the initial sensor perception information and the initial monitoring video includes:

[0035] Performing a preprocessing operation based on the initial operating state data, the initial sensor perception information and the initial monitoring video;

[0036] Receiving the initial operating state data, the initial sensor perception information and the initial monitoring video through the input layer;

[0037] Using a multimodal data fusion method, the initial operating state data, the initial sensor perception information and the initial monitoring video based on different modalities are converted into a unified format, and a feature vector for expressing the initial operating state data, the initial sensor perception information and the initial monitoring video is extracted;

[0038] A first safety officer supervision model is generated based on the feature vector.

[0039] The first safety officer supervision model may include:

[0040] Input layer: Receives raw data from different sensors (such as cameras, radars, on-board computers, etc.). After preprocessing, these data form the input of the model.

[0041] Convolutional layer: Mainly used to extract image features, for example, extracting features such as facial expressions and gestures of security guards from video frames.

[0042] Activation function: Introduce nonlinear factors to enhance the expressiveness of the model. Commonly used activation functions include ReLU, Sigmoid, etc.

[0043] Fully connected layer: maps the feature vector output by the convolutional layer to a vector of fixed length in preparation for the classification task.

[0044] Output layer: Outputs the safety officer's behavior classification results, such as normal driving, distracted driving, fatigue driving, etc.

[0045] The purpose of performing preprocessing operations is to convert raw data into a format that can be processed by the model and improve the training efficiency and accuracy of the model.

[0046] Exemplarily, the pre-processing operation may include:

[0047] Data cleaning: remove noise, outliers, etc.

[0048] Data normalization: scaling data of different features to the same range.

[0049] Data enhancement: Increase data diversity through rotation, scaling, cropping, etc. to improve the generalization ability of the model.

[0050] The purpose of multimodal data fusion is to integrate data of different modalities (such as images, texts, and numerical values) to obtain more comprehensive information.

[0051] In a specific implementation, the initial operating status data, initial sensor perception information, and initial monitoring video of different modalities can be converted into a unified format, and then representative features in the data can be extracted, that is, feature vectors used to express the initial operating status data, initial sensor perception information, and initial monitoring video.

[0052] The purpose of converting to a unified format is to convert the perception information (such as images, point clouds, etc.) obtained from different types of sensors (such as cameras, radars, etc.) and information in surveillance videos into a unified data representation for computer processing and analysis.

[0053] Different sensors acquire different types of data. For example, cameras acquire image data, while radars acquire point cloud data. These data differ in data structure and dimension, making them difficult to process directly.

[0054] Unified processing: Converting data of different modalities into a unified format allows them to be easily input into the same model for processing, thus achieving the fusion of multimodal information.

[0055] For example, assume that a smart vehicle is equipped with a camera and a lidar. The image data acquired by the camera can be represented as a three-dimensional tensor (height, width, number of channels), and the point cloud data acquired by the lidar can be represented as a two-dimensional matrix (each point contains x, y, z coordinates).

[0056] The details are as follows:

[0057] Feature extraction:

[0058] For image data, features such as edges, textures, and colors can be extracted.

[0059] For point cloud data, features such as point density and curvature can be extracted.

[0060] Feature encoding:

[0061] Convert the extracted features into numerical vectors, for example, use CNN to extract image features and use PointNet to extract point cloud features.

[0062] Feature Fusion:

[0063] The eigenvectors of different modes are concatenated or weighted summed to form a unified eigenvector.

[0064] Dimensional unification:

[0065] If the dimensions of different feature vectors are different, dimension expansion or dimension reduction operations can be performed to make them consistent.

[0066] An example of the unified format:

[0067] Assuming that we have extracted a 1024-dimensional feature vector for the image and a 512-dimensional feature vector for the point cloud, we can concatenate them into a 1536-dimensional feature vector as the input of the model.

[0068] By converting data from different modalities into a unified format, multi-sensor information fusion can be achieved: making full use of the information provided by different sensors to improve the accuracy and robustness of perception.

[0069] Simplified model design: Only one model needs to be designed to process data in a unified format, which reduces the complexity of the model.

[0070] Improve model performance: Fusing multimodal information can provide richer contextual information and help the model make more accurate decisions.

[0071] Converting data from different modalities into a unified format is a key step in multimodal data fusion. Through this process, we can better utilize the information from different sensors and improve the perception and decision-making capabilities of intelligent systems.

[0072] Multimodal data fusion methods can include:

[0073] Early fusion: Fusion is performed before feature extraction.

[0074] Late fusion: Fusion is performed before the classifier.

[0075] Intermediate fusion: Fusion between feature extraction and classification.

[0076] Attention mechanism: The attention mechanism can be used to learn the importance of features of different modalities and adaptively adjust the fusion weights.

[0077] The fused multimodal data is represented as a vector of fixed length as the input of the model to train and generate the first safety officer supervision model.

[0078] Optionally, if the feature dimension is too high, PCA, t-SNE and other methods can be used to reduce the dimension; map the feature vector to a low-dimensional space.

[0079] By using the fused feature vector to train the first safety officer supervision model, the model can accurately identify the behavior of the safety officer.

[0080] Specifically, a suitable loss function, such as cross entropy loss, can be selected to measure the difference between the model prediction results and the true labels.

[0081] Use an optimization algorithm (such as gradient descent) to minimize the loss function and update the model parameters.

[0082] By performing preprocessing operations on the initial operating status data, the initial sensor perception information and the initial monitoring video; receiving the initial operating status data, the initial sensor perception information and the initial monitoring video through the input layer; using a multimodal data fusion method to convert the initial operating status data, the initial sensor perception information and the initial monitoring video of different modalities into a unified format, and extracting feature vectors for expressing the initial operating status data, the initial sensor perception information and the initial monitoring video; generating a first safety officer supervision model based on the feature vector to improve monitoring accuracy: through multimodal data fusion, the behavioral characteristics of the safety officer can be more comprehensively captured, the accuracy of monitoring can be improved, and the following beneficial effects can be achieved.

[0083] Enhance model robustness: Data from different modalities can complement each other and improve the model's robustness to noise and interference.

[0084] Adapt to complex scenarios: The model can adapt to various complex driving scenarios, such as night driving, bad weather, etc.

[0085] Provide multi-dimensional information: It can provide a variety of behavioral indicators of safety officers, such as fatigue level, concentration, etc.

[0086] Supports multiple application scenarios: It can not only be used for safety officer behavior monitoring, but also can be extended to other fields, such as driving behavior analysis, human-computer interaction, etc.

[0087] This technical solution achieves real-time and accurate monitoring of safety officer behavior by building a multimodal safety officer behavior monitoring model based on deep learning. The model has good generalization ability and robustness, and can effectively improve the safety of intelligent connected vehicles.

[0088] In an optional embodiment of the present invention, the step of performing a preprocessing operation based on the initial operating state data, the initial sensor perception information and the initial monitoring video includes:

[0089] Performing data cleaning, denoising and normalization operations on the initial operating state data, the initial sensor perception information and the initial monitoring video;

[0090] Constructing a labeling rule for an interactive automatic labeling tool based on the initial operating state data, the initial sensor perception information, and the initial monitoring video;

[0091] The initial operating status data, the initial sensor perception information and the initial monitoring video are imported into the interactive automatic labeling tool, and the interactive automatic labeling tool is controlled to output the labeled initial operating status data, the initial sensor perception information and the initial monitoring video.

[0092] X-AnyLabeling is a powerful interactive automatic labeling tool that can significantly improve the efficiency and accuracy of data labeling. In simple terms, it is a tool that can help users label data such as pictures and videos faster and better.

[0093] In the field of machine learning and deep learning, model training is inseparable from a large amount of high-quality labeled data. These labeled data tell the model which data belongs to which category, so that the model can learn the rules. For example, in image recognition tasks, users need to label whether the picture is a cat, a dog, or another object.

[0094] Advantages of X-AnyLabeling

[0095] Efficient: Using advanced deep learning models, X-AnyLabeling can automatically complete most of the labeling work, greatly reducing the time for manual labeling.

[0096] Accurate: Based on the SOTA (State-of-the-Art) model, X-AnyLabeling has a high labeling accuracy, reducing the cost of manual error correction.

[0097] Flexible: Supports multiple data types and annotation methods to meet various annotation needs.

[0098] Easy to use: It provides a friendly user interface and is easy to operate. Even people without programming experience can quickly get started.

[0099] How X-AnyLabeling works:

[0100] Data import: Import the data to be labeled (such as pictures and videos) into X-AnyLabeling.

[0101] Model reasoning: The system uses a pre-trained model to analyze the data and automatically generates preliminary annotation results.

[0102] Manual verification: Manually check the automatically generated annotation results and correct incorrect annotations.

[0103] Data export: Export the final labeled data for model training.

[0104] In a specific implementation, the embodiments of the present invention efficiently annotate a large amount of training data through automated and semi-automated methods. This method can not only significantly improve the efficiency of annotation, but also ensure the consistency and accuracy of annotation, thereby providing a high-quality data foundation for subsequent model training.

[0105] The specific steps can be as follows:

[0106] Data preparation:

[0107] Collect various types of raw data, including images, videos, audio, etc.

[0108] Perform preliminary cleaning and organization of the data to remove noise and redundant information; Data normalization: scale data with different features to the same range.

[0109] X-AnyLabeling configuration:

[0110] The initial operating status data, initial sensor perception information, and initial surveillance video are determined as labeling tasks. According to the specific labeling tasks, the labeling template and labeling rules of X-AnyLabeling are configured.

[0111] Select a suitable pre-trained model to improve the accuracy of automatic labeling.

[0112] Configure the annotation interface to suit the annotator's habits.

[0113] Automatic labeling:

[0114] Import the prepared data into X-AnyLabeling.

[0115] The system automatically labels the data based on the pre-trained model.

[0116] Manually review the automatic annotation results and make necessary corrections.

[0117] Manual detailed annotation:

[0118] Check the automatic labeling results one by one, focusing on samples that the model has difficulty labeling accurately.

[0119] Correct incorrect annotations and provide additional annotations for unannotated or vaguely labeled samples.

[0120] Data Export:

[0121] Export the final labeled data for model training.

[0122] The following beneficial effects can be achieved by performing data cleaning, denoising and normalization operations on the initial operating status data, the initial sensor perception information and the initial monitoring video; constructing annotation rules for an interactive automatic annotation tool based on the initial operating status data, the initial sensor perception information and the initial monitoring video; importing the initial operating status data, the initial sensor perception information and the initial monitoring video into the interactive automatic annotation tool, and controlling the interactive automatic annotation tool to output the annotated initial operating status data, initial sensor perception information and initial monitoring video.

[0123] Improve annotation efficiency: Automated annotation significantly reduces the workload of manual annotation and improves annotation efficiency.

[0124] Ensure annotation quality: The combination of the introduction of pre-trained models and manual review ensures the consistency and accuracy of annotations.

[0125] Support multiple data types: X-AnyLabeling supports multiple data types and has strong adaptability.

[0126] Provide a visual interface: The visual interface facilitates user operation and lowers the usage threshold.

[0127] Improve model performance: High-quality labeled data is the basis of model training and directly affects model performance.

[0128] X-AnyLabeling provides an efficient and accurate solution for training data labeling. By combining automation and semi-automation, it can significantly improve labeling efficiency and ensure labeling quality. This solution is of great significance for large-scale data labeling tasks.

[0129] In an optional embodiment of the present invention, the step of using a multimodal data fusion method to convert the initial operating state data, the initial sensor perception information and the initial monitoring video of different modalities into a unified format, and extracting a feature vector for expressing the initial operating state data, the initial sensor perception information and the initial monitoring video includes:

[0130] Eliminate data whose annotation information is inconsistent with the annotation rules, as well as data with missing annotation information;

[0131] Convert the annotated initial operating status data, initial sensor perception information and initial monitoring video of different modalities into a unified format;

[0132] The labeled information is used as a supervisory signal to extract a feature vector for expressing the initial operating state data, the initial sensor perception information, and the initial monitoring video.

[0133] For example, fine-grained annotations can be performed on initial operating status data, sensor perception information, and surveillance videos. For example, the vehicle's driving status (going straight, turning), the driver's behavior (steering, braking), and road conditions (congested, unobstructed) can be annotated in the video.

[0134] Cleaning operations may include: removing data whose annotation information is inconsistent with the annotation rules, as well as data with missing annotation information, to ensure data quality.

[0135] The embodiment of the present invention can convert the annotated initial operating status data, initial sensor perception information and initial monitoring video of different modalities into a unified format, and then use the annotated information as a supervisory signal for feature extraction.

[0136] The way to use the annotation information as a supervisory signal for feature extraction can be as follows:

[0137] Annotated data as supervisory information: Use the annotated information as a supervisory signal to guide the feature extraction process. For example, when extracting video features, you can focus on the annotated vehicle behavior area.

[0138] Multimodal feature extraction:

[0139] Image features: Extract features related to the annotation information, such as the vehicle’s outline, lane lines, traffic signs, etc.

[0140] Text features: If there is text data (such as logs from an onboard computer), extract keywords related to the annotated information.

[0141] Sensor data features: Extract sensor data features related to the annotation information, such as acceleration, angular velocity, etc.

[0142] Feature Fusion:

[0143] Weighted fusion based on annotation information: The features of different modalities are weightedly fused according to the confidence or importance of the annotation information.

[0144] Attention mechanism: Use the attention mechanism to adaptively learn the importance of different modal features and dynamically adjust the fusion weights.

[0145] The fused features are represented as a unified feature vector: it is used as the input of the subsequent model to train and generate the first safety officer supervision model.

[0146] The embodiments of the present invention can achieve the following beneficial effects by eliminating data whose annotation information is inconsistent with the annotation rules and data with missing annotation information; converting the annotated initial operating state data, initial sensor perception information and initial monitoring video of different modalities into a unified format; and using the annotation information as a supervision signal to extract feature vectors for expressing the initial operating state data, the initial sensor perception information and the initial monitoring video:

[0147] Improve the discriminability of features: By guiding feature extraction through annotation information, more discriminative features can be extracted, improving the classification and prediction performance of the model.

[0148] Enhance the interpretability of the model: Annotation information can help us understand the decision-making process of the model and improve the transparency of the model.

[0149] Adapt to more complex scenarios: Combined with annotation information, the model can better adapt to complex and changing scenarios, such as driving behaviors of different drivers and under different road conditions.

[0150] Improve the generalization ability of the model: By annotating data, the model can be trained to learn more general feature representations and improve the generalization ability of the model on unseen data.

[0151] Combining annotated data for multimodal fusion can make full use of the annotated information to guide feature extraction and model training, thereby improving the performance and interpretability of the model. This method has broad application prospects in many fields.

[0152] In order to enable those skilled in the art to better understand the embodiments of the present invention, an example is used below to illustrate the embodiments of the present invention.

[0153] For example, the behavior of the safety officer can be monitored in real time by building a safety officer behavior CNN model. The following is a specific implementation method:

[0154] S1. First, a convolutional neural network (CNN) is used to build a safety officer behavior model. The safety officer behavior CNN model includes an input layer, a convolutional layer, an activation function, a fully connected layer, and an output layer. The input layer receives the vehicle's operating status data, sensor perception information, and surveillance videos containing the safety officer's operating behavior. These data can be converted into a format suitable for CNN processing through image preprocessing technology. The convolutional layer uses multiple convolution kernels (such as 3x3 and 5x5) to capture features of different scales. Each convolutional layer is usually followed by a pooling layer (such as maximum pooling) to reduce the amount of calculation and retain important features. The ReLU activation function is used to help introduce nonlinear characteristics so that the model can better learn complex patterns. The fully connected layer maps the features extracted by the previous layers to the final classification results, and a multi-head self-attention mechanism can be used to capture global context information. The output layer is used to classify whether the safety officer's behavior is normal or has potential risks. Two-classification or multi-classification tasks can be used, and the appropriate activation function (such as Sigmoid or Softmax) can be selected according to the actual situation. In actual implementation, the above CNN structure can be implemented using PyTorch or other deep learning frameworks, and trained and tested in a real environment.

[0155] S2. Then, prepare the training data of the safety officer behavior CNN model. Collect and prepare the training data of the model such as the vehicle's operating status, sensor perception information, and monitoring videos containing the safety officer's operating behavior in advance, and preprocess these training data, including data cleaning, denoising, normalization, and format conversion, to meet the input requirements of the convolutional neural network.

[0156] The vehicle's operating status, sensor perception information, and surveillance videos containing the safety officer's operating behavior are used as the input of the safety officer behavior CNN model. These data need to be fused. The multimodal data fusion method is used to convert different types of input data into a unified format and perform feature extraction, thereby forming a unified and comprehensive training data set.

[0157] S3. Multimodal data fusion is a technical method that combines data from different modalities (such as vision, hearing, text, etc.) to improve information processing and understanding capabilities. Generally speaking, modality refers to the way things happen or exist, and multimodality refers to various forms of combinations of two or more modalities. Each source or form of information can be called a modality. The current research field mainly focuses on the processing of three modalities: image, text, and speech. Multimodal data fusion algorithms can be divided into traditional feature integration algorithms and methods based on deep learning. Traditional feature fusion algorithms can be mainly divided into three categories: 1. Algorithms based on Bayesian decision theory 2. Algorithms based on sparse representation theory 3. Algorithms based on deep learning theory.

[0158] In addition, in order to improve the generalization ability of the model, the training data can be enhanced, such as rotating, scaling, and cropping the video data to generate more training samples.

[0159] The preparation of training data is a key step, and the integrated data also needs to be labeled, which directly affects the training effect of the model and the final application performance. This application uses the X-AnyLabeling interactive automatic labeling tool for labeling, thereby realizing the preparation of training data. X-AnyLabeling is a new generation of automatic labeling tool that integrates multiple SOTA deep learning models. It is built and secondary developed based on AnyLabeling and supports a variety of models and functions. X-AnyLabeling provides powerful AI support, which can automatically or semi-automatically complete data labeling tasks, supports a variety of data types, including images, videos, audio, and text, and provides a visual interface for user convenience. First, the data labeling task is automatically completed through X-AnyLabeling, and then the labeling results are manually adjusted. Use the labeled historical data to train the model and adjust the model parameters to optimize performance.

[0160] S4. Next, select a suitable loss function and optimizer, and train the safety officer behavior CNN model. Select the cross entropy loss function as the loss function of the safety officer behavior CNN model, use Adam or SGD as the optimizer of the safety officer behavior CNN model, and adjust the model parameters to minimize the loss function. In the training phase, the model parameters are updated using the back propagation algorithm through the training data to gradually improve the performance of the model. Mini-batch stochastic gradient descent (SGD) can be used to accelerate the training process. In the testing phase, the performance of the model is evaluated using the validation set, and the hyperparameters (such as learning rate, batch size, etc.) are adjusted to achieve the best results.

[0161] S5. Finally, the trained model is deployed to the intelligent networked vehicle supervision and operation platform to realize real-time monitoring of the safety officer's behavior. The supervision and operation platform obtains the real-time vehicle data collected by the T-Box vehicle terminal, and then uses the trained safety officer behavior CNN model to monitor and analyze the data collected by the T-Box in real time to evaluate the safety officer's operational response and behavior pattern. When abnormal behavior or potential risks are detected, the system automatically issues an early warning and takes corresponding measures.

[0162] S1: Build a CNN model of safety officer behavior;

[0163] Steps: Use CNN to build a model, input vehicle operating status, sensor data and surveillance video, and output the classification result of whether the safety officer's behavior is normal.

[0164] Beneficial effects:

[0165] Automatic identification: It can automatically identify abnormal behaviors of security personnel and reduce manual intervention.

[0166] Real-time monitoring: timely identify potential risks through real-time data analysis.

[0167] Data-driven: Train models based on large amounts of data to improve model accuracy.

[0168] Multimodal fusion: Combine multiple data sources to more comprehensively evaluate the safety officer’s behavior.

[0169] S2: prepare training data;

[0170] Steps: Collect and preprocess various types of data, including vehicle operating status, sensor data, and surveillance video, and perform data fusion.

[0171] Beneficial effects:

[0172] Data diversity: Various types of data can provide more comprehensive information and improve the generalization ability of the model.

[0173] Data quality: The preprocessed data has higher quality, which is conducive to model training.

[0174] Data fusion: The fusion of data from different modalities can better capture the characteristics of safety officer behavior.

[0175] S3: Multimodal data fusion;

[0176] Steps: Use traditional feature fusion algorithms to convert different types of data into a unified format.

[0177] Beneficial effects:

[0178] Information complementarity: Data from different modalities can complement each other and provide more complete information.

[0179] Feature extraction: The fused data can extract more discriminative features.

[0180] S4: Model training;

[0181] Steps: Select a suitable loss function and optimizer, and train the model using training data.

[0182] Beneficial effects:

[0183] Model optimization: Improve the performance of the model by continuously adjusting model parameters.

[0184] Generalization ability: Improve the generalization ability of the model through data augmentation and validation set evaluation.

[0185] S5: Model deployment;

[0186] Steps: Deploy the trained model to the intelligent connected vehicle supervision and operation platform to achieve real-time monitoring.

[0187] Beneficial effects:

[0188] Real-time warning: abnormal behavior can be detected in real time and warnings can be issued.

[0189] Intelligent decision-making: providing a basis for subsequent decisions.

[0190] The entire process, from model building, data preparation, model training to final deployment, forms a complete safety officer behavior monitoring system. Through multimodal data fusion and deep learning technology, the system can achieve real-time, accurate and automated monitoring of safety officer behavior, providing strong support for the safe operation of intelligent connected vehicles.

[0191] In an optional embodiment of the present invention, the training data further includes road test data and safety officer demonstration application historical data used to express when the safety officer manually takes over the autonomous driving vehicle during the historical demonstration application process, and the method further includes:

[0192] Generate a second safety officer supervision model based on the road test data and the safety officer demonstration application historical data;

[0193] When the real-time road information and the real-time monitoring video are obtained, the real-time road information and the real-time monitoring video are input into the second safety officer supervision model, and the second supervision determination result is outputted through the second safety officer supervision model;

[0194] When it is determined through the second supervision judgment result that the trigger time of the safety officer's reaction behavior to the real-time road information is greater than a preset time threshold, an alarm message is generated.

[0195] Exemplarily, S21 first collects a large amount of historical data when the safety officer manually takes over the autonomous driving vehicle during the road test and demonstration application of the intelligent connected vehicle, and performs statistical analysis on the historical data to obtain several typical manual takeover scenarios and the vehicle driving status data and video surveillance data of the vehicle's internal and external environments corresponding to each typical manual takeover scenario. Based on these data, a typical manual takeover scenario judgment model, i.e., the second safety officer supervision model, can be generated.

[0196] Typical manual takeover scenarios may include: the autonomous driving system may be unable to accurately identify road conditions or plan routes, or may be unable to handle certain complex scenarios due to instantaneous problems, requiring the driver to take over manually, such as the need to quickly avoid emergencies such as vehicles or pedestrians, or encountering congestion, complex intersections, obstacles, etc.

[0197] Typical manual takeover scenario judgment models use machine learning or deep learning technology to perceive and understand the environment, and then determine whether manual takeover is required based on preset rules or strategies. For example, anomaly detection technology based on multimodal data such as Camera, LiDAR, and Radar. This type of algorithm model uses multimodal sensor data and machine learning technology to determine whether an autonomous vehicle needs manual takeover in a specific scenario.

[0198] The purpose of constructing a typical manual takeover scenario judgment model is to automatically identify typical scenarios that require manual takeover by safety officers, providing a basis for subsequent safety officer behavior monitoring.

[0199] The process may include:

[0200] Data collection: Collect a large amount of historical data, including vehicle status data, environmental perception data (such as cameras, lidar, etc.), and video data when the safety officer takes over.

[0201] Scenario classification: historical data is classified according to different takeover scenarios, for example:

[0202] Sudden obstacle: Pedestrians or obstacles suddenly appear in front of the vehicle.

[0203] Complex intersections: When a vehicle reaches a complex intersection, it needs to make a judgment.

[0204] Bad weather: rain, snow, etc.

[0205] Feature extraction: Extract useful features from the data, such as the vehicle’s speed, acceleration, steering angle, distance and type of objects sensed by the sensor, etc.

[0206] Model training: Use machine learning or deep learning algorithms to train a classification model that can determine whether the current scenario is a typical scenario that requires manual takeover based on the extracted features.

[0207] Beneficial effects:

[0208] Improve monitoring accuracy: By pre-defining typical scenarios, you can more accurately identify the moments that require security officer intervention, reducing false positives and false negatives.

[0209] Reduce manual intervention: The model can automatically determine whether manual takeover is needed, reducing the burden of manual monitoring.

[0210] Provide data support: The collected data can be used to further analyze the behavior patterns of safety officers and system performance.

[0211] S22, then, collect in real time the vehicle driving status data and the video surveillance data of the inside and outside environment of the vehicle during the road test and demonstration application of the intelligent connected vehicle, and input them into the typical manual takeover scenario judgment model, that is, the second safety officer supervision model. When it is judged that the moment is a typical manual takeover scenario, monitor whether the safety officer performs manual takeover operations on the autonomous driving vehicle within the preset time. If the safety officer does not perform manual takeover operations on the autonomous driving vehicle within the preset time, it is considered that the safety officer has failed to perform the duties of a safety officer and is dereliction of duty. Specifically, the real-time road information and the real-time monitoring video can be input into the second safety officer supervision model, and the second supervision judgment result can be output through the second safety officer supervision model. When it is determined through the second supervision judgment result that the trigger time of the safety officer's reaction behavior to the real-time road information is greater than the preset time threshold, an alarm information is generated.

[0212] According to relevant research, the time required for the driver to take over varies greatly depending on the driver's reaction characteristics, the NDRT (Non-Driving Related Task) they are engaged in, and the driving environment they are in. In scenarios with time requirements, the takeover time is mostly in the range of 3 to 7 seconds, while in studies of takeovers without time pressure, the average time for the driver to take over is 4.5 to 6.0 seconds. Therefore, the preset time can be set to 7 seconds.

[0213] If the result of the judgment is that the safety officer has neglected his duty, the relevant data will be saved, including the vehicle driving status data and the video surveillance data of the vehicle's internal and external environment. In order to avoid errors in model judgment, the operator can review the saved data afterwards to confirm whether the safety officer has really neglected his duty.

[0214] The purpose of safety officer behavior monitoring is to monitor in real time whether the safety officer's response time and operations in typical scenarios meet the requirements based on the constructed model.

[0215] The process may include:

[0216] Real-time data collection: collect vehicle status data, environmental perception data and video data in real time.

[0217] Model prediction: Input real-time data into the model to determine whether the current scenario is a typical manual takeover scenario.

[0218] Behavior Monitoring:

[0219] If the model determines that it is a typical manual takeover scenario, the timer starts to monitor whether the safety officer takes over manually within the preset time (for example, 7 seconds).

[0220] If the safety officer fails to take over within the specified time, it is considered dereliction of duty and the relevant data will be recorded.

[0221] Beneficial effects:

[0222] Timely detection of safety officer dereliction of duty: timely detection of safety officer’s slow response or operational errors at critical moments.

[0223] Provide objective evaluation: Based on data analysis, the behavior of safety officers can be objectively evaluated, providing a basis for performance appraisal.

[0224] Through these two steps, a relatively complete safety officer behavior monitoring system can be established. This system can not only improve the efficiency of monitoring the safety officer's behavior, but also provide protection for the safe operation of intelligent connected vehicles.

[0225] In a specific implementation, the embodiment of the present invention can introduce road test data and safety officer demonstration application history data to build a second safety officer supervision model. This new model focuses on the safety officer's reaction time in a specific road scenario, thereby more comprehensively evaluating the safety officer's behavior.

[0226] This can be achieved in the following ways:

[0227] 1. Data collection and preprocessing:

[0228] Road test data: including vehicle sensor data in various road scenarios, vehicle status data, and safety officer takeover behavior data.

[0229] Safety officer demonstration application historical data: including video data of the safety officer in actual driving, vehicle status data and related environmental information.

[0230] Preprocessing: Clean, label, and extract features from the collected data to prepare data for model training.

[0231] 2. Construction of the second safety officer supervision model:

[0232] Data input: The processed road test data and safety officer demonstration application historical data are input into the model.

[0233] Feature extraction: Extract features related to the safety officer’s reaction time, such as road complexity, vehicle speed, obstacle type, safety officer’s line of sight, etc.

[0234] Model training: Train a model that can predict the safety officer's reaction time in a specific scenario based on the input features.

[0235] Threshold setting: Based on empirical data and industry standards, set a preset time threshold as a criterion for judging whether the safety officer's response is timely.

[0236] 3. Real-time monitoring and judgment

[0237] Data input: Input the road information (such as high-precision maps, real-time traffic information) and surveillance videos collected in real time into the model.

[0238] Model prediction: The model predicts the expected reaction time of the safety officer in the current scenario based on the input data.

[0239] Behavior judgment: The predicted reaction time is compared with the preset threshold. If the safety officer's actual reaction time exceeds the threshold, it is judged as a slow reaction.

[0240] Beneficial effects:

[0241] Improve the level of sophistication of monitoring: By introducing road test data and historical data on safety officer demonstration applications, the safety officer's response ability in different scenarios can be evaluated more accurately.

[0242] Enhance the generalization ability of the model: Rich training data helps the model better adapt to various complex road scenarios.

[0243] Discover potential risks: By monitoring the reaction time, potential problems that safety officers may have, such as fatigue driving, lack of concentration, etc., can be discovered in advance.

[0244] In an optional embodiment of the present invention, the initial monitoring video includes an action feature video for characterizing the behavior characteristics of the security guard, and the method further includes:

[0245] Generate a third safety officer supervision model based on the action feature video;

[0246] When the real-time monitoring video is acquired, the real-time monitoring video is input into a third safety officer supervision model, and a third supervision determination result is outputted through the third safety officer supervision model;

[0247] When it is determined through the third supervision judgment result that the safety officer exhibits the target behavior, an alarm message is generated.

[0248] For example, by analyzing the video surveillance data collected by the T-Box vehicle terminal in real time, the fatigue signs or distracted behaviors of the safety officer, such as personnel attention diversion, sleep, irregular operation, etc., can be directly judged to ensure that the behavior of the safety officer is under continuous supervision, that is, the system can detect abnormal behavior and issue an alarm in time. At the same time, intelligent video processing technology can also store video recordings of abnormal behavior or abnormal events as evidence for the safety officer's behavior assessment. The specific method is as follows:

[0249] S31, first, through historical video data analysis, establish typical abnormal behaviors of security personnel, such as personnel attention diversion, sleep, irregular operation, etc. Collect a large amount of historical video data, including normal operation and known abnormal behavior. Through machine learning, label and train the model to identify abnormal behaviors such as attention diversion, sleep, irregular operation, etc.

[0250] The purpose of establishing the abnormal behavior model is to build a model that can automatically identify the abnormal behavior of safety officers, namely the third safety officer supervision model, to provide a basis for subsequent real-time monitoring.

[0251] The process may include:

[0252] Data collection: Collect a large amount of historical video data, including video clips of normal driving and known abnormal behaviors (such as drowsiness, distraction, improper operation, etc.) of safety officers.

[0253] Data labeling: Manually label the video data to identify the various behaviors of the security guards, including normal and abnormal behaviors.

[0254] Feature extraction: Extract features related to abnormal behavior from the video, such as:

[0255] Appearance features: facial expression, eye state, head posture, etc.

[0256] Movement characteristics: human posture, hand movements, eye movements, etc.

[0257] Environmental characteristics: ambient light in the car, driving instruments, etc.

[0258] Model training: Use machine learning algorithms (such as convolutional neural networks (CNNs)) to train the extracted features and build a classification model that can distinguish between normal and abnormal behaviors.

[0259] Beneficial effects:

[0260] Providing a behavioral baseline: By training the model with historical data, a baseline for normal behavior of security officers can be established to facilitate the identification of abnormal behavior.

[0261] Improve detection accuracy: The model can learn various subtle abnormal behavior characteristics to improve detection accuracy.

[0262] Provide a basis for subsequent monitoring: The established model can be applied to real-time video monitoring to promptly detect abnormal behavior of security guards.

[0263] S32, by integrating multiple algorithms to analyze the safety officer's behavior from multiple angles, and then fusing the analysis results of each algorithm to form a comprehensive judgment to determine whether the safety officer has abnormal behavior. Through motion detection algorithms such as frame difference method, background subtraction method and optical flow method, the safety officer and the static background are distinguished and the safety officer in the video screen is identified. The facial features of the safety officer in the video sequence are analyzed through face recognition and tracking algorithms, the safety officer is located and his head movements are continuously tracked to determine fatigue or distraction status. The hand movements of the safety officer are analyzed through gesture recognition algorithms to determine the intention or emotional response that the safety officer's specific gestures may represent. Through the eye tracking algorithm, the safety officer analyzes the position and movement of the eyes to evaluate whether the driver is concentrating or showing signs of fatigue. Through the integrated analysis of the above algorithms, multi-dimensional monitoring of the safety officer's behavior and timely identification of abnormal behavior can be achieved.

[0264] The purpose of multimodal fusion abnormal behavior detection is to analyze the behavior of security personnel from multiple dimensions by integrating multiple algorithms and improving the accuracy of abnormal behavior detection.

[0265] The process may include:

[0266] Multi-algorithm fusion:

[0267] Motion detection: By analyzing the differences between video frames, the security guard's movement trajectory is detected to determine whether he is stationary or moving frequently.

[0268] Face recognition and tracking: Locate the face of the security guard, analyze his / her expression, eye condition, etc., and determine whether he / she is tired or distracted.

[0269] Gesture recognition: Analyze the hand movements of the safety officer to determine whether he is performing irregular operations.

[0270] Eye tracking: Analyze the eye movements of the security guard to determine whether he is paying attention.

[0271] Comprehensive judgment: The analysis results of the above algorithms are integrated, and information from multiple dimensions is comprehensively considered to make a final judgment. For example, if a face recognition result shows that a safety officer is in a fatigued state, and the eye tracking result also shows that he is not paying attention, it can be determined that the safety officer is at risk of fatigue driving.

[0272] Beneficial effects:

[0273] Improve detection accuracy: Through multimodal fusion, the shortcomings of a single algorithm can be made up and the accuracy of detection can be improved.

[0274] Enhanced robustness: Even if an algorithm makes a misjudgment, other algorithms can provide supplementary information to improve the robustness of the system.

[0275] Realize multi-dimensional monitoring: The behavior of safety officers can be monitored from multiple angles to comprehensively evaluate their status.

[0276] Through the above methods, driving safety can be effectively guaranteed. By real-time monitoring of abnormal behaviors such as fatigue and distraction of safety officers, safety officers can be reminded in time to reduce the risk of accidents. Improve system reliability, enhance the ability to monitor safety officer behavior, and improve the reliability of the autonomous driving system. Optimize system design. By analyzing abnormal behavior data, the algorithm and human-computer interaction design of the autonomous driving system can be optimized. Provide data support, which can provide a large amount of data support for safety officer behavior analysis, driving behavior research, etc.

[0277] Through the above steps, an intelligent video analysis system based on multimodal fusion can be established to monitor the behavior of safety officers in real time and detect and warn of abnormal behavior in a timely manner. This system can effectively improve the safety and reliability of intelligent connected vehicles.

[0278] In a specific implementation, the initial monitoring video of the embodiment of the present invention may include all the behavior information of the security guard, providing a data basis for subsequent model training and real-time monitoring.

[0279] The action feature video can be the part directly related to the security officer's behavior extracted from the initial surveillance video, such as hand movements, head movements, etc. By focusing on the action features, the abnormal behavior of the security officer can be identified more accurately.

[0280] The generation process of the third safety officer supervision model is as follows:

[0281] Model construction: Based on the action feature video, a model specifically used to identify the target behavior of the safety officer is trained.

[0282] Target behavior: can be varied, such as dozing off, distraction, improper operation, etc.

[0283] Model training: A method similar to that in S31 is used to annotate and extract features from action feature videos, and then the model is trained.

[0284] The real-time monitoring and judgment methods can be as follows:

[0285] Video input: The real-time surveillance video (or motion feature video) is input into the third model.

[0286] Model output: The model outputs a probability value as the third supervisory judgment result, indicating whether the safety officer is currently performing the target behavior.

[0287] Behavior determination: According to the output probability value, a threshold is set. If it exceeds the threshold, it is considered that the safety officer has the target behavior. When the probability value is higher than the threshold, it can be determined that the safety officer has the target behavior and an alarm message is generated.

[0288] In the embodiment of the present invention, a third safety officer supervision model is generated based on the action feature video; when the real-time monitoring video is acquired, the real-time monitoring video is input into the third safety officer supervision model, and a third supervision determination result is output through the third safety officer supervision model, and the third supervision determination result is used to determine whether the safety officer has a target behavior; the following beneficial effects are achieved:

[0289] Improved behavior recognition accuracy: By focusing on action features, the model can more accurately identify subtle changes in the security officer's actions, thereby improving the detection accuracy of the target behavior.

[0290] Enhanced model targeting: The model is trained for specific target behaviors and is more targeted.

[0291] Simplified model structure: Compared with models that comprehensively consider information from multiple modalities, models that focus on motion features can have a simpler structure and higher computational efficiency.

[0292] Easy to expand: Multiple models can be trained for different target behaviors to detect a variety of abnormal behaviors.

[0293] In an optional embodiment of the present invention, after the step of generating the alarm information, the method further includes:

[0294] Acquire control authority for the vehicle, and control the vehicle based on the control authority.

[0295] In actual applications, if the safety officer is still unable to control the vehicle after the alarm message appears, it means that the safety officer does not have the ability to control the vehicle. At this time, the control authority of the vehicle can be obtained, and the vehicle can be controlled based on the control authority to ensure the safety of the vehicle.

[0296] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0297] Reference Figure 2 , shows a structural block diagram of a vehicle safety officer monitoring device provided in an embodiment of the present invention, which may specifically include the following modules:

[0298] A training data acquisition module 201 is used to acquire training data, wherein the training data includes initial operating state data of the vehicle, initial sensor perception information, and initial monitoring video containing the operating behavior of the safety officer;

[0299] A first safety officer supervision model generating module 202, configured to generate a first safety officer supervision model based on the initial operating state data, the initial sensor perception information and the initial monitoring video;

[0300] The first supervision and determination result output module 203 is used to input the real-time operation status data, the real-time sensor perception information and the real-time monitoring video containing the operation behavior of the safety officer into the first safety officer supervision model when the real-time operation status data, the real-time sensor perception information and the real-time monitoring video are obtained, and output the first supervision and determination result for the safety officer through the first safety officer supervision model;

[0301] The alarm information generating module 204 is used to generate an alarm information when it is determined through the first supervision judgment result that the behavior of the safety officer with respect to the real-time operation status data and the real-time sensor perception information does not meet the preset requirements.

[0302] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0303] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0304] The memory is used to store computer programs;

[0305] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0306] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0307] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0308] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0309] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0310] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0311] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0312] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0313] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0314] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A vehicle safety officer supervision method, characterized in that: include: Acquiring training data, wherein the training data includes initial operating state data of the vehicle, initial sensor perception information, and initial monitoring video containing operating behavior of the safety officer; Generate a first safety officer supervision model based on the initial operating state data, the initial sensor perception information and the initial monitoring video; When real-time operation status data, real-time sensor perception information, and real-time monitoring video containing the operation behavior of the safety officer are obtained, the real-time operation status data, the real-time sensor perception information, and the real-time monitoring video are input into the first safety officer supervision model, and the first supervision judgment result for the safety officer is output through the first safety officer supervision model; When it is determined through the first supervision judgment result that the behavior of the safety officer with respect to the real-time operating status data and the real-time sensor perception information does not meet the preset requirements, an alarm message is generated.

2. The method according to claim 1, characterized in that The training data also includes road test data and safety officer demonstration application history data, and the method also includes: Generate a second safety officer supervision model based on the road test data and the safety officer demonstration application historical data; When the real-time road information and the real-time monitoring video are obtained, the real-time road information and the real-time monitoring video are input into the second safety officer supervision model, and the second supervision determination result is outputted through the second safety officer supervision model; When it is determined through the second supervision judgment result that the trigger time of the safety officer's reaction behavior to the real-time road information is greater than a preset time threshold, an alarm message is generated.

3. The method according to claim 1, characterized in that The initial monitoring video includes an action feature video for characterizing the behavior characteristics of the security guard, and the method further includes: Generate a third safety officer supervision model based on the action feature video; When the real-time monitoring video is acquired, the real-time monitoring video is input into a third safety officer supervision model, and a third supervision determination result is outputted through the third safety officer supervision model; When it is determined through the third supervision judgment result that the safety officer exhibits the target behavior, an alarm message is generated.

4. The method according to claim 1, characterized in that: The first safety officer supervision model includes an input layer, and the step of generating the first safety officer supervision model based on the initial operating state data, the initial sensor perception information and the initial monitoring video includes: Performing a preprocessing operation based on the initial operating state data, the initial sensor perception information and the initial monitoring video; Receiving the initial operating state data, the initial sensor perception information and the initial monitoring video through the input layer; Using a multimodal data fusion method, the initial operating state data, the initial sensor perception information and the initial monitoring video based on different modalities are converted into a unified format, and a feature vector for expressing the initial operating state data, the initial sensor perception information and the initial monitoring video is extracted; A first safety officer supervision model is generated based on the feature vector.

5. The method according to claim 4, characterized in that The step of performing a preprocessing operation based on the initial operating state data, the initial sensor perception information and the initial monitoring video comprises: Performing data cleaning, denoising and normalization operations on the initial operating state data, the initial sensor perception information and the initial monitoring video; Constructing a labeling rule for an interactive automatic labeling tool based on the initial operating state data, the initial sensor perception information, and the initial monitoring video; The initial operating status data, the initial sensor perception information and the initial monitoring video are imported into the interactive automatic labeling tool, and the interactive automatic labeling tool is controlled to output the labeled initial operating status data, the initial sensor perception information and the initial monitoring video.

6. The method according to claim 5, characterized in that The step of adopting a multimodal data fusion method to convert the initial operating state data, the initial sensor perception information and the initial monitoring video based on different modalities into a unified format, and extracting a feature vector for expressing the initial operating state data, the initial sensor perception information and the initial monitoring video includes: Eliminate data whose annotation information is inconsistent with the annotation rules, as well as data with missing annotation information; Convert the annotated initial operating status data, initial sensor perception information and initial monitoring video of different modalities into a unified format; The labeled information is used as a supervisory signal to extract a feature vector for expressing the initial operating state data, the initial sensor perception information, and the initial monitoring video.

7. The method according to any one of claims 1 to 3, characterized in that: After the step of generating the warning information, the method further includes: Acquire control authority for the vehicle, and control the vehicle based on the control authority.

8. A vehicle safety officer monitoring device, characterized in that: include: A training data acquisition module, used to acquire training data, wherein the training data includes initial operating state data of the vehicle, initial sensor perception information, and initial monitoring video containing the operating behavior of the safety officer; A first safety officer supervision model generation module, used to generate a first safety officer supervision model based on the initial operating state data, the initial sensor perception information and the initial monitoring video; A first supervision and determination result output module is used to input the real-time operation status data, the real-time sensor perception information and the real-time monitoring video containing the operation behavior of the safety officer into the first safety officer supervision model when the real-time operation status data, the real-time sensor perception information and the real-time monitoring video are obtained, and output the first supervision and determination result for the safety officer through the first safety officer supervision model; The alarm information generation module is used to generate an alarm message when it is determined through the first supervision judgment result that the behavior of the safety officer with respect to the real-time operating status data and the real-time sensor perception information does not meet the preset requirements.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images