A dangerous behavior recognition and early warning method based on multi-modal analysis

Abstract

The application discloses a dangerous behavior recognition and early warning method based on multi-modal analysis, and relates to the technical field of artificial intelligence, comprising collecting multiple types of data through a device, preprocessing the collected multiple types of data; for each type of data, a pre-trained model is used for feature extraction and fusion to generate a multi-modal fusion feature vector; through a contrast learning framework, the multi-modal fusion feature vector is paired with a risk description text, and a similarity score is calculated; a large language model is trained based on the multi-modal fusion feature vector, real-time analysis and recognition of input data are carried out by using the large language model, and a behavior description is generated; based on the real-time generated behavior description and the similarity score, the risk level is dynamically evaluated, and an early warning is generated; and the large language model is continuously optimized through incremental learning and transfer learning methods. Through advanced machine learning and deep learning algorithms, the application realizes efficient recognition and real-time early warning of potential dangerous behaviors in complex environments.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for identifying and warning of dangerous behaviors based on multimodal analysis. Background Technology

[0002] With the rapid development of technology and the comprehensive digital transformation of society, electronic data has become indispensable in criminal investigations. However, the rapid advancement of information technology has also brought about an extremely complex data environment, resulting in an explosive growth in the amount of data involved in cases. In this process, not only has the amount of data surged, but the types and formats of data have also become increasingly diversified. A large amount of unstructured data, such as audio and video, images, documents, and data from social media, mobile devices, and cloud services, poses unprecedented challenges to traditional criminal investigation data governance and analysis.

[0003] Faced with such a complex data environment, traditional analytical methods prove inadequate. Traditional methods typically rely on manual data screening, processing, and analysis, which is not only labor-intensive and inefficient but also easily constrained by human factors, leading to incomplete and inaccurate results. Furthermore, the manual data processing is often only suitable for structured data, such as tables and records, and has very limited capabilities for processing unstructured data such as audio, video, and images. In addition, when dealing with emerging and complex cases, traditional analytical methods lack specificity and flexibility, failing to quickly adapt to and respond to changing data and information, resulting in time-consuming and slow-to-result processes. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a dangerous behavior identification and early warning method based on multimodal analysis to solve the problems of insufficient multimodal data fusion, low real-time performance, and poor model adaptability in existing methods.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, embodiments of the present invention provide a method for identifying and warning of dangerous behaviors based on multimodal analysis, which includes collecting multiple types of data through a device and preprocessing the collected multiple types of data, wherein the multiple types of data include image, audio, video and text data;

[0008] Based on the preprocessed data of various types, a pre-trained model is used to extract and fuse features for each data type to generate a multimodal fusion feature vector.

[0009] By using a contrastive learning framework, multimodal fusion feature vectors are paired with risk description text, and a similarity score is calculated.

[0010] A large language model is trained based on multimodal fusion feature vectors, and the trained large language model is used to analyze and identify input data in real time to generate behavioral descriptions.

[0011] Based on real-time generated behavioral descriptions and similarity scores, the risk level is dynamically assessed and early warnings are generated.

[0012] Regularly collect new data and continuously optimize large language models through incremental learning and transfer learning methods.

[0013] As a preferred embodiment of the dangerous behavior identification and early warning method based on multimodal analysis described in this invention, the step of extracting and fusing features using a pre-trained model for each type of pre-processed data to generate a multimodal fusion feature vector includes the following steps:

[0014] The preprocessed region of interest image is input into the ResNet-50 model. The last fully connected layer of the ResNet-50 model is removed, and the image feature vector is extracted through a global average pooling layer.

[0015] The 13-dimensional MFCC feature vector (MFCC) output after preprocessing is input into the BiLSTM network. The MFCC sequence is processed by the bidirectional LSTM unit to extract the final hidden state of the BiLSTM network and generate the audio feature vector.

[0016] The preprocessed, standardized keyframe sequence is input into the I3D model, and the video feature vector is extracted through the I3D model.

[0017] The cleaned text output from the preprocessing module is input into the pre-trained BERT model. The BERT model generates a context embedding vector for each word. The pooling operation is used to aggregate the embedding vectors of all words into a fixed-dimensional text feature vector. The text feature vector is then adjusted to 512 dimensions through linear projection.

[0018] The extracted image feature vectors, audio feature vectors, video feature vectors, and text feature vectors are L2 normalized.

[0019] The normalized modal feature vectors are concatenated in sequence to form a unified multimodal representation vector;

[0020] A multi-head self-attention mechanism is introduced to weight and integrate the multimodal representation vectors to obtain the final multimodal fusion feature vector.

[0021] As a preferred embodiment of the dangerous behavior identification and early warning method based on multimodal analysis described in this invention, the step of pairing multimodal fusion feature vectors with risk description text and calculating a similarity score through a contrastive learning framework includes the following steps:

[0022] Given a dataset, for each sample, its feature vector is fused with its corresponding risk description text to form a positive sample pair. For each positive sample pair, risk description texts from other samples are randomly selected to form a negative sample pair, resulting in a dataset containing... A number of positive sample pairs and P A set of sample pairs containing negative sample pairs, wherein P The number of negative samples corresponding to each positive sample;

[0023] A shared Transformer encoder is used to encode both the multimodal fusion feature vector and the risk description text simultaneously, generating a unified embedding space.

[0024] Cosine similarity is used to measure the similarity between the fused feature embedding and the text description embedding;

[0025] A contrastive loss function in the form of cross-entropy is used, combined with a temperature parameter to adjust the similarity distribution, to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs;

[0026] For each fused feature vector, calculate its similarity score with all risk description texts.

[0027] As a preferred embodiment of the dangerous behavior identification and early warning method based on multimodal analysis described in this invention, the step of training a large language model based on multimodal fusion feature vectors, and using the trained large language model to perform real-time analysis and identification of input data to generate behavior descriptions includes the following steps.

[0028] A linear projection layer is used to map the high-dimensional multimodal fusion feature vector to the input dimension required by LLM;

[0029] Construct a multimodal dataset with risk labels, and divide the multimodal dataset into a training set, a validation set, and a test set;

[0030] For each sample, define the cross-entropy loss, and average the losses of all training samples to obtain the overall cross-entropy loss;

[0031] Initialize all parameters of the LLM, configure the Adam optimizer, set the learning rate and other hyperparameters, perform forward propagation for each batch in the training set, calculate the predicted probability of the LLM model, and calculate the cross-entropy loss of the current batch. Calculate the gradient of the loss function with respect to the LLM model parameters using the backpropagation algorithm, update the model parameters using the Adam optimizer, and repeat the process of forward propagation, loss calculation, backpropagation and parameter update for multiple rounds of iterative training until the LLM model reaches the convergence condition on the validation set and meets the preset performance indicators.

[0032] The vector InputLLM, transformed by the linear projection layer, is used as input to the trained LLM model. Through the forward propagation of the LLM, the behavioral description output by the LLM model is obtained. The generated behavioral description is then post-processed to obtain the final behavioral description.

[0033] As a preferred embodiment of the dangerous behavior identification and early warning method based on multimodal analysis described in this invention, the step of dynamically assessing the risk level and generating an early warning based on real-time generated behavior descriptions and similarity scores includes the following steps:

[0034] Set the initial threshold and sliding window size In time step Collect recent For each sample, calculate the similarity score and the mean and standard deviation of these similarity scores;

[0035] Calculate the current dynamic threshold based on the mean and standard deviation, based on the most recent... The similarity score distribution of each sample is automatically adjusted with dynamic thresholds.

[0036] For the sample at the current time step, obtain its highest similarity score. To determine whether the current sample belongs to the risky behavior category, it is represented as follows:

[0037]

[0038]

[0039] in, Indicates time step The risk assessment results This indicates that the current sample is behaving normally. The risk category with the highest similarity score. For dynamic thresholds, For the current sample and risk category Similarity score;

[0040] like ≠ If the condition is met, an alert will be triggered; otherwise, no alert will be generated.

[0041] The generated alert content is sent to relevant personnel through preset communication channels, and detailed information for each alert is recorded.

[0042] As a preferred embodiment of the dangerous behavior identification and early warning method based on multimodal analysis described in this invention, the step of periodically collecting new data and continuously optimizing the large language model through incremental learning and transfer learning methods includes the following steps.

[0043] Define the time interval for data collection, collect multimodal data periodically, use pre-trained models to perform preliminary annotation on new data, and review and correct the automatic annotation results;

[0044] The labeled and validated multimodal data are integrated to form a new dataset;

[0045] A gradient-based online learning algorithm is used to gradually update the model parameters through incremental training;

[0046] Select a pre-trained language model that is highly relevant to the target application domain, and fine-tune it on a domain-specific dataset using multi-round training and hyperparameter optimization strategies.

[0047] As a preferred embodiment of the dangerous behavior identification and early warning method based on multimodal analysis described in this invention, the preprocessing of the collected data of various types includes the following steps:

[0048] Denoising, standardization, and semantic segmentation are performed on image data;

[0049] Noise reduction and feature extraction are performed on the audio data;

[0050] Perform frame extraction and frame processing on video data;

[0051] Perform text cleaning on the text data.

[0052] In a second aspect, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements any step of the dangerous behavior identification and early warning method based on multimodal analysis as described in the first aspect of the present invention.

[0053] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the dangerous behavior identification and early warning method based on multimodal analysis as described in the first aspect of the present invention.

[0054] The beneficial effects of this invention are as follows: by integrating features of different types of data through multimodal fusion, it makes up for the shortcomings of single modality, provides a more comprehensive and multi-dimensional behavioral description, improves recognition accuracy, and realizes rapid analysis and behavior recognition of real-time input data through the efficient processing capabilities of LLM. The detailed behavioral description generated by LLM not only improves interpretability, but also provides rich semantic information for risk assessment, supporting more accurate risk judgment. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart of the dangerous behavior identification and early warning method based on multimodal analysis in Example 1.

[0057] Figure 2 This is a flowchart of generating multimodal fusion feature vectors in Example 1. Detailed Implementation

[0058] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0059] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0060] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0061] Example 1, referring to Figure 1 and Figure 2 This is the first embodiment of the present invention, which provides a method for identifying and warning of dangerous behaviors based on multimodal analysis, including the following steps:

[0062] S1. Collect various types of data through the device and preprocess the collected data.

[0063] S1.1. Real-time acquisition of image, audio, video and text data via camera and microphone devices.

[0064] S1.2. Let the input image be I, with dimensions H×W×C, where H is the height, W is the width, and C is the number of color channels (usually 3). Apply a 3x3 median filter to each color channel of each input image to obtain the denoised image.

[0065] It should be noted that the applied 3x3 median filter, compared to the Gaussian filter, is better at preserving the edge information of the image when removing salt-and-pepper noise, and is suitable for common noise types in monitoring images;

[0066] Calculate the minimum and maximum pixel values ​​of the denoised image, normalize each pixel value, scale the pixel value to the range of [0,1] to ensure the uniformity of the input data, and obtain the standardized image;

[0067] A pre-trained U-Net model is used for semantic segmentation to accurately identify regions of interest (ROIs) in images, such as human bodies and vehicles, and to filter background noise, thereby improving the efficiency and accuracy of subsequent feature extraction. Specifically, the standardized image is input into the pre-trained U-Net model, which outputs a binary segmentation mask. The ROI is extracted through the mask to obtain an image containing only the ROI, while the remaining regions are set to 0.

[0068] S1.3 Perform a short-time Fourier transform (STFT) on the input audio signal to obtain the spectrum. Estimate the noise spectrum during the silent period (the period without speech or main sound source). Subtract the noise spectrum from the spectrum of the input signal to obtain the clean spectrum. Convert the clean spectrum back to the time domain through inverse STFT to obtain the cleaned audio signal.

[0069] It should be noted that the Spectral Subtraction method effectively eliminates background noise and improves the clarity of audio signals by estimating the background noise spectrum and subtracting the noise spectrum from the spectrum of the input signal.

[0070] 13-dimensional Mel frequency cepstral coefficients (MFCCs) are extracted as audio feature representations. STFT is applied to the denoised audio signal to generate a spectrogram. The spectrum is converted to the Mel frequency scale. The logarithm of the Mel spectrogram is taken to enhance the features of the low-energy region. Discrete cosine transform (DCT) is applied to extract the first 13 MFCC features, denoted as MFCC.

[0071] MFCCs are widely used features in audio processing, effectively capturing the time-frequency characteristics of audio signals and suitable for speech and audio classification tasks.

[0072] S1.4. Use the histogram comparison method to sample key frames. This method automatically detects scene changes in the video by comparing the differences in color histograms of adjacent frames, ensuring that key frames can capture important visual information.

[0073] Specifically, frames are extracted from the input video at a frame rate of 2 frames per second. The color histogram difference between adjacent frames is calculated. If the color histogram difference between adjacent frames exceeds a predefined threshold, the frame is marked as a key frame. All key frames are collected to form a frame sequence.

[0074] Each keyframe is adjusted to a uniform size (224x224 pixels), bilinear interpolation is used to maintain image quality, and image preprocessing techniques, including normalization and center cropping, are applied to obtain the standardized frames.

[0075] S1.5 For text data, use Natural Language Processing (NLP) technology to remove stop words and punctuation marks. Specifically, perform word segmentation on the input text to obtain a word sequence, and use a predefined stop word list to remove stop words and punctuation marks to obtain the cleaned text.

[0076] S2. Based on the preprocessed data of various types, for each data type, a pre-trained model is used to extract and fuse features to generate a multimodal fusion feature vector, including the following steps.

[0077] S2.1 Input the preprocessed region of interest image into the ResNet-50 model, remove the last fully connected layer of the ResNet-50 model to obtain the feature representation of the intermediate layer, extract the image feature vector through the global average pooling layer, and perform L2 normalization on the extracted image feature vector to ensure that the norm of the feature vector is 1, thereby improving the stability of subsequent fusion.

[0078] It should be noted that a pre-trained ResNet-50 model is used for image feature extraction. Due to its deep network structure and residual connections, ResNet-50 can effectively capture high-level semantic information of images and is suitable for risk behavior recognition in complex scenes.

[0079] The 13-dimensional MFCC feature vector MFCC output after preprocessing is input into the BiLSTM network. The MFCC sequence is processed by the bidirectional LSTM unit to obtain the hidden state at each time step. The final hidden state of the BiLSTM network is extracted to generate the audio feature vector. The audio feature vector is then L2 normalized.

[0080] It should be noted that a bidirectional long short-term memory network (BiLSTM) is used to process MFCC features and generate audio feature vectors. BiLSTM can capture the temporal information of the audio signal and improve the expressive power of audio features.

[0081] The pre-trained I3D (Inflated 3D ConvNet) model is used to extract the spatiotemporal features of the video. The I3D model captures the spatiotemporal dynamic information in the video through 3D convolution and is suitable for behavior recognition in complex video scenes. The standardized key frame sequence output after preprocessing is input into the I3D model, and the video feature vector is extracted by the I3D model. The video feature vector is then L2 normalized.

[0082] The pre-trained BERT model is used to generate text feature vectors. The BERT model, through a bidirectional Transformer architecture, can effectively capture the contextual semantic information of the text and is suitable for feature extraction of complex text descriptions. The cleaned text output from the preprocessing module is input into the pre-trained BERT model. The BERT model generates a context embedding vector for each word. Pooling operations (such as taking the [CLS] vector from the last layer) are used to aggregate the embedding vectors of all words into a fixed-dimensional text feature vector. The text feature vector is adjusted to 512 dimensions through linear projection to be consistent with the dimensions of other modal features. The text feature vector is then L2 normalized.

[0083] S2.2. Concatenate the normalized modal feature vectors in sequence to form a unified multimodal representation vector. ;

[0084] A multi-head self-attention mechanism is introduced to weight and integrate multimodal representation vectors. The multi-head self-attention mechanism can dynamically adjust the weights of each modality, highlight the features of key modalities and suppress the interference of noisy modalities based on the contextual information of the input data.

[0085] Specifically, the number of heads in the multi-head self-attention mechanism is set to... The dimensions of each head are ,Will Through three linear transformation matrices , , Generate a matrix of query, key, and value elements, represented as follows:

[0086]

[0087]

[0088]

[0089] in, The dimensions of the key vector and query vector. The dimension of the value vector. = It is usually set to 64. , , respectively query ,key ,value The linear transformation weight matrix;

[0090] Will , , Divided into Each head has a dimension of [number]. , is represented as ,

[0091]

[0092]

[0093]

[0094] in, , , They represent the first The query submatrix, key submatrix, and value submatrix of each attention head. This indicates the index of the attention head currently being processed. =1,2,…, ;

[0095] For each head Calculate attention weights , is represented as ,

[0096]

[0097] in, Attention weights represent the relevance of the query to the key. Convert the similarity scores into a probability distribution. The similarity score between the query and the key is calculated; the attention weight matrix is ​​used to weight the vector. This generates the output of the current attention head;

[0098] Calculate the output of each head for,

[0099]

[0100] The outputs of all heads are concatenated to form a concatenated feature vector. The concatenated feature vector is then mapped back to the original dimension through a linear transformation matrix to generate a fused feature vector. Layer normalization and ReLU activation are then applied to the fused feature vector to enhance the expressive power and stability of the features, resulting in the final multimodal fused feature vector.

[0101] It should be noted that the multi-head self-attention mechanism can capture the complex relationships between multimodal features in parallel from different representation subspaces, enhance the expressive power of feature fusion, dynamically adjust the weights of each modality, highlight the features of key modalities based on contextual information, suppress the interference of noisy modalities, and improve the recognition accuracy and robustness of the overall system.

[0102] S3. Using a contrastive learning framework, pair the multimodal fusion feature vector with the risk description text and calculate a similarity score, including the following steps:

[0103] S3.1, Define a batch dataset.

[0104]

[0105] in, For the first The fused feature vector of each sample, Provide the corresponding risk behavior description text;

[0106] For each sample fuse them with feature vectors Its corresponding risk description text Forming positive sample pairs ;

[0107] For each positive sample pair Randomly select other samples ≠ Risk description text Forming negative sample pairs , forming a collection A number of positive sample pairs and P A set of sample pairs containing negative sample pairs, wherein P The number of negative samples corresponding to each positive sample;

[0108] S3.2 uses a shared Transformer encoder to simultaneously encode the multimodal fusion feature vector and the risk description text, generating a unified embedding space;

[0109] Specifically, a shared Transformer encoder is defined, whose parameters are shared across all inputs, and the feature vectors are fused. Input a Transformer encoder to generate fused feature embeddings , is represented as ,

[0110]

[0111] in, For the embedded dimension;

[0112] Describe the risky behavior text The text feature vectors generated by the BERT model are adjusted for dimensionality through a linear transformation and then input into the Transformer encoder to generate text description embeddings. , is represented as ,

[0113]

[0114] in, The text feature vector after linear projection;

[0115] The generated embedding vectors are L2 normalized to ensure that the norm of the embedding vectors is 1, thereby improving the stability of similarity calculation. This is represented as follows:

[0116]

[0117] in, , These are the normalized fusion feature embedding and the text description embedding, respectively;

[0118] S3.3. Use cosine similarity to measure the similarity between the fused feature embedding and the text description embedding;

[0119] For each pair of embedding vectors ( , ), calculate cosine similarity , is represented as ,

[0120]

[0121] Since the vectors have been normalized, the formula simplifies to a vector dot product;

[0122] For a batch One sample, generate one Similarity rating matrix , is represented as ,

[0123]

[0124] in, Indicates the first The first fusion feature and the second Similarity of text descriptions This is the matrix transpose operator, which swaps the rows and columns of a matrix.

[0125] S3.4. Employ a cross-entropy form of the contrastive loss function, combined with temperature parameters to adjust the similarity distribution, maximize the similarity of positive sample pairs, and minimize the similarity of negative sample pairs, thereby optimizing the model's discriminative ability.

[0126] Specifically, set the temperature parameters. The smoothness of the similarity distribution is controlled, typically set between 0.07 and 0.2 for each sample. The similarity between their positive sample pairs is The similarity of negative sample pairs is ( ≠ The contrast loss for each sample is calculated using the softmax function. The expression is,

[0127]

[0128] The total contrast loss is obtained by averaging the contrast losses of all samples, expressed as follows:

[0129]

[0130] By maximizing positive sample pairs The similarity is minimized for all negative sample pairs. By assessing the similarity, the model learns more discriminative feature representations and minimizes the contrastive loss using gradient descent optimization algorithms (such as the Adam optimizer). Update the parameters of the Transformer encoder.

[0131] For each fused feature vector Calculate its relationship with all risk description texts. Similarity score For each sample This forms a similarity score vector.

[0132] S4. Train a large-scale language model based on multimodal fusion feature vectors, and use the trained large-scale language model to perform real-time analysis and recognition of input data to generate behavioral descriptions, including the following steps.

[0133] S4.1. A linear projection layer is used to map the high-dimensional multimodal fusion feature vector to the input dimension required by LLM, ensuring that the dimension of the input data matches the requirements of LLM, while maintaining the linear relationship of the features and the integrity of the information.

[0134] Specifically, a linear projection layer is set up to obtain the multimodal fusion feature vector, and the multimodal fusion feature vector is then transformed using a linear transformation. Mapping to LLM, we obtain the mapped LLM input vector. .

[0135] S4.2 Construct a multimodal dataset with risk labels, and divide the multimodal dataset into a training set, a validation set, and a test set;

[0136] Furthermore, a large number of samples are collected from multimodal data sources. Each sample contains a preprocessed and multimodal fusion-derived vector and a corresponding risk label. For each sample... By using a linear projection layer Convert to This forms a labeled dataset, represented as follows:

[0137]

[0138] in, For the sample size, ∈ For the first Risk category labels for each sample The total number of risk categories. This indicates the first input into the LLM. Data for one sample;

[0139] Dataset The model is divided into training, validation, and test sets to ensure the independence and effectiveness of model training, validation, and evaluation.

[0140] For each sample, define the cross-entropy loss, and average the losses of all training samples to obtain the overall cross-entropy loss;

[0141] The Adam optimizer (Adaptive Moment Estimation) is used to optimize the model parameters. All parameters of the LLM are initialized, including the weights and biases of the linear projection layer and the parameters inside the LLM. The Adam optimizer is configured, and the learning rate and other hyperparameters are set. Forward propagation is performed on each batch in the training set to calculate the predicted probability of the LLM model and the cross-entropy loss of the current batch. The gradient of the loss function with respect to the LLM model parameters is calculated through the backpropagation algorithm. The model parameters are updated using the Adam optimizer. The process of forward propagation, loss calculation, backpropagation and parameter update is repeated for multiple rounds of iterative training until the LLM model reaches the convergence condition and meets the preset performance indicators on the validation set. The loss and accuracy on the validation set are monitored. If the performance does not improve within a certain number of consecutive epochs, training is stopped early to prevent overfitting.

[0142] It should be noted that the Adam optimizer combines momentum and adaptive learning rate adjustment, enabling efficient convergence in large-scale data and high-dimensional parameter spaces.

[0143] S4.3, Transform the vector after the linear projection layer As input, it is fed into the trained LLM model. Through the forward propagation of the LLM, the behavioral description output by the LLM model is obtained. The generated behavioral description is post-processed, such as grammar correction and keyword extraction, to ensure the accuracy and readability of the description, and the final behavioral description is obtained for early warning generation and risk response.

[0144] S5. Based on real-time generated behavioral descriptions and similarity scores, dynamically assess the risk level and generate early warnings, including the following steps:

[0145] S5.1, Employ dynamic thresholding based on a sliding window. The assessment method dynamically adjusts the risk assessment threshold by monitoring the similarity scores of the most recent W samples, adapting to changes in the environment and fluctuations in risk levels.

[0146] Specifically, set an initial threshold. This serves as the baseline risk level for the start of operation;

[0147] Set the sliding window size In time step Collect recent The similarity scores for each sample are calculated, and the mean of these similarity scores is obtained. and standard deviation The expression is,

[0148]

[0149] in, For time steps The similarity score between the sample and its risk category;

[0150] Calculate the current dynamic threshold based on the mean and standard deviation. The expression is,

[0151]

[0152] in, To adjust parameters (such as) 1.5), used to control the sensitivity and risk tolerance of the threshold;

[0153] Dynamic threshold It can automatically adjust based on the similarity score distribution of the most recent W samples, ensuring adaptability and flexibility to risk levels under different environments, and obtain the current time step. dynamic threshold It can be used for risk assessment and early warning generation;

[0154] S5.2 For the sample at the current time step, obtain its highest similarity score. To determine whether the current sample belongs to the risky behavior category, it is represented as follows:

[0155]

[0156]

[0157] in, Indicates time step The risk assessment results This indicates that the current sample is behaving normally. The risk category with the highest similarity score. For dynamic thresholds, For the current sample and risk category Similarity score;

[0158] like ≠ If the condition is met, an alert will be triggered; otherwise, no alert will be generated.

[0159] The generated alert content includes the following information: Risk behavior description: A detailed description of the detected risk behavior; Risk category: Indicates the category to which the risk behavior belongs. Mitigation recommendations: Based on the risk category, provide corresponding mitigation and response recommendations to guide relevant personnel in taking action;

[0160] The generated alerts are sent to relevant personnel through pre-defined communication channels (such as SMS, email, and application notifications), and detailed information for each alert is recorded. The detailed information for each alert includes timestamp, risk category, similarity score, mitigation measures, etc., to facilitate subsequent auditing and analysis.

[0161] For risk categories It is a predefined set of risk behavior categories based on actual application scenarios and security management needs. ={ , ,…, }, each This represents a specific type of risky behavior. For example, in intelligent traffic management, risk categories might include "driving against traffic," "running a red light," and "dangerous overtaking," or in risk data analysis of intelligent scenarios, risk categories might include "abnormal transactions," "abnormal travel," and "abnormal communication." During model training, each risk category... Each risk description text has a corresponding risk description text. Through comparative learning, the model learns the similarity relationship between the multimodal fusion vector and the risk description text, ensuring that samples of the same category have high similarity in the feature space. In the inference stage, the most matching risk category is determined by calculating the similarity score between the fusion feature vector of the current sample and the text description embedding vectors of all risk categories.

[0162] S6. Regularly collect new data and continuously optimize large-scale language models through incremental learning and transfer learning methods, including the following steps.

[0163] S6.1 Define the data collection time interval, such as once a month, to ensure that the model can adapt to new data in a timely manner, collect multimodal data regularly, use the pre-trained model to perform preliminary labeling of new data, identify potential risky behaviors, and review and correct the automatic labeling results to ensure the accuracy of data labeling.

[0164] The labeled and validated multimodal data are integrated to form a new dataset, which is then stored in a secure and scalable database to ensure data accessibility and integrity.

[0165] A gradient-based online learning algorithm is adopted to gradually update the model parameters through incremental training, ensuring the model's adaptability to new data while retaining the memory of old knowledge.

[0166] Specifically, the new dataset is added to the existing training set to form an updated training set. A training subset and a validation subset are then divided from the updated training set to ensure the diversity and balance of the data distribution. An optimization algorithm suitable for online learning, such as the Adam optimizer, is selected, and appropriate learning rates and other hyperparameters are configured. The training subset is then input into the model to calculate the predicted probabilities. The contrastive loss function is used to calculate the loss of the model on the training set. The gradient of the loss function with respect to the model parameters is calculated, and the model parameters are updated using gradient descent.

[0167] The model performance is evaluated on the validation subset, and metrics such as accuracy, recall, and F1 score are calculated to ensure that the model performs in a balanced manner on new and old data. Knowledge distillation or elastic weight consolidation (EWC) techniques are used to ensure that the model does not lose its memory of old knowledge when learning new data.

[0168] It should be noted that online learning algorithms can quickly adapt to new data, improve the model's real-time response capabilities, and ensure that the model retains old knowledge while learning new knowledge through techniques that prevent catastrophic forgetting, thus maintaining overall performance. Incremental learning methods do not require retraining the entire model, saving computing resources and time.

[0169] S6.2 Select a pre-trained language model that is highly relevant to the target application domain, and improve the model's ability to detect risky behaviors in that domain on a specific dataset;

[0170] Furthermore, select a pre-trained model related to the target domain. For example, if the target domain is criminal investigation analysis, select a pre-trained BERT model / RoBERTa model. Collect and organize multimodal data and risk behavior samples in the specific domain to form a domain-specific dataset. Perform the same data preprocessing steps as the initial training on the domain-specific dataset, including noise reduction, standardization, and feature extraction, to ensure data quality and consistency. Input the domain-specific dataset into the pre-trained model for fine-tuning training.

[0171] Furthermore, multi-round training and hyperparameter optimization strategies are employed to ensure the model's efficient performance and generalization ability on specific domain data.

[0172] This embodiment also provides a computer device applicable to the dangerous behavior identification and early warning method based on multimodal analysis, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the dangerous behavior identification and early warning method based on multimodal analysis as proposed in the above embodiment.

[0173] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0174] This embodiment also provides a storage medium storing a computer program. When executed by a processor, the program implements the method for identifying and warning of dangerous behaviors based on multimodal analysis as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0175] In summary, this invention achieves high-quality, multi-dimensional data acquisition and preparation by collecting and preprocessing various types of data through equipment. This ensures the clarity and consistency of the input data, eliminates noise and interference factors, and utilizes the feature extraction capabilities of the pre-trained model to capture key features of each modality, enhancing the expressiveness and discriminative power of feature vectors. Multimodal fusion, by integrating features from different types of data, compensates for the shortcomings of a single modality, providing a more comprehensive and multi-dimensional behavioral description and improving recognition accuracy. The multi-head self-attention mechanism intelligently weights features from each modality, optimizing the fusion process and ensuring the prominence of key features and suppression of noise. Contrastive learning, by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, significantly enhances the model's ability to distinguish between different risky behaviors. The LLM (Large Language Model) enhances the ability to reduce false recognition rates. The shared Transformer encoder ensures consistency between multimodal features and textual descriptions within the same embedding space, improving the integration and comparability of feature representations. By introducing a temperature parameter to adjust the similarity distribution, the model maintains stable performance even with varying data distributions and noise levels. Leveraging the efficient processing capabilities of LLM, it enables rapid analysis and behavior recognition of real-time input data, meeting the demands for immediate response in practical applications. The detailed behavioral descriptions generated by LLM not only improve interpretability but also provide rich semantic information for risk assessment, supporting more accurate risk judgment. The large-scale language model, trained through multiple iterations, possesses excellent generalization capabilities, adapting to various complex and dynamic risk behavior patterns.

[0176] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for identifying and warning of dangerous behaviors based on multimodal analysis, characterized in that: include, The device collects various types of data and preprocesses the collected data, including image, audio, video, and text data. Based on the preprocessed data of various types, a pre-trained model is used to extract and fuse features for each data type to generate a multimodal fusion feature vector. By using a contrastive learning framework, multimodal fusion feature vectors are paired with risk description text, and a similarity score is calculated. A large language model is trained based on multimodal fusion feature vectors, and the trained large language model is used to analyze and identify input data in real time to generate behavioral descriptions. Based on real-time generated behavioral descriptions and similarity scores, the risk level is dynamically assessed and early warnings are generated. Regularly collect new data and continuously optimize large language models through incremental learning and transfer learning methods; The process of pairing multimodal fused feature vectors with risk description text and calculating a similarity score using a contrastive learning framework includes the following steps: Given a dataset, for each sample, its feature vector is fused with its corresponding risk description text to form a positive sample pair. For each positive sample pair, risk description texts from other samples are randomly selected to form a negative sample pair, resulting in a dataset containing... A number of positive sample pairs and P A set of sample pairs containing negative sample pairs, wherein P The number of negative samples corresponding to each positive sample; A shared Transformer encoder is used to encode both the multimodal fusion feature vector and the risk description text simultaneously, generating a unified embedding space. Cosine similarity is used to measure the similarity between the fused feature embedding and the text description embedding; A contrastive loss function in the form of cross-entropy is used, combined with a temperature parameter to adjust the similarity distribution, to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs; For each fused feature vector, calculate its similarity score with all risk description texts; The method involves training a large-scale language model based on multimodal fusion feature vectors, and then using the trained model to perform real-time analysis and recognition of input data to generate behavioral descriptions. This includes the following steps: A linear projection layer is used to map the high-dimensional multimodal fusion feature vector to the input dimension required by LLM; Construct a multimodal dataset with risk labels, and divide the multimodal dataset into a training set, a validation set, and a test set; For each sample, define the cross-entropy loss, and average the losses of all training samples to obtain the overall cross-entropy loss; Initialize all parameters of the LLM, configure the Adam optimizer, set the learning rate and other hyperparameters, perform forward propagation for each batch in the training set, calculate the predicted probability of the LLM model, and calculate the cross-entropy loss of the current batch. Calculate the gradient of the loss function with respect to the LLM model parameters using the backpropagation algorithm, update the model parameters using the Adam optimizer, and repeat the process of forward propagation, loss calculation, backpropagation and parameter update for multiple rounds of iterative training until the LLM model reaches the convergence condition on the validation set and meets the preset performance indicators. The vector InputLLM, transformed by the linear projection layer, is used as input to the trained LLM model. Through the forward propagation of the LLM, the behavioral description output by the LLM model is obtained. The generated behavioral description is then post-processed to obtain the final behavioral description.

2. The dangerous behavior identification and early warning method based on multimodal analysis as described in claim 1, characterized in that: Based on the preprocessed data of various types, for each data type, a pre-trained model is used to extract and fuse features to generate a multimodal fused feature vector, including the following steps. The preprocessed region of interest image is input into the ResNet-50 model. The last fully connected layer of the ResNet-50 model is removed, and the image feature vector is extracted through a global average pooling layer. The 13-dimensional MFCC feature vector (MFCC) output after preprocessing is input into the BiLSTM network. The MFCC sequence is processed by the bidirectional LSTM unit to extract the final hidden state of the BiLSTM network and generate the audio feature vector. The preprocessed, standardized keyframe sequence is input into the I3D model, and the video feature vector is extracted through the I3D model. The cleaned text output from the preprocessing module is input into the pre-trained BERT model. The BERT model generates a context embedding vector for each word, and uses pooling operations to aggregate the embedding vectors of all words into a fixed-dimensional text feature vector. The text feature vector is then adjusted to 512 dimensions through linear projection. The extracted image feature vectors, audio feature vectors, video feature vectors, and text feature vectors are L2 normalized. The normalized modal feature vectors are concatenated in sequence to form a unified multimodal representation vector; A multi-head self-attention mechanism is introduced to weight and integrate the multimodal representation vectors to obtain the final multimodal fusion feature vector.

3. The dangerous behavior identification and early warning method based on multimodal analysis as described in claim 2, characterized in that: The process of dynamically assessing risk levels and generating early warnings based on real-time generated behavioral descriptions and similarity scores includes the following steps: Set the initial threshold and sliding window size In time step Collect recent For each sample, calculate the similarity score and the mean and standard deviation of these similarity scores; Calculate the current dynamic threshold based on the mean and standard deviation, based on the most recent... The similarity score distribution of each sample is automatically adjusted with dynamic thresholds. For the sample at the current time step, obtain its highest similarity score. To determine whether the current sample belongs to the risky behavior category, it is represented as follows: in, Indicates time step The risk assessment results This indicates that the current sample is behaving normally. The risk category with the highest similarity score. For dynamic thresholds, For the current sample and risk category Similarity score; like ≠ If the condition is met, an alert will be triggered; otherwise, no alert will be generated. The generated alert content is sent to relevant personnel through preset communication channels, and detailed information for each alert is recorded.

4. The dangerous behavior identification and early warning method based on multimodal analysis as described in claim 3, characterized in that: The process of periodically collecting new data and continuously optimizing large language models through incremental learning and transfer learning methods includes the following steps: Define the time interval for data collection, collect multimodal data periodically, use pre-trained models to perform preliminary annotation on new data, and review and correct the automatic annotation results; The labeled and validated multimodal data are integrated to form a new dataset; A gradient-based online learning algorithm is used to gradually update the model parameters through incremental training; Select a pre-trained language model that is highly relevant to the target application domain, and fine-tune it on a domain-specific dataset using multi-round training and hyperparameter optimization strategies.

5. The dangerous behavior identification and early warning method based on multimodal analysis as described in claim 1, characterized in that: The preprocessing of the collected data of various types includes the following steps: Denoising, standardization, and semantic segmentation are performed on image data; Noise reduction and feature extraction are performed on the audio data; Perform frame extraction and frame processing on video data; Perform text cleaning on the text data.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the dangerous behavior identification and early warning method based on multimodal analysis as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the dangerous behavior identification and early warning method based on multimodal analysis as described in any one of claims 1 to 5.

Images