Dangerous behavior identification and early warning method based on multi-modal analysis

Abstract

The invention discloses a dangerous behavior identification and early warning method based on multi-modal analysis, and relates to the technical field of artificial intelligence, and the method comprises the steps: collecting various types of data through equipment, and carrying out the preprocessing of the collected various types of data; for each data type, performing feature extraction and fusion by adopting a pre-training model to generate a multi-modal fusion feature vector; matching the multi-modal fusion feature vector with the risk description text through a contrast learning framework, and calculating a similarity score; training a large-scale language model based on the multi-modal fusion feature vector, and performing real-time analysis and identification on the input data by using the large-scale language model to generate behavior description; dynamically evaluating the risk level based on the behavior description and the similarity score generated in real time, and generating an early warning; and continuously optimizing the large language model through incremental learning and transfer learning methods. According to the invention, through advanced machine learning and deep learning algorithms, efficient identification and real-time early warning of potential dangerous behaviors in a complex environment are realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a dangerous behavior recognition and early warning method based on multimodal analysis. Background Art

[0002] With the rapid development of science and technology and the comprehensive digital transformation of society, the status and role of electronic data in case investigation have become indispensable. However, the rapid advancement of information technology has also brought about an extremely complex data environment, resulting in an explosive growth in the amount of data involved in cases. In this process, not only has the amount of data increased dramatically, but the types and formats of data have also become more diverse. Among them, a large amount of unstructured data such as audio and video, pictures, documents, and data from social media, mobile devices, and cloud services have brought unprecedented challenges to the traditional case investigation data governance and research and analysis work.

[0003] Faced with such a complex data environment, traditional analysis methods seem to be unable to cope with it. Traditional analysis methods usually rely on manual screening, processing and analysis of data, which is not only labor-intensive and inefficient, but also easily restricted by human factors, resulting in incomplete and inaccurate analysis results. At the same time, the process of manually combing data is often only applicable to structured data, such as tables and records, and the processing ability for unstructured data such as audio, video and pictures is very limited. In addition, in the face of emerging complex scenario cases, traditional analysis methods lack pertinence and flexibility, and cannot quickly adapt to and respond to changing data and information, resulting in a time-consuming and slow processing process. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a dangerous behavior identification and early warning method based on multimodal analysis to solve the problems of insufficient multimodal data fusion, low real-time performance and poor model adaptability.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, an embodiment of the present invention provides a dangerous behavior identification and warning method based on multimodal analysis, which includes collecting multiple types of data through a device and preprocessing the collected multiple types of data, wherein the multiple types of data include image, audio, video and text data; Based on the preprocessed data of various types, for each data type, a pre-trained model is used to extract and fuse features to generate a multi-modal fusion feature vector; Through the contrastive learning framework, the multimodal fusion feature vector is paired with the risk description text and the similarity score is calculated; Based on the multimodal fusion feature vector, a large language model is trained, and the trained large language model is used to analyze and identify the input data in real time to generate a behavior description; Dynamically assess risk levels and generate early warnings based on real-time generated behavior descriptions and similarity scores; New data is collected regularly to continuously optimize large language models through incremental learning and transfer learning methods.

[0007] As a preferred solution of the dangerous behavior identification and early warning method based on multimodal analysis of the present invention, wherein: based on the pre-processed multiple types of data, for each data type, a pre-trained model is used to extract and fuse features to generate a multimodal fusion feature vector, including the following steps: The image of the region of interest output after preprocessing is input into the ResNet-50 model, the last fully connected layer of the ResNet-50 model is removed, and the image feature vector is extracted through the global average pooling layer; The 13-dimensional MFCC feature vector MFCC output after preprocessing is input into the BiLSTM network, the MFCC sequence is processed by the bidirectional LSTM unit, the final hidden state of the BiLSTM network is extracted, and the audio feature vector is generated; The standardized key frame sequence output after preprocessing is input into the I3D model, and the video feature vector is extracted through the I3D model; The cleaned text output by the preprocessing module is input into the pre-trained BERT model. The BERT model generates a context embedding vector for each word, uses a pooling operation to aggregate the embedding vectors of all words into a text feature vector of fixed dimension, and adjusts the text feature vector to 512 dimensions through linear projection; Perform L2 normalization on the extracted image feature vector, audio feature vector, video feature vector, and text feature vector; Connect the normalized feature vectors of each modality in sequence to form a unified multimodal representation vector; A multi-head self-attention mechanism is introduced to perform weighted integration of multimodal representation vectors to obtain the final multimodal fusion feature vector.

[0008] As a preferred solution of the dangerous behavior identification and early warning method based on multimodal analysis described in the present invention, wherein: the multimodal fusion feature vector is paired with the risk description text through a comparative learning framework, and a similarity score is calculated, including the following steps: Set the data set. For each sample, the fused feature vector and its corresponding risk description text form a positive sample pair. For each positive sample pair, randomly select the risk description text of other samples to form a negative sample pair. Positive sample pairs and PA set of sample pairs of negative sample pairs, where P is the number of negative samples corresponding to each positive sample; A shared Transformer encoder is used to simultaneously encode the multimodal fusion feature vector and the risk description text to generate a unified embedding space; Cosine similarity is used to measure the similarity between the fusion feature embedding and the text description embedding; A contrast loss function in the form of cross entropy is used, and the temperature parameter is combined to adjust the similarity distribution, maximize the similarity of positive sample pairs, and minimize the similarity of negative sample pairs; For each fused feature vector, its similarity score with all risk description texts is calculated.

[0009] As a preferred solution of the dangerous behavior identification and early warning method based on multimodal analysis of the present invention, wherein: the large language model is trained based on the multimodal fusion feature vector, and the trained large language model is used to perform real-time analysis and identification on the input data to generate a behavior description, including the following steps: A linear projection layer is used to map the high-dimensional multimodal fusion feature vector to the input dimension required by LLM; Construct a multimodal dataset with risk labels and divide the multimodal dataset into training set, validation set and test set; For each sample, define the cross entropy loss, average the losses of all training samples to get the overall cross entropy loss; Initialize all parameters of LLM, configure Adam optimizer, set learning rate and other hyperparameters, perform forward propagation for each batch in the training set, calculate the prediction probability of LLM model, and calculate the cross entropy loss of the current batch. Calculate the gradient of loss function to LLM model parameters through back propagation algorithm, update model parameters using Adam optimizer, repeat the process of forward propagation, loss calculation, back propagation and parameter update, and perform multiple rounds of iterative training until the LLM model reaches convergence conditions and meets the preset performance indicators on the validation set; The vector InputLLM converted by the linear projection layer is used as input and input into the trained LLM model. Through the forward propagation of LLM, the behavior description of the output of the LLM model is obtained. The generated behavior description is post-processed to obtain the final behavior description.

[0010] As a preferred solution of the dangerous behavior identification and early warning method based on multimodal analysis of the present invention, the method of dynamically evaluating the risk level and generating an early warning based on the behavior description and similarity score generated in real time includes the following steps: Set the initial threshold and sliding window size , at time step , collect recent The similarity scores of samples are calculated, and the mean and standard deviation of these similarity scores are calculated; According to the mean and standard deviation, calculate the current dynamic threshold, according to the recent Similarity score distribution of samples, automatically adjusting dynamic threshold; For the sample at the current time step, get its highest similarity score , to determine whether the current sample belongs to the risk behavior category, expressed as, in, Represents the time step The risk assessment results Indicates that the current sample is normal behavior. For the risk category with the highest similarity score, is the dynamic threshold, For the current sample and risk category Similarity score of like ≠ , then an early warning is triggered, otherwise, no alarm is generated; The generated alarm content is sent to relevant personnel through the preset communication channels, and the detailed information of each warning is recorded.

[0011] As a preferred solution of the dangerous behavior identification and early warning method based on multimodal analysis of the present invention, the periodic collection of new data and continuous optimization of the large language model through incremental learning and transfer learning methods include the following steps: Define the time interval for data collection, regularly collect multimodal data, use pre-trained models to preliminarily annotate new data, and review and correct the automatic annotation results; Integrate annotated and verified multimodal data to form a new dataset; Adopting a gradient-based online learning algorithm, the model parameters are gradually updated through incremental training; Select a pre-trained language model that is highly relevant to the target application domain, and fine-tune it using multiple rounds of training and hyperparameter optimization strategies on domain-specific datasets.

[0012] As a preferred solution of the dangerous behavior identification and early warning method based on multimodal analysis of the present invention, the preprocessing of the collected multiple types of data includes the following steps: Perform denoising, standardization and semantic segmentation on image data; Perform noise reduction and feature extraction on audio data; Perform frame extraction and frame processing on video data; Perform text cleaning on text data.

[0013] In a second aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the dangerous behavior identification and warning method based on multimodal analysis as described in the first aspect of the present invention is implemented.

[0014] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the dangerous behavior identification and warning method based on multimodal analysis as described in the first aspect of the present invention is implemented.

[0015] The beneficial effects of the present invention are: integrating the features of different types of data through multimodal fusion, making up for the shortcomings of a single modality, providing a more comprehensive and multi-dimensional behavior description, and improving recognition accuracy. Through the efficient processing capability of LLM, rapid analysis and behavior recognition of real-time input data can be achieved. The detailed behavior description generated by LLM not only improves the interpretability, but also provides rich semantic information for risk assessment, supporting more accurate risk judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0017] Figure 1 This is a flow chart of the dangerous behavior identification and early warning method based on multimodal analysis in Example 1.

[0018] Figure 2 This is a flowchart for generating a multimodal fusion feature vector in Example 1. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0020] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0021] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0022] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides a dangerous behavior identification and early warning method based on multimodal analysis, comprising the following steps: S1. Collect various types of data through the device and pre-process the collected various types of data; S1.1. Collect image, audio, video and text data in real time through camera and microphone devices.

[0023] S1.2, let the input image be I, whose size is H×W×C, where H is the height, W is the width, and C is the number of color channels (usually 3). Apply a 3x3 median filter to each color channel of each input image to obtain a denoised image; It should be noted that the 3x3 median filter used can better preserve the edge information of the image when removing salt-and-pepper noise than the Gaussian filter, and is suitable for common noise types in surveillance images. Calculate the minimum and maximum pixel values ​​of the denoised image, normalize each pixel value, scale the pixel value to the range of [0,1] to ensure the uniformity of the input data, and obtain a standardized image; A pre-trained U-Net model is used for semantic segmentation to accurately identify regions of interest (ROI) in images, such as human bodies and vehicles, filter background noise, and improve the efficiency and accuracy of subsequent feature extraction. Specifically, the standardized image is input into the pre-trained U-Net model, which outputs a binary segmentation mask. The region of interest is extracted through the mask to obtain an image containing only the region of interest, and the rest of the region is set to 0.

[0024] S1.3, perform short-time Fourier transform (STFT) on the input audio signal to obtain a spectrum, estimate the noise spectrum in a silent period (a period without speech or main sound source), subtract the noise spectrum from the spectrum of the input signal to obtain a clean spectrum, and convert the clean spectrum back to the time domain through inverse STFT to obtain a cleaned audio signal; It should be noted that the Spectral Subtraction method effectively eliminates background noise and improves the clarity of the audio signal by estimating the background noise spectrum and subtracting the noise spectrum from the spectrum of the input signal; Extract 13-dimensional Mel frequency cepstral coefficients (MFCCs) as audio feature representation; apply STFT to the denoised audio signal to generate a spectrogram, convert the spectrum to the Mel frequency scale, take the logarithm of the Mel spectrogram to enhance the features of the low energy area, and apply discrete cosine transform (DCT) to extract the first 13 MFCC features, recorded as MFCC; MFCCs are widely used features in audio processing, which can effectively capture the time-frequency characteristics of audio signals and are suitable for speech and audio classification tasks.

[0025] S1.4, using the histogram comparison method to sample key frames. This method automatically detects scene changes in the video by comparing the color histogram differences of adjacent frames, ensuring that the key frames can capture important visual information; Specifically, frames are extracted from the input video at a frame rate of 2 frames per second, and the color histogram difference between adjacent frames is calculated. If the color histogram difference between adjacent frames exceeds a predefined threshold, the frame is marked as a key frame, and all key frames are collected to form a frame sequence; Each keyframe is resized to a uniform size (224x224 pixels), bilinear interpolation is used to maintain image quality, and image preprocessing techniques, including normalization and center cropping, are applied to obtain standardized frames; S1.5. For text data, use natural language processing (NLP) technology to remove stop words and punctuation marks. Specifically, perform word segmentation on the input text to obtain a word sequence, use a predefined stop word list to remove stop words and punctuation marks, and obtain a cleaned text.

[0026] S2. Based on the preprocessed data of various types, for each data type, a pre-trained model is used to extract and fuse features to generate a multimodal fusion feature vector, including the following steps: S2.1. Input the region of interest image output after preprocessing into the ResNet-50 model, remove the last fully connected layer of the ResNet-50 model to obtain the feature representation of the intermediate layer, extract the image feature vector through the global average pooling layer, and perform L2 normalization on the extracted image feature vector to ensure that the norm of the feature vector is 1, thereby improving the stability of subsequent fusion; It should be noted that the pre-trained ResNet-50 model is used for image feature extraction. Due to its deep network structure and residual connection, ResNet-50 can effectively capture the high-level semantic information of the image and is suitable for risk behavior identification in complex scenarios; The 13-dimensional MFCC feature vector MFCC output after preprocessing is input into the BiLSTM network, the MFCC sequence is processed by the bidirectional LSTM unit, the hidden state of each time step is obtained, the final hidden state of the BiLSTM network is extracted, the audio feature vector is generated, and the audio feature vector is L2 normalized; It should be noted that the bidirectional long short-term memory network (BiLSTM) is used to process MFCC features and generate audio feature vectors. BiLSTM can capture the front and back timing information of the audio signal and improve the expressiveness of audio features; The pre-trained I3D (Inflated 3D ConvNet) model is used to extract the spatiotemporal features of the video. The I3D model captures the spatiotemporal dynamic information in the video through 3D convolution and is suitable for behavior recognition in complex video scenes. The standardized key frame sequence output after preprocessing is input into the I3D model, and the video feature vector is extracted through the I3D model, and the video feature vector is L2 normalized; Use the pre-trained BERT model to generate text feature vectors. The BERT model can effectively capture the contextual semantic information of the text through the bidirectional Transformer architecture and is suitable for feature extraction of complex text descriptions. Input the cleaned text output by the preprocessing module into the pre-trained BERT model. The BERT model generates a context embedding vector for each word, and uses pooling operations (such as taking the [CLS] vector of the last layer) to aggregate the embedding vectors of all words into a text feature vector of fixed dimension. The text feature vector is adjusted to 512 dimensions through linear projection to be consistent with the feature dimensions of other modalities, and the text feature vector is L2 normalized. S2.2. Connect the normalized feature vectors of each modality in order to form a unified multimodal representation vector ; A multi-head self-attention mechanism is introduced to perform weighted integration of multi-modal representation vectors. The multi-head self-attention mechanism can dynamically adjust the weight of each modality, highlight the characteristics of key modalities, and suppress the interference of noise modalities according to the context information of the input data; Specifically, the number of heads of the multi-head self-attention mechanism is set to , the dimension of each head is ,Will Through three linear transformation matrices , , Generate query, key and value matrices, represented as, in, are the dimensions of the key vector and query vector, is the dimension of the value vector, = , usually set to 64, , , Query ,key ,value The linear transformation weight matrix of ; Will , , Divided into heads, each head has a dimension of , expressed as, in, , , Respectively represent The query submatrix, key submatrix, and value submatrix of each attention head, represents the index of the currently processed attention head, =1,2,…, ; For each head , calculate the attention weight , expressed as, in, is the attention weight, indicating the relevance of the query to the key, Convert the similarity scores into probability distributions, The similarity score between the query and the key is calculated; the attention weight matrix is ​​used to weight the value vector , thereby generating the output of the current attention head; Compute the output for each head for, The outputs of all heads are concatenated to form a concatenated feature vector, which is then mapped back to the original dimension through a linear transformation matrix to generate a fused feature vector. The fused feature vector is layer-normalized and ReLU activated to enhance the expressiveness and stability of the features, thus obtaining the final multimodal fused feature vector. It should be noted that the multi-head self-attention mechanism can capture the complex relationship between multimodal features in parallel from different representation subspaces, enhance the expressiveness of feature fusion, dynamically adjust the weights of each modality, highlight the features of key modalities according to contextual information, suppress the interference of noise modalities, and improve the recognition accuracy and robustness of the overall system.

[0027] S3, through the contrastive learning framework, the multimodal fusion feature vector is paired with the risk description text and the similarity score is calculated, including the following steps: S3.1. Set a batch of data sets. in, For the The fusion feature vector of samples, Describe the corresponding risk behavior text; For each sample , and fuse the feature vector The corresponding risk description text Forming positive sample pairs ; For each positive sample pair , randomly select other samples ≠ Risk description text Forming negative sample pairs , forming a Positive sample pairs and P A set of sample pairs of negative sample pairs, where P is the number of negative samples corresponding to each positive sample; S3.2 uses a shared Transformer encoder to simultaneously encode the multimodal fusion feature vector and risk description text to generate a unified embedding space; Specifically, a shared Transformer encoder TransformerEncoder is set, whose parameters are shared on all inputs, and the fused feature vector Input Transformer encoder to generate fused feature embedding , expressed as, in, is the embedding dimension; Describe the risk behavior in text The text feature vector generated by the BERT model is input into the Transformer encoder after adjusting the dimension through linear transformation to generate the text description embedding , expressed as, in, is the text feature vector after linear projection; The generated embedding vector is L2 normalized to ensure that the norm of the embedding vector is 1 and to improve the stability of the similarity calculation, which is expressed as, in, , They are normalized fusion feature embedding and text description embedding respectively; S3.3, cosine similarity is used to measure the similarity between fusion feature embedding and text description embedding; For each pair of embedding vectors ( , ), calculate the cosine similarity , expressed as, Since the vectors are normalized, the formula simplifies to the vector dot product; For a batch samples, generating a The similarity score matrix , expressed as, in, Indicates The fusion features and The similarity of text descriptions, is the matrix transpose operator, which swaps the rows and columns of a matrix; S3.4, using the contrast loss function in the form of cross entropy, combined with the temperature parameter to adjust the similarity distribution, maximize the similarity of positive sample pairs, and minimize the similarity of negative sample pairs, to optimize the model's discriminative ability; Specifically, set the temperature parameters Controls the smoothness of the similarity distribution, usually between 0.07 and 0.2. , the similarity of the positive sample pair is , the similarity of the negative sample pair is ( ≠ ), the softmax function is used to calculate the contrast loss of each sample , the expression is, The contrast loss of all samples is averaged to get the total contrast loss, which is expressed as, By maximizing the positive sample pair The similarity of all negative samples is minimized The model learns a more discriminative feature representation and uses a gradient descent optimization algorithm (such as the Adam optimizer) to minimize the contrast loss. , update the parameters of the Transformer encoder For each fused feature vector , calculate its correlation with all risk description texts Similarity score of , for each sample , forming its similarity score vector.

[0028] S4, training a large language model based on the multimodal fusion feature vector, using the trained large language model to perform real-time analysis and recognition on the input data, and generating a behavior description, including the following steps: S4.1. A linear projection layer is used to map the high-dimensional multimodal fusion feature vector to the input dimension required by LLM, ensuring that the dimension of the input data matches the requirements of LLM while maintaining the linear relationship and information integrity of the features. Specifically, a linear projection layer is set to obtain the multimodal fusion feature vector, and the multimodal fusion feature vector is transformed into Map to LLM and get the mapped LLM input vector .

[0029] S4.2. Construct a multimodal dataset with risk labels and divide the multimodal dataset into a training set, a validation set, and a test set; Furthermore, a large number of samples are collected from multimodal data sources. Each sample contains a vector after preprocessing and multimodal fusion and a corresponding risk label. , through the linear projection layer Convert to , forming a labeled data set, expressed as, in, is the sample size, ∈ For the The risk category labels of samples, is the total number of risk categories, Indicates the first Data of samples; The dataset Divide into training set, validation set and test set to ensure the independence and effectiveness of model training, validation and evaluation.

[0030] For each sample, define the cross entropy loss, average the losses of all training samples to get the overall cross entropy loss; Adopt Adam optimizer (Adaptive Moment Estimation) to optimize model parameters, initialize all parameters of LLM, including weights and biases of linear projection layer, and parameters inside LLM, configure Adam optimizer, set learning rate and other hyperparameters, perform forward propagation for each batch in training set, calculate prediction probability of LLM model, and calculate cross entropy loss of current batch, calculate gradient of loss function to LLM model parameters through back propagation algorithm, update model parameters using Adam optimizer, repeat the process of forward propagation, loss calculation, back propagation and parameter update, perform multiple rounds of iterative training until LLM model reaches convergence condition and meets preset performance index on validation set, monitor loss and accuracy on validation set, and stop training in advance if performance does not improve within several consecutive Epochs to prevent overfitting; It should be noted that the Adam optimizer combines momentum and adaptive learning rate adjustment, and can converge efficiently in large-scale data and high-dimensional parameter space.

[0031] S4.3. The vector transformed by the linear projection layer As input, it is input into the trained LLM model. Through the forward propagation of LLM, the behavior description output by the LLM model is obtained. The generated behavior description is post-processed, such as grammar correction and keyword extraction, to ensure the accuracy and readability of the description, and the final behavior description is obtained for early warning generation and risk response.

[0032] S5. Based on the real-time generated behavior description and similarity score, dynamically assess the risk level and generate an early warning, including the following steps: S5.1. Dynamic Threshold Based on Sliding Window The evaluation method dynamically adjusts the risk judgment threshold by monitoring the similarity scores of the most recent W samples to adapt to environmental changes and fluctuations in risk levels; Specifically, set the initial threshold , as the baseline risk level at the start of operation; Set the sliding window size , at time step , collect recent The similarity scores of samples are calculated and the mean of these similarity scores is calculated. and standard deviation , the expression is, in, is the time step The similarity score of the sample to its risk category; Calculate the current dynamic threshold based on the mean and standard deviation , the expression is, in, To adjust parameters (such as 1.5), used to control the sensitivity of thresholds and risk tolerance; Dynamic Threshold It can automatically adjust according to the similarity score distribution of the most recent W samples to ensure adaptability and flexibility to the risk level in different environments, and obtain the current time step Dynamic threshold Can be used for risk assessment and early warning generation; S5.2. For the sample at the current time step, obtain its highest similarity score , to determine whether the current sample belongs to the risk behavior category, expressed as, in, Represents the time step The risk assessment results Indicates that the current sample is normal behavior. For the risk category with the highest similarity score, is the dynamic threshold, For the current sample and risk category Similarity score of like ≠ , then an early warning is triggered, otherwise, no alarm is generated; The generated alert content contains the following information: Risk behavior description: Use the generated behavior description to describe the detected risk behavior in detail; Risk category: Indicate the category to which the risk behavior belongs ; Mitigation measures recommendations: Based on the risk category, provide corresponding mitigation and response measures recommendations to guide relevant personnel to take action; Send the generated alert content to relevant personnel through preset communication channels (such as SMS, email and application notifications), and record the details of each alert; Record the details of each alert, including timestamp, risk category, similarity score, mitigation measures, etc., to facilitate subsequent audit and analysis; For risk categories It is a set of risk behavior categories pre-defined according to actual application scenarios and security management requirements. ={ , ,…, }, each Represents a specific type of risk behavior. For example, in intelligent traffic management, risk categories may include "traveling against traffic", "running a red light", "dangerous overtaking", etc.; or in risk data analysis of intelligent scenarios, risk categories may include "abnormal transactions", "abnormal travel", "abnormal communication", etc.; during model training, each risk category There are corresponding risk description texts. Through comparative learning, the model learns the similarity relationship between the multimodal fusion vector and the risk description text, ensuring that samples of the same category have high similarity in the feature space. In the inference stage, the most matching risk category is determined by calculating the similarity score between the fusion feature vector of the current sample and the text description embedding vectors of all risk categories.

[0033] S6. Regularly collect new data and continuously optimize large language models through incremental learning and transfer learning methods, including the following steps: S6.1. Define the interval for data collection, such as once a month, to ensure that the model can adapt to new data in a timely manner, collect multimodal data regularly, use pre-trained models to preliminarily annotate new data, identify potential risk behaviors, and review and correct automatic annotation results to ensure the accuracy of data annotation; Integrate annotated and verified multimodal data to form new datasets, and store the new datasets in a secure and scalable database to ensure data accessibility and integrity; Adopting a gradient-based online learning algorithm, the model parameters are gradually updated through incremental training to ensure the model's adaptability to new data while retaining the memory of old knowledge; Specifically, add the new data set to the existing training set to form an updated training set, divide the training subset and the validation subset from the updated training set to ensure the diversity and balance of data distribution, select an optimization algorithm suitable for online learning, such as the Adam optimizer, configure a suitable learning rate and other hyperparameters, input the training subset into the model, calculate the prediction probability, use the contrast loss function to calculate the loss of the model on the training set, calculate the gradient of the loss function to the model parameters, and use the gradient descent method to update the model parameters; Evaluate model performance on the validation subset, calculate accuracy, recall, F1 score and other indicators, ensure that the model performs evenly on new and old data, and use knowledge distillation or elastic weight consolidation (EWC) technology to ensure that the model does not lose memory of old knowledge when learning new data; It should be noted that online learning algorithms can quickly adapt to new data and improve the real-time response capabilities of the model. By preventing catastrophic forgetting, the model can retain old knowledge while learning new knowledge and maintain overall performance. The incremental learning method does not require retraining the entire model, saving computing resources and time.

[0034] S6.2. Select a pre-trained language model that is highly relevant to the target application domain, and improve the model’s ability to detect risky behaviors in that domain on a dataset in that domain; Furthermore, select a pre-trained model related to the target domain. For example, if the target domain is the field of case analysis, select the pre-trained BERT model / RoBERTa model, collect and organize multimodal data and risk behavior samples in the specific domain to form a domain-specific dataset, perform the same data preprocessing steps as the initial training on the domain-specific dataset, including noise reduction, standardization, feature extraction, etc., to ensure data quality and consistency, and input the domain-specific dataset into the pre-trained model for fine-tuning training; Furthermore, multiple rounds of training and hyperparameter optimization strategies are adopted to ensure the efficient performance and generalization ability of the model on data in specific fields.

[0035] This embodiment also provides a computer device, which is suitable for the dangerous behavior identification and early warning method based on multimodal analysis, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the dangerous behavior identification and early warning method based on multimodal analysis as proposed in the above embodiment.

[0036] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0037] The present embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the dangerous behavior identification and early warning method based on multimodal analysis as proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0038] In summary, the present invention collects various types of data through equipment and performs preprocessing, thereby achieving the acquisition and preparation of high-quality, multi-dimensional data, ensuring the clarity and consistency of input data, eliminating noise and interference factors, and utilizing the feature extraction capability of the pre-trained model to capture the key features of each modal data, thereby improving the expressiveness and distinguishability of the feature vector. Multimodal fusion integrates the features of different types of data to make up for the deficiencies of a single modality, provide a more comprehensive and multi-dimensional behavior description, and improve recognition accuracy. The multi-head self-attention mechanism intelligently weights the features of each modality, optimizes the fusion process, ensures the prominence of key features and the suppression of noise, and contrastive learning significantly enhances the model's distinction between different risk behaviors by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. The shared Transformer encoder ensures the consistency of multimodal features and text descriptions in the same embedding space, improves the integration and comparability of feature representation, and adjusts the similarity distribution by introducing temperature parameters, so that the model can still maintain stable performance when facing different data distributions and noise. Through the efficient processing capability of LLM, it realizes rapid analysis and behavior recognition of real-time input data, and meets the needs of instant response in practical applications. The detailed behavior description generated by LLM not only improves the interpretability, but also provides rich semantic information for risk assessment, supporting more accurate risk judgment. After multiple rounds of iterative training, the large language model has good generalization ability and can adapt to a variety of complex and dynamic risk behavior patterns.

[0039] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A dangerous behavior identification and early warning method based on multimodal analysis, characterized by: include, Collecting multiple types of data through the device and preprocessing the collected multiple types of data, wherein the multiple types of data include image, audio, video and text data; Based on the preprocessed data of various types, for each data type, a pre-trained model is used to extract and fuse features to generate a multi-modal fusion feature vector; Through the contrastive learning framework, the multimodal fusion feature vector is paired with the risk description text and the similarity score is calculated; Based on the multimodal fusion feature vector, a large language model is trained, and the trained large language model is used to analyze and identify the input data in real time to generate a behavior description; Dynamically assess risk levels and generate early warnings based on real-time generated behavior descriptions and similarity scores; New data is collected regularly to continuously optimize large language models through incremental learning and transfer learning methods.

2. The method for identifying and warning dangerous behaviors based on multimodal analysis according to claim 1, characterized in that: The method, based on the pre-processed data of various types, uses a pre-trained model to extract and fuse features for each data type to generate a multi-modal fusion feature vector, including the following steps: The image of the region of interest output after preprocessing is input into the ResNet-50 model, the last fully connected layer of the ResNet-50 model is removed, and the image feature vector is extracted through the global average pooling layer; The 13-dimensional MFCC feature vector MFCC output after preprocessing is input into the BiLSTM network, the MFCC sequence is processed by the bidirectional LSTM unit, the final hidden state of the BiLSTM network is extracted, and the audio feature vector is generated; The standardized key frame sequence output after preprocessing is input into the I3D model, and the video feature vector is extracted through the I3D model; The cleaned text output by the preprocessing module is input into the pre-trained BERT model. The BERT model generates a context embedding vector for each word, uses a pooling operation to aggregate the embedding vectors of all words into a text feature vector of fixed dimension, and adjusts the text feature vector to 512 dimensions through linear projection; Perform L2 normalization on the extracted image feature vector, audio feature vector, video feature vector, and text feature vector; Connect the normalized feature vectors of each modality in sequence to form a unified multimodal representation vector; A multi-head self-attention mechanism is introduced to perform weighted integration of multimodal representation vectors to obtain the final multimodal fusion feature vector.

3. The dangerous behavior identification and early warning method based on multimodal analysis according to claim 2, characterized in that: The method of pairing the multimodal fusion feature vector with the risk description text and calculating the similarity score through the contrast learning framework includes the following steps: Set the data set. For each sample, the fused feature vector and its corresponding risk description text form a positive sample pair. For each positive sample pair, randomly select the risk description text of other samples to form a negative sample pair. Positive sample pairs and P A set of sample pairs of negative sample pairs, where P is the number of negative samples corresponding to each positive sample; A shared Transformer encoder is used to simultaneously encode the multimodal fusion feature vector and the risk description text to generate a unified embedding space; Cosine similarity is used to measure the similarity between the fusion feature embedding and the text description embedding; A contrast loss function in the form of cross entropy is used, and the temperature parameter is combined to adjust the similarity distribution, maximize the similarity of positive sample pairs, and minimize the similarity of negative sample pairs; For each fused feature vector, its similarity score with all risk description texts is calculated.

4. The method for identifying and warning dangerous behaviors based on multimodal analysis according to claim 3, characterized in that: The method of training a large language model based on a multimodal fusion feature vector, using the trained large language model to perform real-time analysis and recognition on input data, and generating a behavior description includes the following steps: A linear projection layer is used to map the high-dimensional multimodal fusion feature vector to the input dimension required by LLM; Construct a multimodal dataset with risk labels and divide the multimodal dataset into training set, validation set and test set; For each sample, define the cross entropy loss, average the losses of all training samples to get the overall cross entropy loss; Initialize all parameters of LLM, configure Adam optimizer, set learning rate and other hyperparameters, perform forward propagation for each batch in the training set, calculate the prediction probability of LLM model, and calculate the cross entropy loss of the current batch. Calculate the gradient of loss function to LLM model parameters through back propagation algorithm, update model parameters using Adam optimizer, repeat the process of forward propagation, loss calculation, back propagation and parameter update, and perform multiple rounds of iterative training until the LLM model reaches convergence conditions and meets the preset performance indicators on the validation set; The vector InputLLM converted by the linear projection layer is used as input and input into the trained LLM model. Through the forward propagation of LLM, the behavior description of the output of the LLM model is obtained. The generated behavior description is post-processed to obtain the final behavior description.

5. The method for identifying and warning dangerous behaviors based on multimodal analysis according to claim 4, characterized in that: The method of dynamically assessing the risk level and generating an early warning based on the behavior description and similarity score generated in real time includes the following steps: Set the initial threshold and sliding window size , at time step , collect recent The similarity scores of samples are calculated, and the mean and standard deviation of these similarity scores are calculated; According to the mean and standard deviation, calculate the current dynamic threshold, according to the recent Similarity score distribution of samples, automatically adjusting dynamic threshold; For the sample at the current time step, get its highest similarity score , to determine whether the current sample belongs to the risk behavior category, expressed as, in, Represents the time step The risk assessment results Indicates that the current sample is normal behavior. For the risk category with the highest similarity score, is the dynamic threshold, For the current sample and risk category Similarity score of like ≠ , then an early warning is triggered, otherwise, no alarm is generated; The generated alarm content is sent to relevant personnel through the preset communication channels, and the detailed information of each warning is recorded.

6. The dangerous behavior identification and early warning method based on multimodal analysis according to claim 5, characterized in that: The method of regularly collecting new data and continuously optimizing large language models through incremental learning and transfer learning methods includes the following steps: Define the time interval for data collection, regularly collect multimodal data, use pre-trained models to preliminarily annotate new data, and review and correct the automatic annotation results; Integrate annotated and verified multimodal data to form a new dataset; Adopting a gradient-based online learning algorithm, the model parameters are gradually updated through incremental training; Select a pre-trained language model that is highly relevant to the target application domain, and fine-tune it using multiple rounds of training and hyperparameter optimization strategies on domain-specific datasets.

7. The dangerous behavior identification and early warning method based on multimodal analysis according to claim 1, characterized in that: The preprocessing of the collected data of various types includes the following steps: Perform denoising, standardization and semantic segmentation on image data; Perform noise reduction and feature extraction on audio data; Perform frame extraction and frame processing on video data; Perform text cleaning on text data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the dangerous behavior identification and early warning method based on multimodal analysis as described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the dangerous behavior identification and early warning method based on multimodal analysis described in any one of claims 1 to 7 are implemented.

Images