Intelligent monitoring method and system based on cross-modality and storage medium

Abstract

The invention provides an intelligent monitoring method and system based on cross-modality, and a storage medium. The method comprises the following steps: constructing and training a cross-modality fusion model based on semantic enhancement; acquiring a video stream of a monitoring platform, and performing key frame extraction on the video stream; and detecting video key frames through the cross-modal fusion model according to custom scene description needing early warning, and if a video picture conforming to the custom scene description is detected, giving an alarm to a platform. The method has the beneficial effects that a cross-modal fusion model based on semantic enhancement is provided aiming at the defects of traditional manual monitoring and the limitation of an image detection scheme based on a target object, and recognition and intelligent monitoring of a monitoring video stream aiming at a user-defined description scene are realized; meanwhile, the key frames of the video stream are obtained, so that video abstraction can be quickly and accurately carried out, video redundancy and processing complexity are reduced, and the speed and accuracy of self-defined description scene recognition are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent security monitoring technology, and in particular to a cross-modality-based intelligent monitoring method, system and storage medium. Background Art

[0002] In complex environments such as construction sites, it is often necessary to monitor and warn specific scenes. Traditional methods mainly rely on manual identification, resulting in a waste of manpower, while monitoring methods based on image detection require pre-definition of the target object before detection, and the label information contains little information, does not fully explore the correlation between language and image information, and has great limitations.

[0003] That is, the existing solutions are insufficient for traditional manual monitoring, and the application of image detection solutions based on targets has limitations. Therefore, how to capture and warn of scenes with custom descriptions without relying on manual work is the current problem to be solved. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a cross-modal intelligent monitoring method, system and storage medium to achieve monitoring and early warning of customizable description scenarios to reduce the serious impact that may be caused by emergencies.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a cross-modal intelligent monitoring method, comprising the steps of:

[0006] S1. Build and train a cross-modal fusion model based on semantic enhancement;

[0007] S2, obtaining a video stream from a monitoring platform, and extracting key frames from the video stream;

[0008] S3. According to the custom scene description required for early warning, the video key frames are detected through the cross-modal fusion model. If a video screen that meets the custom scene description is detected, an alarm is issued to the platform.

[0009] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0010] A cross-modality-based intelligent monitoring system, the system comprising:

[0011] A sample construction module is used to obtain a cross-modal data set, wherein the cross-modal data set includes text data describing a scene image and collected video stream data;

[0012] A feature extraction module extracts key features of scene images and text descriptions in the cross-modal dataset;

[0013] Model building module, building a cross-modal fusion model based on semantic enhancement;

[0014] A model training module, which trains the semantically enhanced cross-modal fusion model according to the cross-modal data set, calculates the similarity between the cross-modal data, and obtains a model for recognizing custom description scenes;

[0015] The timing monitoring module collects and uploads the high-definition video stream of the monitoring platform on a regular basis;

[0016] The key frame extraction module extracts key frames based on the uploaded video stream;

[0017] The alarm module detects the video key frames according to the custom scene description required for the warning in combination with the cross-modal fusion model. When a video picture that meets the custom scene description is detected, an alarm message is generated and the scene picture is saved.

[0018] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0019] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned cross-modality-based intelligent monitoring method.

[0020] The beneficial effects of the present invention are as follows: the present invention is a cross-modal intelligent monitoring method, system and storage medium, aiming at the shortcomings of traditional manual monitoring and the limitations of the image detection scheme based on the target, and proposes a cross-modal fusion model based on semantic enhancement, so as to realize the recognition and intelligent monitoring of the monitoring video stream for the custom description scene; at the same time, the key frames of the video stream are acquired, so that the video summary can be performed quickly and accurately, the redundancy of the video and the complexity of the processing are reduced, and the speed and accuracy of the recognition of the custom description scene are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a cross-modality-based intelligent monitoring method according to an embodiment of the present invention;

[0022] Figure 2 A structural diagram of a cross-modality-based intelligent monitoring system according to an embodiment of the present invention;

[0023] Description of labels:

[0024] 201, sample construction module; 202, feature extraction module; 203, model construction module;

[0025] 204. Model training module; 205. Timing detection module; 206. Key frame extraction module; 207. Alarm module. DETAILED DESCRIPTION

[0026] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0027] Please refer to Figure 1 , a cross-modality-based intelligent monitoring method, comprising the steps of:

[0028] S1. Build and train a cross-modal fusion model based on semantic enhancement;

[0029] S2, obtaining a video stream from a monitoring platform, and extracting key frames from the video stream;

[0030] S3. According to the custom scene description required for early warning, the video key frames are detected through the cross-modal fusion model. If a video screen that meets the custom scene description is detected, an alarm is issued to the platform.

[0031] From the above description, it can be seen that the beneficial effects of the present invention are: in view of the shortcomings of traditional manual monitoring and the limitations of target-based image detection solutions, a cross-modal fusion model based on semantic enhancement is proposed to realize the recognition and intelligent monitoring of monitoring video streams for custom description scenes; at the same time, the key frames of the video stream are acquired to quickly and accurately perform video summarization, reduce the redundancy of the video and the complexity of processing, and improve the speed and accuracy of custom description scene recognition.

[0032] Furthermore, the step S1 comprises the steps of:

[0033] S11, constructing a cross-modal dataset, where each pair of samples in the cross-modal dataset includes a text description and a corresponding scene image;

[0034] S12, extracting features of the scene image and text description respectively according to a preset cross-modal data set, and enhancing the text description features based on the semantic enhancement of the encoded features;

[0035] S13. Construct a cross-modal fusion model based on semantic enhancement to align the scene image features and the text description features, aggregate the aligned scene image features and text description features, and construct a contrast loss function to train the model.

[0036] From the above description, it can be seen that a cross-modal dataset is constructed and preprocessed, a cross-modal fusion model based on semantic enhancement is constructed, and the model is trained according to the cross-modal dataset so that the model can be retrieved according to the semantic description.

[0037] Furthermore, in step S12, the features of the scene image and the text description are extracted respectively according to the preset cross-modal data set, including:

[0038] Divide the text description into the smallest unit token in the text;

[0039] Each token is encoded and represented through a pre-trained language representation model to obtain a text feature vector;

[0040] The scene image is enhanced and divided into blocks, and the image sequence is represented as I=I1+I2+I3…I m , feature encoding is performed on each image sequence through a pre-trained image classification model to obtain an image feature vector.

[0041] From the above description, we can see that after splitting the text description into tokens, on the one hand, the adaptability of the pre-trained language representation model to different languages ​​and text types is enhanced; on the other hand, processing smaller token units can improve processing efficiency. At the same time, by performing image enhancement on the scene image, the visual quality is improved, important information is highlighted, and the adaptability of the image is enhanced; image segmentation of the enhanced scene image can reduce computational complexity and shorten processing time.

[0042] Furthermore, in step S12, the text description feature is enhanced based on the semantic enhancement of the coding feature, including:

[0043] Obtaining a context vector representation of an input text through a pre-trained language representation model, and extracting global features of the input text;

[0044] A recurrent neural network based on a self-attention mechanism further calculates a weight matrix, and extracts local features of the input text in combination with the weight matrix;

[0045] The global features and the local features are added and fused and then output.

[0046] From the above description, we can see that text descriptions may have different semantics in different global contexts. Therefore, it is necessary to consider overall semantic matching and obtain global information of text features through pre-trained language models. The recurrent neural network based on the self-attention mechanism can effectively extract local information because the attention mechanism and its hidden state contain rich sequence information. The two are added and fused to mine deeper semantic information of the text description.

[0047] Furthermore, in step S13, a cross-modal fusion model based on semantic enhancement is constructed to align the scene image features and the text description features, including:

[0048] The scene image features and the enhanced text description features are taken as features F1 and input into a fully connected layer respectively, where the input dimension is the embedding feature length and the output dimension is the mapping length, to obtain the mapping feature F2;

[0049] Mapping the feature F2 through an activation function to obtain a feature F3;

[0050] The feature F3 is input into another fully connected layer network, where both the input and output dimensions are the mapping feature length, and a Dropout layer is introduced to obtain the output feature F4;

[0051] The feature F4 is normalized to obtain the transformed scene image and text description features.

[0052] From the above description, we can see that a neural network is built to process scene image features and text description features. The GELU activation function is used for mapping in the hidden layer, which helps to improve the convergence speed and performance of the training process. Secondly, the introduction of the Dropout layer can reduce the risk of overfitting. Finally, standardization is performed before output to avoid the numerical range of the features being too large or too small, thereby accelerating the training process and improving the stability and generalization ability of the model.

[0053] Furthermore, in step S13, the aligned scene image features and text description features are aggregated and a contrast loss function is constructed to train the model, including:

[0054] The aligned text description feature vector F t Represented as query vector Q, the normalized scene image feature F I Represented as a key vector K and a value vector V, or the scene image feature F I Represented as query vector Q, the text description feature F t Represented as a key vector K and a value vector V;

[0055] Calculate the text query vector Q and key vector K T Dot product, scaling and normalization are performed to obtain the correlation coefficient between different modal features and multiply it with the value vector V to obtain the text-image feature attention vector or image-text feature attention vector. The cross attention vector calculation formula is:

[0056]

[0057] F t '=Attn(Q t ,K I ,V I )

[0058] F I '=Attn(Q I ,K t ,V t )

[0059] Where Attn represents the cross attention mechanism, Q, K, V represent the query vector, key vector and value vector respectively, Q t , K t ,V t A vector representing the text description, Q I ,K I ,V I represents the vector of the scene image, softmax represents the normalization function, d k represents the dimension of the key vector, F t ' represents the text-image feature attention vector, F I ' represents the image-text feature attention vector;

[0060] The feature vector F t and F t ' and the eigenvector F I and F I 'Aggregate separately to obtain the final text description and scene image feature vector. The aggregation formula is as follows;

[0061] T t =αF t +(1-α)F t '

[0062] T I =αF I +(1-α)F I '

[0063] Among them, T t and T I represents the encoded representation of the aggregated text and image, F t and F I Represents the original encoding representation of the image, F t ' represents the text-image feature attention vector, F I ' represents the image-text feature attention vector;

[0064] The aggregated text description and scene image feature vectors are input into the contrastive loss function to train the model and perform backpropagation and optimization.

[0065] From the above description, we can see that by aggregating the two, the model can make full use of the information of these two different modalities and avoid the limitations of single modal information. It can more accurately understand the user's intentions, improve the accuracy of retrieval, and help to explore the potential associations between images and texts, so that the model can learn more comprehensive and richer scene representations, and then make more accurate judgments in subsequent tasks; at the same time, constructing a contrast loss function and training it can help optimize the parameters of the model, so that it can be continuously adjusted and improved during the training process to achieve better performance.

[0066] Further, step S2 includes:

[0067] S20, regularly collect and upload the high-definition video stream of the monitoring platform;

[0068] S21, using the KTS kernel temporal segmentation algorithm to segment the uploaded video into several disjoint segments and input them into the self-attention network, obtain the attention vector and generate the attention weight matrix through normalization;

[0069] S22, inputting the attention weight matrix into a structure composed of multiple layers of neural networks connected in series, and calculating the importance score of the video frame;

[0070] S23. Among the image frames whose calculated scores are greater than the threshold, two image frames whose inter-frame distance is greater than a specific threshold are saved as key frames of the video stream, wherein the inter-frame distance is the difference between the two frame sequence numbers.

[0071] From the above description, it can be seen that the importance scores of video frames are calculated through the KTS kernel temporal segmentation algorithm, the self-attention network and the multi-layer neural network. Among them, due to the continuity of video frames, the introduction of inter-frame distance can prevent the judgment based solely on score calculation, and the video frames with a larger inter-frame distance that meets a certain frame distance can be judged as key frames based on the content calculation score. The accuracy is improved.

[0072] Furthermore, step S22 is specifically as follows:

[0073] Input the attention weight matrix into a structure composed of multiple layers of neural networks connected in series, calculate the importance score of the video frame, and compress it within the range of 0-1;

[0074] The multi-layer neural network includes an input layer, a linear layer, a Dropout layer, a RELU activation layer and an output layer.

[0075] From the above description, we can see that the serial connection structure of the multi-layer neural network can gradually extract and refine the importance features of the video frame. From the initial transformation of the linear layer, to the regularization of the Dropout layer, and then through the nonlinear mapping of the RELU activation layer, the calculated video frame importance score is finally compressed into the range of 0-1 through the output layer. The step-by-step processing method can make the model better adapt to different types of video data and improve the performance and reliability of the model in different scenarios.

[0076] Please refer to Figure 2 , a cross-modality-based intelligent monitoring system, the system comprising:

[0077] A sample construction module is used to obtain a cross-modal data set, wherein the cross-modal data set includes text data describing a scene image and collected video stream data;

[0078] A feature extraction module extracts key features of scene images and text descriptions in the cross-modal dataset;

[0079] Model building module, building a cross-modal fusion model based on semantic enhancement;

[0080] A model training module, which trains the semantically enhanced cross-modal fusion model according to the cross-modal data set, calculates the similarity between the cross-modal data, and obtains a model for recognizing custom description scenes;

[0081] The timing monitoring module collects and uploads the high-definition video stream of the monitoring platform on a regular basis;

[0082] The key frame extraction module extracts key frames based on the uploaded video stream;

[0083] The alarm module detects the video key frames according to the custom scene description required for the warning in combination with the cross-modal fusion model. When a video picture that meets the custom scene description is detected, an alarm message is generated and the scene picture is saved.

[0084] From the above description, it can be seen that the beneficial effects of the present invention are: in view of the shortcomings of traditional manual monitoring and the limitations of target-based image detection solutions, a cross-modal fusion model based on semantic enhancement is proposed to realize the recognition and intelligent monitoring of monitoring video streams for custom description scenes; at the same time, the key frames of the video stream are acquired to quickly and accurately perform video summarization, reduce the redundancy of the video and the complexity of processing, and improve the speed and accuracy of custom description scene recognition.

[0085] A computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps in the cross-modality-based intelligent monitoring method described in the claims above are implemented.

[0086] From the above description, it can be seen that the beneficial effects of the present invention are: in view of the shortcomings of traditional manual monitoring and the limitations of target-based image detection solutions, a cross-modal fusion model based on semantic enhancement is proposed to realize the recognition and intelligent monitoring of monitoring video streams for custom description scenes; at the same time, the key frames of the video stream are acquired to quickly and accurately perform video summarization, reduce the redundancy of the video and the complexity of processing, and improve the speed and accuracy of custom description scene recognition.

[0087] The present invention provides a cross-modal intelligent monitoring method, system and storage medium, which are suitable for monitoring, capturing and warning scenes with custom descriptions without relying on manual work.

[0088] Please refer to Figure 1 , Embodiment 1 of the present invention is: a cross-modality-based intelligent monitoring method, comprising the steps of:

[0089] S1. Build and train a cross-modal fusion model based on semantic enhancement;

[0090] In this embodiment, step S1 includes the steps of:

[0091] S11, constructing a cross-modal dataset, where each pair of samples in the cross-modal dataset includes a text description and a corresponding scene image;

[0092] S12, extracting features of the scene image and text description respectively according to a preset cross-modal data set, and enhancing the text description features based on the semantic enhancement of the encoded features;

[0093] S13. Construct a cross-modal fusion model based on semantic enhancement to align the scene image features and the text description features, aggregate the aligned scene image features and text description features, and construct a contrast loss function to train the model.

[0094] S2, obtaining a video stream from a monitoring platform, and extracting key frames from the video stream;

[0095] S3. According to the custom scene description required for early warning, the video key frames are detected through the cross-modal fusion model. If a video screen that meets the custom scene description is detected, an alarm is issued to the platform.

[0096] Please refer to Figure 1 , Embodiment 2 of the present invention is:

[0097] A cross-modal intelligent monitoring method, based on the first embodiment, in step S12, respectively extracting features of scene images and text descriptions according to a preset cross-modal data set, including:

[0098] Divide the text description into the smallest unit token in the text;

[0099] Each token is encoded and represented through a pre-trained language representation model to obtain a text feature vector;

[0100] In this embodiment, given an input sentence S=s1+s2+s3…s n, where n represents the length of the sentence. A pre-trained language representation model is used to encode each token of the text. In this embodiment, the DistilBERT model is used:

[0101] V t (S)=DistilBERT(S)

[0102] The final encoding matrix is ​​V t (S) = [h1,h2,h3…h n ]∈R n×d Represents a text feature vector.

[0103] The scene image is enhanced and divided into blocks, and the image sequence is represented as I=I1+I2+I3…I m , feature encoding is performed on each image sequence through a pre-trained image classification model to obtain an image feature vector.

[0104] In this embodiment, the input image is first enhanced, then transformed to 224×224 size and divided into 7×7 image blocks. The image sequence is represented as I=I1+I2+I3…I m ,Finally, the image is input into the pre-trained image classification model to perform feature encoding on each image sequence. In this embodiment, the ViT model is used:

[0105] V i (I)=ViT(I)

[0106] Finally, the corresponding encoding matrix is ​​obtained to represent the image feature vector.

[0107] In step S12, the text description feature is enhanced based on the semantic enhancement of the coding feature, including:

[0108] Obtaining a context vector representation of an input text through a pre-trained language representation model, and extracting global features of the input text;

[0109] A recurrent neural network based on a self-attention mechanism further calculates a weight matrix, and extracts local features of the input text in combination with the weight matrix;

[0110] The global features and the local features are added and fused and then output.

[0111] In this embodiment, the language representation model used is DistilBERT, and in other equivalent embodiments, it is not limited to the DistilBERT model. DistilBERT is used to obtain the context vector representation of the input text to obtain the global information of the text features, and then the weight matrix is ​​further calculated using the self-attention-based Bi-LSTM network. Since its attention mechanism and its hidden state contain rich sequence information, local information can be effectively extracted. By adding features, the global features and local features extracted by the two networks are added and fused.

[0112] In step S13, a cross-modal fusion model based on semantic enhancement is constructed to align the scene image features and the text description features, including:

[0113] The scene image features and the enhanced text description features are taken as features F1 and input into a fully connected layer respectively, where the input dimension is the embedding feature length and the output dimension is the mapping length, to obtain the mapping feature F2;

[0114] Mapping the feature F2 through an activation function to obtain a feature F3;

[0115] In this embodiment, the GELU activation function is used for mapping. In other equivalent embodiments, it is not limited to the GELU activation function.

[0116] The feature F3 is input into another fully connected layer network, where both the input and output dimensions are the mapping feature length, and a Dropout layer is introduced to obtain the output feature F4;

[0117] The feature F4 is normalized to obtain the transformed scene image and text description features.

[0118] In this embodiment, Layer-Norm normalization is used to make the network more stable during training. The formula is as follows:

[0119]

[0120] E(x) is the mean of the input data, Var(x) is the variance of the input data, and ∈ is a very small number to prevent the denominator from being zero. γ and β are learnable parameters, representing scaling and displacement, respectively.

[0121] In step S13, the aligned scene image features and text description features are aggregated and a contrast loss function is constructed to train the model, including:

[0122] The aligned text description feature vector F t Represented as query vector Q, the normalized scene image feature F IRepresented as a key vector K and a value vector V, or the scene image feature F I Represented as query vector Q, the text description feature F t Represented as a key vector K and a value vector V;

[0123] Calculate the text query vector Q and key vector K T Dot product, scaling and normalization are performed to obtain the correlation coefficient between different modal features and multiply it with the value vector V to obtain the text-image feature attention vector or the image-text feature attention vector;

[0124] In this embodiment, firstly, the text feature vector F t Represents as query vector Q, and image features F I Represented as key K and value V. Second, calculate the text query vector Q and image key vector K T The dot product is then scaled and normalized to obtain the correlation coefficient between different modal features, and then multiplied with the value vector V to obtain the text-image feature attention vector; similarly, the image feature F I Expressed as query vector Q, the text feature representation is regarded as key K and value V. Repeating the above operation, we can get the image-text feature attention vector, where the cross attention vector calculation formula is:

[0125]

[0126] F t '=Attn(Q t ,K I ,V I )

[0127] F I '=Attn(Q I ,K t ,V t )

[0128] Where Attn represents the cross attention mechanism, Q, K, V represent the query vector, key vector and value vector respectively, Q t ,K t ,V t A vector representing the text description, Q I ,K I ,V I represents the vector of the scene image, softmax represents the normalization function, d k represents the dimension of the key vector, F t ' represents the text-image feature attention vector, F I ' represents the image-text feature attention vector.

[0129] The feature vector F tand F t ' and the eigenvector F I and F I 'Aggregate separately to obtain the final text description and scene image feature vector. The aggregation formula is as follows;

[0130] T t =αF t +(1-α)F t '

[0131] T I =αF I +(1-α)F I '

[0132] Among them, T t and T I represents the encoded representation of the aggregated text and image, F t and F I Represents the original encoding representation of the image, F t ' represents the text-image feature attention vector, F I ' represents the image-text feature attention vector;

[0133] The aggregated text description and scene image feature vectors are input into the contrastive loss function to train the model and perform backpropagation and optimization.

[0134] Among them, the loss function is:

[0135]

[0136] Among them, the temperature coefficient τ is a set hyperparameter, q and k can represent similarity metrics, the numerator represents the similarity between positive examples, and the denominator represents the similarity between positive examples and negative examples. The greater the similarity of the same category, the smaller the similarity of different categories, and the smaller the loss.

[0137] Please refer to Figure 1 , Embodiment 3 of the present invention is:

[0138] A cross-modality-based intelligent monitoring method, which is different from the first or second embodiment in that step S2 includes:

[0139] Step S2 includes:

[0140] S20, regularly collect and upload the high-definition video stream of the monitoring platform;

[0141] S21, using the KTS kernel temporal segmentation algorithm to segment the uploaded video into several disjoint segments and input them into the self-attention network, obtain the attention vector and generate the attention weight matrix through normalization;

[0142] In this embodiment, the video is first segmented into several disjoint segments using the kernel temporal segmentation algorithm KTS. The best set of change points of a specific video relative to the frame-level features is regarded as the shot boundary, and these change points are based on the frame-level similarity determined using a positive definite kernel.

[0143] In this embodiment, the video frames of the segmented video clips are input into the self-attention network. After obtaining the attention vector, a softmax normalization operation is used to generate an attention weight matrix, which represents the importance of the input features relative to the frame at time t.

[0144] S22, inputting the attention weight matrix into a structure composed of multiple layers of neural networks connected in series, and calculating the importance score of the video frame;

[0145] S22 specifically includes: inputting the attention weight matrix into a structure serially connected by a multi-layer neural network, calculating the importance score of the video frame, and compressing it within the range of 0-1;

[0146] The multi-layer neural network includes an input layer, a linear layer, a Dropout layer, a RELU activation layer and an output layer.

[0147] In this embodiment, the Dropout parameter is set to 0.3, and the output layer is composed of sigmoid activation units.

[0148] S23. Among the image frames whose calculated scores are greater than the threshold, two image frames whose inter-frame distance is greater than a specific threshold are saved as key frames of the video stream, wherein the inter-frame distance is the difference between the two frame sequence numbers.

[0149] Please refer to Figure 2 , Embodiment 4 of the present invention is:

[0150] A cross-modality-based intelligent monitoring system, the system comprising:

[0151] A sample construction module is used to obtain a cross-modal data set, wherein the cross-modal data set includes text data describing a scene image and collected video stream data;

[0152] A feature extraction module extracts key features of scene images and text descriptions in the cross-modal dataset;

[0153] Model building module, building a cross-modal fusion model based on semantic enhancement;

[0154] A model training module, which trains the semantically enhanced cross-modal fusion model according to the cross-modal data set, calculates the similarity between the cross-modal data, and obtains a model for recognizing custom description scenes;

[0155] The timing monitoring module collects and uploads the high-definition video stream of the monitoring platform on a regular basis;

[0156] The key frame extraction module extracts key frames based on the uploaded video stream;

[0157] The alarm module detects the video key frames according to the custom scene description required for the warning in combination with the cross-modal fusion model. When a video picture that meets the custom scene description is detected, an alarm message is generated and the scene picture is saved.

[0158] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the program implements the steps of a cross-modal intelligent monitoring method in Embodiment 1 of the present invention.

[0159] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0160] In summary, the present invention provides a cross-modal intelligent monitoring method, system and storage medium. Aiming at the shortcomings of traditional manual monitoring and the limitations of target-based image detection solutions, a cross-modal fusion model based on semantic enhancement is proposed. The model extracts text and image coding features, and semantically enhances the text features, quickly aligns the text features and image features, aggregates the text and image features, and trains and optimizes the model through a contrast loss function to realize the recognition and intelligent monitoring of custom description scenes. At the same time, the key frames of the video stream are acquired to quickly and accurately perform video summarization, reduce the redundancy of the video and the complexity of processing, and improve the speed and accuracy of recognizing custom description scenes.

[0161] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A cross-modal intelligent monitoring method, characterized in that: Includes steps: S1. Build and train a cross-modal fusion model based on semantic enhancement; S2, obtaining a video stream from a monitoring platform, and extracting key frames from the video stream; S3. According to the custom scene description required for early warning, the video key frames are detected through the cross-modal fusion model. If a video screen that meets the custom scene description is detected, an alarm is issued to the platform.

2. The cross-modality-based intelligent monitoring method according to claim 1, characterized in that: The step S1 comprises the steps of: S11, constructing a cross-modal dataset, where each pair of samples in the cross-modal dataset includes a text description and a corresponding scene image; S12, extracting features of the scene image and text description respectively according to a preset cross-modal data set, and enhancing the text description features based on the semantic enhancement of the encoded features; S13. Construct a cross-modal fusion model based on semantic enhancement to align the scene image features and the text description features, aggregate the aligned scene image features and text description features, and construct a contrast loss function to train the model.

3. The cross-modality-based intelligent monitoring method according to claim 2, characterized in that: The step S12 extracts the features of the scene image and the text description respectively according to the preset cross-modal data set, including: Divide the text description into the smallest unit token in the text; Each token is encoded and represented through a pre-trained language representation model to obtain a text feature vector; The scene image is enhanced and divided into blocks, and the image sequence is represented as I=I1+I2+I3…I m , feature encoding is performed on each image sequence through a pre-trained image classification model to obtain an image feature vector.

4. The cross-modality-based intelligent monitoring method according to claim 2, characterized in that: In step S12, the text description feature is enhanced based on the semantic enhancement of the coding feature, including: Obtaining a context vector representation of an input text through a pre-trained language representation model, and extracting global features of the input text; A recurrent neural network based on a self-attention mechanism further calculates a weight matrix, and extracts local features of the input text in combination with the weight matrix; The global features and the local features are added and fused and then output.

5. The cross-modality-based intelligent monitoring method according to claim 2, characterized in that: The step S13 constructs a cross-modal fusion model based on semantic enhancement to align the scene image features and the text description features, including: The scene image features and the enhanced text description features are taken as features F1 and input into a fully connected layer respectively, where the input dimension is the embedding feature length and the output dimension is the mapping length, to obtain the mapping feature F2; Mapping the feature F2 through an activation function to obtain a feature F3; The feature F3 is input into another fully connected layer network, where both the input and output dimensions are the mapping feature length, and a Dropout layer is introduced to obtain the output feature F4; The feature F4 is normalized to obtain the transformed scene image and text description features.

6. The cross-modality-based intelligent monitoring method according to claim 2, characterized in that: In step S13, the aligned scene image features and text description features are aggregated and a contrast loss function is constructed to train the model, including: The aligned text description feature vector F t Represented as query vector Q, the normalized scene image feature F I Represented as a key vector K and a value vector V, or the scene image feature F I Represented as query vector Q, the text description feature F t Represented as a key vector K and a value vector V; Calculate the text query vector Q and key vector K T Dot product, scaling and normalization are performed to obtain the correlation coefficient between different modal features and multiply it with the value vector V to obtain the text-image feature attention vector or image-text feature attention vector. The cross attention vector calculation formula is: F t ‘=Attn(Q t ,K I ,V I ) F I ‘=Attn(Q I ,K t ,V t ) Where Attn represents the cross attention mechanism, Q, K, V represent the query vector, key vector and value vector respectively, Q t ,K t ,V t A vector representing the text description, Q I ,K I ,V I represents the vector of the scene image, softmax represents the normalization function, d k represents the dimension of the key vector, F t ' represents the text-image feature attention vector, F I ' represents the image-text feature attention vector; The feature vector F t and F t ' and the eigenvector F I and F I 'Aggregate separately to obtain the final text description and scene image feature vector. The aggregation formula is as follows; T t =αF t +(1-α)F t ' T I =αF I +(1-α)F I ' Among them, T t and T I represents the encoded representation of the aggregated text and image, F t and F I Represents the original encoding representation of the image, F t ' represents the text-image feature attention vector, F I ' represents the image-text feature attention vector; The aggregated text description and scene image feature vectors are input into the contrastive loss function to train the model and perform backpropagation and optimization.

7. The cross-modality-based intelligent monitoring method according to claim 1, characterized in that: Step S2 includes: S20, regularly collect and upload the high-definition video stream of the monitoring platform; S21, using the KTS kernel temporal segmentation algorithm to segment the uploaded video into several disjoint segments and input them into the self-attention network, obtain the attention vector and generate the attention weight matrix through normalization; S22, inputting the attention weight matrix into a structure composed of multiple layers of neural networks connected in series, and calculating the importance score of the video frame; S23. Among the image frames whose calculated scores are greater than the threshold, two image frames whose inter-frame distance is greater than a specific threshold are saved as key frames of the video stream, wherein the inter-frame distance is the difference between the two frame sequence numbers.

8. The cross-modality-based intelligent monitoring method according to claim 7, characterized in that: Step S22 is specifically as follows: Input the attention weight matrix into a structure composed of multiple layers of neural networks connected in series, calculate the importance score of the video frame, and compress it within the range of 0-1; The multi-layer neural network includes an input layer, a linear layer, a Dropout layer, a RELU activation layer and an output layer.

9. A cross-modal intelligent monitoring system, characterized in that: The system comprises: A sample construction module is used to obtain a cross-modal data set, wherein the cross-modal data set includes text data describing a scene image and collected video stream data; A feature extraction module extracts key features of scene images and text descriptions in the cross-modal dataset; Model building module, building a cross-modal fusion model based on semantic enhancement; A model training module, which trains the semantically enhanced cross-modal fusion model according to the cross-modal data set, calculates the similarity between the cross-modal data, and obtains a model for recognizing custom description scenes; The timing monitoring module collects and uploads the high-definition video stream of the monitoring platform on a regular basis; The key frame extraction module extracts key frames based on the uploaded video stream; The alarm module detects the video key frames according to the custom scene description required for the warning in combination with the cross-modal fusion model. When a video picture that meets the custom scene description is detected, an alarm message is generated and the scene picture is saved.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of a cross-modality-based intelligent monitoring method described in any one of claims 1 to 8 are implemented.

Images