Video analysis method and device, equipment and storage medium

Abstract

The invention provides a video analysis method and device, equipment and a storage medium, and is applied to the technical field of computer technology and video recognition. The method comprises the following steps: based on a pre-trained video text multi-modal model, reasoning a plurality of pre-acquired video sequences to obtain video features, and further performing similarity calculation on the video features and a feature base library to obtain a similarity result corresponding to each video sequence, and determining at least one abnormal video sequence in the plurality of video sequences based on a preset similarity threshold and the similarity result corresponding to each video sequence, and pushing the at least one abnormal video sequence and the behavior characteristics corresponding to each abnormal video sequence to the user. And updating the feature base library based on the calibration of the user on the abnormal video. According to the method, the accuracy and the analysis efficiency of target behavior analysis in the video are improved, the detection capability of abnormal behaviors is gradually enhanced by continuously receiving the user feedback optimization model, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the fields of computer technology and video recognition technology, and in particular to a video analysis method, device, equipment and storage medium. Background Art

[0002] The behavior recognition model is a technology that uses computer vision and machine learning technology to identify and analyze human actions, postures, expressions and other behaviors in videos or images. It is mainly used in security monitoring, smart home and other fields. In the field of security monitoring, the behavior recognition model is used to analyze surveillance videos in real time, detect abnormal behaviors, and realize automatic identification and real-time warning of abnormalities.

[0003] At present, the widely used behavior recognition method is to use a behavior recognition model to classify the target behavior in the video, or to realize behavior recognition based on a multimodal video-text retrieval model.

[0004] However, the above method can only classify and identify limited categories of behaviors, and relies on professionals to construct text prompt words. Users cannot understand the impact of manually added text on model performance, resulting in low behavior recognition accuracy. Summary of the invention

[0005] The present application provides a video analysis method, apparatus, device and storage medium to solve the problem that the classification and identification of limited categories of behaviors in the prior art relies on professionals to construct text prompt words, and users cannot understand the impact of manually added text on model performance, resulting in low behavior recognition accuracy.

[0006] In a first aspect, the present application provides a video analysis method, comprising:

[0007] Based on the pre-trained video-text multimodal model, the pre-acquired multiple video sequences and feature base database are inferred and calculated to obtain the similarity result corresponding to each video sequence. The features of the behavior categories in the feature base database can be dynamically updated.

[0008] Based on a preset similarity threshold and a similarity result corresponding to each video sequence, determining at least one abnormal video sequence from the plurality of video sequences;

[0009] The at least one abnormal video sequence and the behavior feature corresponding to each abnormal video sequence are pushed to the user.

[0010] In combination with the first aspect, in some embodiments, the feature base includes a text feature base and / or a video feature base, the text feature base includes text features of multiple behavior categories, and the video feature base includes video features of multiple behavior categories.

[0011] In combination with the first aspect, in some embodiments, the method further includes:

[0012] Based on the semantic information of each abnormal video sequence, cluster analysis is performed on the at least one abnormal video sequence to obtain a plurality of clusters, each of which includes a plurality of abnormal video sequences and video features corresponding to each abnormal video sequence;

[0013] For each cluster, generate text features corresponding to the cluster;

[0014] Pushing each cluster and the text feature corresponding to each cluster to the user;

[0015] receiving a user calibration result sent by the user, wherein the user calibration result includes characteristics of a behavior type calibrated by the user;

[0016] The feature base database is updated based on the user calibration result.

[0017] In combination with the first aspect, in some embodiments, updating the feature base database based on the user calibration result includes:

[0018] If the user calibration result includes the features of the behavior category of each cluster calibrated by the user, then based on the text features corresponding to the behavior category of each cluster calibrated by the user, the text feature base library is updated;

[0019] If the user calibration result includes the features of the behavior category of each abnormal video sequence calibrated by the user, the video feature base library is updated based on the features of the behavior category of each abnormal video sequence calibrated by the user.

[0020] In combination with the first aspect, in some embodiments, generating the text features corresponding to the clusters includes:

[0021] For each cluster, based on a pre-trained visual language model, the cluster is analyzed to generate text information corresponding to the cluster, wherein the visual language model is trained based on a pre-built data set;

[0022] Encoding the text information corresponding to the cluster and / or the text information of the cluster sent by the user to obtain initial text features corresponding to the cluster;

[0023] Based on the correlation information between each initial text feature and the cluster and a preset text quantity threshold, the text feature corresponding to the cluster is determined.

[0024] In combination with the first aspect, in some embodiments, determining at least one abnormal video sequence from multiple video sequences based on a preset similarity threshold and a similarity result corresponding to each video sequence includes:

[0025] If the similarity threshold includes a text feature threshold, and the similarity between any video sequence and the text feature of the abnormal behavior category in the text feature base library is greater than the text feature threshold, the video sequence is determined to be an abnormal video sequence;

[0026] If the similarity threshold includes a video feature threshold, and the similarity between any video sequence and the video feature of the abnormal behavior category in the video feature base library is greater than the video feature threshold, the video sequence is determined to be an abnormal video sequence.

[0027] In combination with the first aspect, in some embodiments, the method further includes:

[0028] Based on a pre-constructed video-text dataset, the pre-constructed initial video-text multimodal model is trained by the LoRA algorithm to obtain the video-text multimodal model.

[0029] In a second aspect, the present application provides a video analysis device, comprising:

[0030] An inference module is used to perform inference calculations on a plurality of pre-acquired video sequences and a feature base library based on a pre-trained video-text multimodal model, and obtain a similarity result corresponding to each video sequence. The features of the behavior categories in the feature base library can be dynamically updated;

[0031] A processing module, configured to determine at least one abnormal video sequence from among the multiple video sequences based on a preset similarity threshold and a similarity result corresponding to each video sequence;

[0032] The push module is used to push the at least one abnormal video sequence and the behavior characteristics corresponding to each abnormal video sequence to the user.

[0033] In combination with the second aspect, in some embodiments, the feature base includes a text feature base and / or a video feature base, the text feature base includes text features of multiple behavior categories, and the video feature base includes video features of multiple behavior categories.

[0034] In conjunction with the second aspect, in some embodiments, the processing module may also be used to:

[0035] Based on the semantic information of each abnormal video sequence, cluster analysis is performed on the at least one abnormal video sequence to obtain a plurality of clusters, each of which includes a plurality of abnormal video sequences and video features corresponding to each abnormal video sequence;

[0036] For each cluster, generate text features corresponding to the cluster;

[0037] Pushing each cluster and the text feature corresponding to each cluster to the user;

[0038] receiving a user calibration result sent by the user, wherein the user calibration result includes characteristics of a behavior type calibrated by the user;

[0039] The feature base database is updated based on the user calibration result.

[0040] In conjunction with the second aspect, in some embodiments, the processing module updates the feature base library based on the user calibration result, specifically including:

[0041] If the user calibration result includes the features of the behavior category of each cluster calibrated by the user, then based on the text features corresponding to the behavior category of each cluster calibrated by the user, the text feature base library is updated;

[0042] If the user calibration result includes the features of the behavior category of each abnormal video sequence calibrated by the user, the video feature base library is updated based on the features of the behavior category of each abnormal video sequence calibrated by the user.

[0043] In conjunction with the second aspect, in some embodiments, the processing module generates text features corresponding to the clusters, specifically including:

[0044] For each cluster, based on a pre-trained visual language model, the cluster is analyzed to generate text information corresponding to the cluster, wherein the visual language model is trained based on a pre-built data set;

[0045] Encoding the text information corresponding to the cluster and / or the text information of the cluster sent by the user to obtain initial text features corresponding to the cluster;

[0046] Based on the correlation information between each initial text feature and the cluster and a preset text quantity threshold, the text feature corresponding to the cluster is determined.

[0047] In conjunction with the second aspect, in some embodiments, the processing module is further used to:

[0048] If the similarity threshold includes a text feature threshold, and the similarity between any video sequence and the text feature of the abnormal behavior category in the text feature base library is greater than the text feature threshold, the video sequence is determined to be an abnormal video sequence;

[0049] If the similarity threshold includes a video feature threshold, and the similarity between any video sequence and the video feature of the abnormal behavior category in the video feature base library is greater than the video feature threshold, the video sequence is determined to be an abnormal video sequence.

[0050] In conjunction with the second aspect, in some embodiments, the processing module is further used to:

[0051] Based on a pre-constructed video-text dataset, the pre-constructed initial video-text multimodal model is trained by the LoRA algorithm to obtain the video-text multimodal model.

[0052] In a third aspect, the present application provides an electronic device, comprising: a processor, a memory and a display connected to the processor in communication;

[0053] The memory stores computer-executable instructions;

[0054] The processor executes the computer-executable instructions stored in the memory to implement the video analysis method described in any of the aforementioned aspects.

[0055] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the video analysis method described in any of the aforementioned aspects.

[0056] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the video analysis method described in any of the preceding aspects.

[0057] The video analysis method, device, equipment and storage medium provided by the present application perform reasoning calculations on multiple pre-acquired video sequences and feature bases based on a pre-trained video-text multimodal model, obtain similarity results corresponding to each video sequence, and determine at least one abnormal video sequence in multiple video sequences based on a pre-set similarity threshold and the similarity results corresponding to each video sequence, and push the at least one abnormal video sequence and the behavioral features corresponding to each abnormal video sequence to the user. The above method improves the accuracy and efficiency of target behavior analysis in the video, and by continuously receiving user feedback to optimize the model, the ability to detect abnormal behavior is gradually enhanced, which improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 is an application system architecture diagram of an exemplary video analysis method;

[0060] Figure 2 A schematic diagram of a flow chart of a first embodiment of a video analysis method provided in an embodiment of the present application;

[0061] Figure 3 It is a schematic diagram of the partial structure of the model;

[0062] Figure 4 A schematic diagram of a flow chart of a second embodiment of a video analysis method provided in an embodiment of the present application;

[0063] Figure 5 It is a schematic diagram of the first visual interface;

[0064] Figure 6 A schematic diagram of the second visual interface;

[0065] Figure 7 A schematic diagram of the third visual interface;

[0066] Figure 8 A schematic diagram of a flow chart of a third embodiment of a video analysis method provided in an embodiment of the present application;

[0067] Fig. 9 A schematic diagram of a flow chart of a fourth embodiment of a video analysis method provided in an embodiment of the present application;

[0068] Fig.10 A schematic diagram of the fourth visual interface;

[0069] Fig.11 A flowchart of a fifth embodiment of the video analysis method provided in the present application;

[0070] Fig.12 A flowchart of a sixth embodiment of the video analysis method provided in the present application;

[0071] Fig.13 A flowchart of a specific example of a video analysis method provided in an embodiment of the present application;

[0072] Fig.14 A schematic diagram of the structure of a video analysis device embodiment provided in an embodiment of the present application;

[0073] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0074] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0075] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0076] The behavior recognition model is a technology that uses computer vision and machine learning technology to identify and analyze human movements, postures, expressions and other behaviors in videos or images. It is mainly used in security monitoring, smart home and other fields. In the field of security monitoring, the behavior recognition model is used to analyze surveillance videos in real time, detect abnormal behaviors, and realize automatic recognition and real-time warning of abnormalities. At present, the widely used behavior recognition method is to use the behavior recognition model to classify the target behavior in the video, or to realize behavior recognition based on the multimodal video-text retrieval model. However, the above method can only classify and identify limited categories of behaviors, and relies on text prompts constructed by professionals. Users cannot understand the impact of manually added text on model performance, resulting in low behavior recognition accuracy.

[0077] In view of the above problems, the present application provides a video analysis method, device, equipment and storage medium. Compared with the traditional closed set classification model, it can break through the limitation of the classification model category and judge the behavior category of the characters in the video by calculating the similarity between the video and the text. And compared with the user constructing the text by himself, it lowers the user's use threshold, improves the model operation efficiency, reduces the workload of the user's manual screening, and improves the accuracy of video analysis. Specifically, the currently widely used behavior recognition model is mainly based on the closed set classification model, which classifies and recognizes limited categories of behaviors by training the classifier model. Or it is based on the multimodal video-text retrieval model for behavior recognition, which can break through the limitation of the classification model category. Although the existing multimodal-based behavior recognition model can solve the pain point that the traditional classification model cannot perform open set recognition, when the user encounters the required text description that is not in the current text base library, the user needs to add the text by himself, and there is currently no way to visualize the added text in the industry, which will cause the user to be unable to evaluate the quality and effect of the added text in a WYSIWYG way. Taking these issues into consideration, the inventors studied whether it is possible to cluster abnormal videos and recommend related texts based on a video-text multimodal behavior analysis system based on user interaction feedback, and then use feature visualization to allow users to filter texts in a what-you-see-is-what-you-get manner, and perform behavior analysis of targets in the video based on an updated feature database to ensure the accuracy of the behavior analysis.

[0078] The video analysis method provided in the embodiment of the present application is mainly used in the fields of security monitoring, smart home, etc. In particular, in the field of security monitoring, it can analyze the monitoring video in real time, detect abnormal behavior, and then issue a system alarm to the user. Figure 1 FIG. 1 is an exemplary application system architecture diagram of a video analysis method, such as Figure 1 As shown, the system includes a video preprocessing module, a video text multimodal pretraining and comparison module, a result postprocessing module, a system alarm display module, a video clustering module, a text prompt word recommendation generation module, a text feature visualization and screening module, a video feature calibration module, and a performance evaluation and feedback module. The user can preprocess the video stream to be processed through the video preprocessing module to obtain multiple video sequences, and then perform inference calculations on the video text multimodal model obtained by pretraining with the video text multimodal pretraining and comparison module to obtain the similarity between each video sequence and each feature in the text feature base library and / or the video feature base library, and analyze the similarity obtained by the video text multimodal pretraining and comparison module through the result postprocessing module to obtain an abnormal video, and display the abnormal video to the user through the system alarm display module.

[0079] Optionally, the video-text multimodal pre-training and comparison module is based on a video-text dataset built for behavioral analysis tasks, and is combined with a fine-tuning method of contrastive learning to train a video-text multimodal model. The fine-tuning process uses the Low-Rank Adaptation of Large Language Models (LoRA) algorithm.

[0080] In order to improve the performance and recognition accuracy of the model and reduce false positives, the video clustering module is used to perform cluster analysis on multiple abnormal videos, and the video feature visualization module is used to generate a visualization interface for the clustered abnormal videos and display it to the user. During the user's calibration of the clustered abnormal videos, the text prompt word recommendation generation module will generate text information for each cluster cluster, and the user can also add text descriptions by themselves through the visualization interface. The text feature visualization and screening module visualizes and screens the generated text information and the text descriptions added by the user to obtain the final text features corresponding to each cluster cluster. The user can also calibrate each video through the video feature calibration module, and update the feature base based on the results of the text feature visualization and screening module and the video feature calibration module. The performance evaluation and feedback module will provide real-time feedback to the user on the performance of the updated feature base in the video text multimodal model.

[0081] Optionally, the visual interface presented to the user mentioned above may be presented to the user directly by the video processing device, or may be presented to the user through the user's terminal device, which is not specifically limited in the embodiments of the present application. The above video processing device may be an electronic device such as a laptop, a server, a cloud server, etc., and the user terminal device may be an electronic device such as a smart phone, a smart watch, a laptop, etc.

[0082] The embodiments of the present application do not specifically limit the form and type of the above-mentioned physical devices.

[0083] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0084] Figure 2 A schematic diagram of a flow chart of a first embodiment of a video analysis method provided in an embodiment of the present application, such as Figure 2 As shown, the method specifically includes:

[0085] S201: Based on a pre-trained video-text multimodal model, inference calculations are performed on multiple pre-acquired video sequences and feature base libraries to obtain similarity results corresponding to each video sequence.

[0086] In this step, in order to accurately detect the abnormal behavior of the target in the video, the multiple video sequences and feature base library acquired in advance are input into the video text multimodal model for inference calculation, so as to obtain the similarity result corresponding to each video sequence, wherein, for each video sequence, the similarity result includes the similarity between the video sequence and each feature in the feature base library. The feature base library also includes the features of the behavior categories calibrated by the user for the video sequence. Among them, the features of the behavior categories calibrated by the user for the video sequence are calibrated by the user in real time during the real-time processing of the video sequence, and then the features of the behavior categories in the feature base library are dynamically updated.

[0087] Specifically, in the process of processing multiple video sequences, the user can perform feature calibration based on multiple abnormal video sequences, and then dynamically update the feature base library based on the user's calibration results.

[0088] Exemplarily, the similarity may be calculated by calculating cosine similarity or Euclidean distance, which is not specifically limited in the embodiments of the present application.

[0089] Optionally, the feature base library includes a text feature base library and / or a video feature base library, the text feature base library includes text features of multiple behavior categories, and the video feature base library includes video features of multiple behavior categories. Then, the multiple video sequences and the feature base library are subjected to inference calculation, which may be a similarity calculation between each video sequence and each text feature in the text feature base library, or a similarity calculation between each video sequence and each video feature in the video feature base library, or a similarity calculation between each video sequence and each feature in the text feature base library and the video feature base library. The embodiment of the present application does not specifically limit the feature categories in the feature base library.

[0090] S202: Determine at least one abnormal video sequence from multiple video sequences based on a preset similarity threshold and a similarity result corresponding to each video sequence.

[0091] In this step, after the similarity between each video sequence and each feature in the feature base is obtained through the above-mentioned inference calculation, since the feature base includes features of multiple behavior categories, in order to detect anomalies in the video, at least one abnormal video sequence is determined among the multiple video sequences.

[0092] Specifically, if the similarity threshold includes a text feature threshold, and the similarity between any video sequence and the text feature of the abnormal behavior category in the text feature base library is greater than the text feature threshold, the video sequence is determined to be an abnormal video sequence. If the similarity threshold includes a video feature threshold, and the similarity between any video sequence and the video feature of the abnormal behavior category in the video feature base library is greater than the video feature threshold, the video sequence is determined to be an abnormal video sequence.

[0093] S203: Push at least one abnormal video sequence and the behavior feature corresponding to each abnormal video sequence to the user.

[0094] In this step, after the abnormal video sequences are determined, all abnormal video sequences and the behavior features corresponding to each abnormal video sequence are pushed to the user.

[0095] Specifically, the abnormal video and the behavioral characteristics corresponding to the abnormal video, the text description of the video and other information can be pushed to the user directly on the display, and an alarm prompt can be given. The alarm prompt can be a voice prompt or a pop-up screen prompt, or a text message alarm prompt can be sent to the user.

[0096] Optionally, the abnormal video and the behavioral characteristics corresponding to the abnormal video, the text description of the video and other information can be pushed to the user's terminal device, dynamically displayed on the display interface of the user terminal device, and then prompted by voice, pop-up screen or text message.

[0097] Optionally, the above-mentioned video-text multimodal model is based on a pre-constructed video-text dataset and is trained by the LoRA algorithm. The LoRA algorithm is used to fine-tune the model to improve model performance. During the fine-tuning process, the video encoder is fine-tuned using a feedforward network combined with the LoRA algorithm. Similar to the traditional method of adding a LoRA module to the self-attention module, the feedforward network combined with the LoRA algorithm adds a LoRA module next to the multilayer perceptron after the self-attention of the original Transformer encoder, and the output value of the LoRA module is added to the output of the multilayer perceptron to obtain the final encoding result. Figure 3 It is a schematic diagram of the partial structure of the model. Figure 3 As shown, since the feedforward network combined with the LoRA algorithm introduces low-rank matrix decomposition in the multi-layer perceptron bypass, it can learn the relevant information of downstream tasks while maintaining the original parameters and performance of the model. Therefore, it can quickly adapt to the needs of behavior analysis tasks without overfitting or affecting the effect of the original pre-trained model.

[0098] The video analysis method provided in this embodiment is based on a pre-trained video text multimodal model, performs inference calculations on multiple pre-acquired video sequences and feature bases, obtains similarity results corresponding to each video sequence, and based on a pre-set similarity threshold and the similarity results corresponding to each video sequence, determines at least one abnormal video sequence in multiple video sequences, and pushes the at least one abnormal video sequence and the behavior features corresponding to each abnormal video sequence to the user. The above method improves the accuracy and efficiency of target behavior analysis in the video, and by continuously receiving user feedback to optimize the model, the ability to detect abnormal behavior is gradually enhanced, which improves the user experience.

[0099] Figure 4 A schematic diagram of a flow chart of a second embodiment of a video analysis method provided in an embodiment of the present application, such as Figure 4 As shown, based on the above embodiment, in the process of processing multiple video sequences, the user can perform feature calibration based on multiple abnormal video sequences, and then the specific implementation method of dynamically updating the feature base library based on the user's calibration results includes:

[0100] S401: Based on the semantic information of each abnormal video sequence, cluster analysis is performed on at least one abnormal video sequence to obtain a plurality of clusters.

[0101] In this step, in order to improve the accuracy and flexibility of the video text multimodal model, and to lower the threshold for users and reduce the workload of manual screening, after detecting abnormal video sequences, at least one abnormal video sequence can be clustered to obtain multiple clusters, wherein each cluster includes multiple abnormal video sequences and video features corresponding to each abnormal video sequence, and the semantic information of the abnormal video sequences in each cluster is similar.

[0102] Exemplarily, the clustering method used above may include:

[0103] K-means clustering (k-means clustering): It iteratively optimizes the location of cluster centers to minimize the sum of squared errors within the cluster.

[0104] Agglomerative hierarchical clustering: Start with each point as a separate cluster and gradually merge the nearest clusters until all data points are merged into one cluster or a predetermined number of clusters is reached.

[0105] Divisive hierarchical clustering: All data starts as one cluster and then gradually splits into smaller clusters.

[0106] Balanced Iterative Reducing and Clustering Using Hierarchies (BIRCH): Clustering is performed by building a hierarchy between data points.

[0107] Density-Based Spatial Clustering of Applications with Noise (DBSCAN): It can identify clusters of arbitrary shapes and is robust to noise and outliers.

[0108] Ordering Points To Identify the Clustering Structure (OPTICS): Similar to DBSCAN, but performs better for datasets of varying density.

[0109] Mean shift clustering: It finds the area with the highest data point density by iteratively updating the candidate cluster center points, without knowing the number of clusters in advance.

[0110] The embodiments of the present application do not specifically limit the specific implementation method of clustering.

[0111] S402: For each cluster, generate text features corresponding to the cluster.

[0112] In this step, after a plurality of clusters are obtained, a text feature corresponding to each cluster is automatically generated.

[0113] Specifically, for each cluster, the cluster is analyzed based on a pre-trained visual language model to generate text information corresponding to the cluster. The text information corresponding to the cluster and / or the text information sent by the user to the cluster is encoded to obtain the initial text features corresponding to the cluster. Based on the correlation information between each initial text feature and the cluster and a pre-set text quantity threshold, the text features corresponding to the cluster are determined.

[0114] S403: Push each cluster and the text feature corresponding to each cluster to the user.

[0115] In this step, in order to improve the accuracy and flexibility of the model, the user can interact with the user so that the user can intuitively view the optimization process of the model performance. After obtaining the text features of each cluster, all clusters and the text features corresponding to each cluster are aggregated into feature balls (2D or 3D) and then visually displayed in the interface and pushed to the user. The user can modify the calibration in the visual interface.

[0116] For example, Figure 5 This is a schematic diagram of the first visual interface, such as Figure 5 As shown in the figure, after clustering, the abnormal videos in each cluster are represented by different shapes, and the number of videos in each cluster is clearly visible.

[0117] S404: Receive the user calibration result sent by the user.

[0118] Users can modify the calibration in the visual interface.

[0119] Optionally, the user can modify the calibration in the visualization interface by calibrating each cluster or each video. Figure 6 To visualize the second interface diagram, Figure 5 Based on the above, a text description is automatically added to each cluster, and the user can also enter the text description by himself. Then the user can select the text features retained by the cluster, and the user can mark each cluster. For example, if the user thinks that a cluster is a false positive, then the text features corresponding to the cluster will be marked as negative text features. Among them, negative text features refer to normal text features, not abnormal text features. If the corresponding video sequence is normal, not abnormal, it is a false positive.

[0120] In another possible implementation, the user can also calibrate each abnormal video. Figure 7A schematic diagram of the third visual interface is shown in FIG. Figure 7 As shown, the user labels each abnormal video in the cluster. If the user believes that an abnormal video is a false alarm, the feature of the abnormal video will be marked as a negative video feature.

[0121] S405: Update the feature base database based on the user calibration result.

[0122] In this step, the feature base database can be updated based on the user calibration results to ensure the accuracy of the behavior analysis.

[0123] Specifically, if the user calibration result includes the features of the behavior category of each cluster cluster calibrated by the user, the text feature base library is updated based on the text features corresponding to the behavior category of each cluster cluster calibrated by the user; if the user calibration result includes the features of the behavior category of each abnormal video sequence calibrated by the user, the video feature base library is updated based on the features of the behavior category of each abnormal video sequence calibrated by the user.

[0124] The video analysis method provided in this embodiment performs cluster analysis on at least one abnormal video sequence based on the semantic information of each abnormal video sequence to obtain multiple clusters, generates text features corresponding to each cluster for each cluster, pushes each cluster and the text features corresponding to each cluster to the user, receives the user calibration results sent by the user, and updates the feature base based on the user calibration results. Through the above-mentioned real-time interactive method of user calibration and system feedback, users can filter text and videos in a WYSIWYG manner, and finally provide real-time feedback on the impact of user operations on model performance, which is more conducive to intuitively improving model performance and reducing false alarms.

[0125] Figure 8 The flowchart of the third embodiment of the video analysis method provided in the embodiment of the present application is as follows: Figure 8 As shown, based on the above embodiments, in step S405, the specific implementation method of updating the feature base database based on the user calibration result includes:

[0126] S801: If the user calibration result includes features of the behavior category of each cluster calibrated by the user, then based on the text features corresponding to the behavior category of each cluster calibrated by the user, the text feature base library is updated.

[0127] In this step, after receiving the user calibration result fed back by the user, the user calibration result is checked. If the user calibration result includes the text features corresponding to the behavior category of each clustering cluster calibrated by the user, the text base library is updated.

[0128] Specifically, in the above embodiment Figure 6For example, if the user believes that a cluster is a false positive, the text feature corresponding to the cluster will be marked as a negative text feature. Then the text feature corresponding to the cluster is added to the text feature base library and marked as a negative text feature.

[0129] S802: If the user calibration result includes the features of the behavior category of each abnormal video sequence calibrated by the user, then based on the features of the behavior category of each abnormal video sequence calibrated by the user, the video feature base library is updated.

[0130] In this step, after receiving the user calibration result fed back by the user, the user calibration result is checked. If the user calibration result includes the characteristics of the behavior category of each abnormal video sequence calibrated by the user, the video feature base library is updated.

[0131] Specifically, in the above embodiment Figure 7 For example, the user labels each abnormal video in the cluster. If the user believes that a certain abnormal video is a false alarm, the feature of the abnormal video will be marked as a negative video feature. The video feature of the abnormal video will be added to the video feature base library and marked as a negative sample. If there are abnormal videos in the subsequent processing that have similar video features to the abnormal video, they will be classified as negative examples.

[0132] The video analysis method provided in this embodiment, if the user calibration result includes the characteristics of the behavior category of each cluster cluster calibrated by the user, then the text feature base library is updated based on the text features corresponding to the behavior category of each cluster cluster calibrated by the user; if the user calibration result includes the characteristics of the behavior category of each abnormal video sequence calibrated by the user, then the video feature base library is updated based on the characteristics of the behavior category of each abnormal video sequence calibrated by the user. By effectively updating the feature base library based on the user calibration result, the accuracy and generalization ability of the behavior recognition model are improved.

[0133] Based on the above method embodiments 2 and 3, the specific description of the dynamic update of the feature base database is described below by way of example:

[0134] It is assumed that the method of the embodiment of the present application is applied to a behavior recognition system for shopping mall security monitoring, which is designed to automatically identify and report abnormal behaviors in the shopping mall.

[0135] For user calibration process:

[0136] First, cluster analysis is performed on abnormal behaviors in the mall surveillance video. For example:

[0137] Cluster A: contains multiple video clips in which people are fighting over goods, which may involve theft.

[0138] Cluster B: contains multiple video clips in which people are pushing and fighting each other.

[0139] Cluster C: contains multiple video clips in which people suddenly fall to the ground, possibly involving a fall or a sudden illness.

[0140] For each cluster, the system generates descriptive text features that are written based on the common features of the video clips in the cluster. For example:

[0141] Text features of cluster A: "Many people are fighting over goods, there is physical conflict, and theft may be involved."

[0142] Text features of cluster B: "Many people were pushing and fighting each other, and the scene was chaotic."

[0143] Text features of cluster C: "Someone suddenly fell to the ground, possibly due to a fall or a sudden illness."

[0144] The system collects abnormal behavior category labels assigned by security monitoring personnel for each cluster: "theft behavior", "fighting behavior" and "fall / sudden illness".

[0145] Text feature extraction: In this example, the user-defined labels are text features themselves. Therefore, the system directly adds these labels as text features to the text feature base library and sets them as negative examples, i.e. abnormal behaviors.

[0146] Update the database: The system creates new entries for these tags in the text feature database. Each entry contains a behavior category label (such as "theft behavior") and the text features associated with it (i.e. the label itself).

[0147] The system retains the original data or previously extracted features (such as spatiotemporal features extracted by CNN) of each video clip in the cluster. At the same time, the system also collects the behavior category labels marked by security monitoring personnel for each cluster.

[0148] It should be noted that although in this example the user calibrates at the cluster level rather than the video sequence level, the system still retains a unique identifier for each video clip and associates it with the cluster label, making it easier to align video clips with their corresponding labels when needed.

[0149] Update the database: The system creates a new entry for each video clip in the video feature database. Each entry contains a video clip identifier, extracted video features (such as CNN features), and the associated behavior category label (such as "theft"), and is set as a negative example, that is, abnormal behavior.

[0150] Fig. 9 A schematic diagram of a fourth embodiment of the video analysis method provided in the present application is shown in FIG. Fig. 9 As shown, based on the above embodiments, step S402 specifically includes:

[0151] S901: For each cluster, based on the pre-trained visual language model, analyze the cluster and generate text information corresponding to the cluster.

[0152] In this step, in order to lower the threshold for users, facilitate users, save user time and improve the accuracy of video analysis, a large visual language model is used to analyze each cluster, thereby generating text information corresponding to the cluster.

[0153] Exemplarily, the visual language model is trained based on a pre-constructed data set, which includes video-text data in order to obtain richer text feature representation. The visual language model is trained based on the data set. Among them, the model can establish an association between visual input and text description. The key frames in each cluster are used as visual input. Representative frames of each cluster (such as centroid, median, or randomly selected multiple frames) can be selected as input, or the feature vectors of all frames can be aggregated (such as average pooling) as input. The visual language model is used to reason about the input and generate text descriptions associated with the visual content. These text descriptions can summarize the common themes or emotions of the videos in the clusters.

[0154] Optional, such as Figure 6 As shown, the text recommended by the system is the text information automatically generated in this step. Since the process is visualized by the user, the user can also add text information by himself.

[0155] S902: Encode the text information corresponding to the cluster and / or the text information of the cluster sent by the user to obtain the initial text features corresponding to the cluster.

[0156] In this step, after obtaining the text information corresponding to each cluster and / or the text information sent by the user to the cluster, the text information is encoded to generate initial text features.

[0157] Exemplarily, the text information is encoded into 512-dimensional text features for subsequent dimensionality reduction and screening.

[0158] S903: Determine the text feature corresponding to the cluster based on the correlation information between each initial text feature and the cluster and a preset text quantity threshold.

[0159] In this step, after obtaining the initial text features of each cluster, in order to accurately obtain the text features of each cluster, it is necessary to perform text feature screening, and determine the text features corresponding to the cluster in the initial text features based on a pre-set text quantity threshold.

[0160] For example, Fig.10 This is a schematic diagram of the fourth interface, as shown in Fig.10 As shown in the figure, the initial text features are processed by dimensionality reduction, and the distribution of the text features is visualized. The text features of each cluster are combined with the video features of each abnormal video in each cluster to generate a visualization diagram of feature correlation. The square scattered points in the feature ball represent the text features after dimensionality reduction, the triangular scattered points represent the video features in the cluster, and the circular points represent the cluster center of the video cluster. The user can intuitively see the correlation between each text feature and the cluster. The closer the text is to the cluster center, the higher the correlation. The preset number of initial text features closest to the cluster center can be automatically selected as the text features of the cluster, or the user can select the initial text features to be retained.

[0161] Optionally, the above-mentioned dimensionality reduction of text features may adopt a dimensionality reduction algorithm, for example, a dimensionality reduction algorithm based on feature selection or a dimensionality reduction algorithm based on feature extraction, which is not specifically limited in the embodiments of the present application.

[0162] The video analysis method provided in this embodiment analyzes each cluster based on a pre-trained visual language model, generates text information corresponding to the cluster, encodes the text information corresponding to the cluster and / or the text information sent by the user to obtain the initial text features corresponding to the cluster, and determines the text features corresponding to the cluster based on the correlation information between each initial text feature and the cluster and a pre-set text quantity threshold. The above-mentioned use of the visual language model to automatically generate recommended text prompt words lowers the user's usage threshold compared to users constructing text by themselves, and automatically screens through visual text features, improves the model operation efficiency, and reduces the workload of users' manual screening.

[0163] Fig.11 The flowchart of the fifth embodiment of the video analysis method provided in the present application is as follows: Fig.11 As shown, based on the above embodiments, step S202 specifically includes:

[0164] S1101: If the similarity threshold includes a text feature threshold, and the similarity between any video sequence and the text feature of the abnormal behavior category in the text feature base library is greater than the text feature threshold, the video sequence is determined to be an abnormal video sequence.

[0165] S1102: If the similarity threshold includes the video feature threshold, and the similarity between any video sequence and the video feature of the abnormal behavior category in the video feature base library is greater than the video feature threshold, the video sequence is determined to be an abnormal video sequence.

[0166] In order to accurately analyze the video, a similarity threshold is set in advance. If the similarity threshold includes a text feature threshold, the similarity between each video sequence and the text feature of the abnormal behavior category in the text feature base library is compared with the text feature threshold, and the video sequence whose similarity between each video sequence and the text feature of the abnormal behavior category in the text feature base library is greater than the text feature threshold is determined as an abnormal video sequence. If the similarity threshold includes a video feature threshold, similarly, the similarity between each video sequence and the video feature of the abnormal behavior category in the video feature base library is compared with the video feature threshold, and the video sequence whose similarity between each video sequence and the video feature of the abnormal behavior category in the video feature base library is greater than the video feature threshold is determined as an abnormal video sequence.

[0167] The video analysis method provided in this embodiment is that if the similarity threshold includes a text feature threshold, and the similarity between any video sequence and the text feature of the abnormal behavior category in the text feature base library is greater than the text feature threshold, the video sequence is determined to be an abnormal video sequence; if the similarity threshold includes a video feature threshold, and the similarity between any video sequence and the video feature of the abnormal behavior category in the video feature base library is greater than the video feature threshold, the video sequence is determined to be an abnormal video sequence. By comparing the text features and video features of the video sequence with the abnormal behavior data in the feature base library, the video can be analyzed from two different but complementary dimensions. The possibility of misjudgment is reduced and the accuracy of recognition is improved. And it can flexibly adapt to different types of data sources.

[0168] Fig.12 The flowchart of the sixth embodiment of the video analysis method provided in the present application is as follows: Fig.12 As shown, based on the above embodiments, the method further includes:

[0169] S1201: Perform object detection on the pre-acquired video stream to obtain the position information of each object in the video stream.

[0170] S1202: Track each target based on the position information of each target to obtain multiple video sequences.

[0171] In order to accurately analyze the behavior of the target in the video stream and determine the abnormality and issue an alarm, it is necessary to pre-process the pre-acquired video stream to obtain multiple video sequences.

[0172] Specifically, target detection is performed on the target in the video stream. Target detection is to detect the target position in the picture, and then target tracking processing is performed. The same target is associated in time sequence, and the corresponding video sequence of each target is output.

[0173] Exemplarily, first, the video stream is decomposed into a series of separate image frames. In order to improve the detection accuracy, the image frames can be preprocessed, such as resizing, normalization, denoising, etc. Then, a target detection algorithm is selected, such as a single detection algorithm (You Only Look Once, YOLO), a single shot multibox detector (SSD), a Histogram of Oriented Gradients (HOG), a Deformable Part Model (DPM), etc. Then, a target tracking process is performed to use a target tracking algorithm, such as Simple Online and Realtime Tracking (SORT), Deep Simple Online and Realtime Tracking (DeepSORT), etc., to associate the detected targets across frames.

[0174] The video analysis method provided in this embodiment performs target detection on the pre-acquired video stream to obtain the position information of each target in the video stream, and tracks each target based on the position information of each target to obtain multiple video sequences. By performing target detection and target tracking on the pre-acquired video stream, the real-time and accuracy of monitoring are improved, the depth of behavior analysis is enhanced, and the sensitivity of abnormal detection is improved.

[0175] Fig.13 A flow chart of a specific example of the video analysis method provided in the embodiment of the present application is as follows: Fig.13 As shown, the method includes:

[0176] S1301: Video stream preprocessing.

[0177] This step is the same as the specific implementation method of the video stream preprocessing in the above embodiment, and will not be repeated here.

[0178] S1302: Video text multimodal model analysis.

[0179] This step is implemented in the same manner as step S201 in the aforementioned embodiment, and will not be described in detail here.

[0180] S1303: Results and post-processing.

[0181] S1304: System alarm.

[0182] The specific implementation of step S1303 and step S1304 is the same as the specific implementation of step S202 and step S203 in the above embodiment, and will not be repeated here.

[0183] S1305: Video feature clustering.

[0184] This step is implemented in the same manner as step S401 in the above-mentioned embodiment, and will not be described in detail here.

[0185] S1306: Visualizing video features.

[0186] S1307: Generate text prompt word recommendations.

[0187] S1308: Text feature visualization and screening.

[0188] S1309: Video feature calibration.

[0189] S1310: Update the feature database.

[0190] The above steps are the same as the specific implementation methods of the aforementioned method embodiments 2 to 4, and will not be repeated here.

[0191] S1311: Performance evaluation and feedback.

[0192] From the above steps, it can be seen that after the feature base library is updated, when the video sequence is subjected to inference analysis, the updated feature base library is applied to the model. After a closed-loop cycle of interaction with the user, the performance of the model is continuously optimized. After each receipt of user information, the evaluation result of the model optimization based on the information sent by the user is fed back to the user in real time, thereby enabling the user to filter texts and videos in a what-you-see-is-what-you-get manner. Finally, the impact of the user's operation on the model performance is fed back in real time, which is more beneficial to intuitively improve the model performance and reduce false alarms.

[0193] The video analysis method provided in this embodiment is based on the clustering results of abnormal videos, combined with the recommended text prompt words of the large language model, and then the user is allowed to calibrate and select the clustering results and the recommended text, and finally the video features and text features of the model are updated according to the user's selection. The video text multimodal behavior analysis system based on user interactive feedback uses a feature visualization method to allow users to filter text in a what-you-see-is-what-you-get manner, and finally provides real-time feedback on the impact of user operations on model performance.

[0194] Fig.14 The schematic diagram of the structure of the video analysis device embodiment 1 provided in the present application is as follows: Fig.14 As shown, the video analysis device 1400 includes:

[0195] The reasoning module 1401 is used to perform reasoning calculations on multiple pre-acquired video sequences and feature base libraries based on a pre-trained video-text multimodal model to obtain similarity results corresponding to each video sequence. The features of the behavior categories in the feature base library can be dynamically updated.

[0196] The processing module 1402 is configured to determine at least one abnormal video sequence from a plurality of video sequences based on a preset similarity threshold and a similarity result corresponding to each video sequence.

[0197] The push module 1403 is used to push at least one abnormal video sequence and the behavior feature corresponding to each abnormal video sequence to the user.

[0198] Optionally, the feature base library includes a text feature base library and / or a video feature base library, the text feature base library includes text features of multiple behavior categories, and the video feature base library includes video features of multiple behavior categories.

[0199] In one or more embodiments, the processing module 1402 may also be used to:

[0200] Based on the semantic information of each abnormal video sequence, cluster analysis is performed on at least one abnormal video sequence to obtain multiple clusters, each of which includes multiple abnormal video sequences and video features corresponding to each abnormal video sequence.

[0201] For each cluster, generate the text features corresponding to the cluster.

[0202] Each cluster and the text feature corresponding to each cluster are pushed to the user.

[0203] A user calibration result sent by a user is received, where the user calibration result includes characteristics of a behavior type calibrated by the user.

[0204] The feature base database is updated based on the user calibration results.

[0205] In one or more embodiments, the processing module 1402 updates the feature base library based on the user calibration result, specifically including:

[0206] If the user calibration result includes the features of the behavior category of each clustering cluster calibrated by the user, the text feature base library is updated based on the text features corresponding to the behavior category of each clustering cluster calibrated by the user.

[0207] If the user calibration result includes the features of the behavior category of each abnormal video sequence calibrated by the user, the video feature base library is updated based on the features of the behavior category of each abnormal video sequence calibrated by the user.

[0208] In one or more embodiments, the processing module 1402 generates text features corresponding to the clusters, specifically including:

[0209] For each cluster, the cluster is analyzed based on a pre-trained visual language model to generate text information corresponding to the cluster. The visual language model is trained based on a pre-built data set.

[0210] The text information corresponding to the clusters and / or the text information of the clusters sent by the user is encoded to obtain the initial text features corresponding to the clusters.

[0211] Based on the correlation information between each initial text feature and the cluster and a preset text quantity threshold, the text feature corresponding to the cluster is determined.

[0212] In one or more embodiments, the processing module 1402 is further configured to:

[0213] If the similarity threshold includes a text feature threshold, and the similarity between any video sequence and the text feature of the abnormal behavior category in the text feature base library is greater than the text feature threshold, the video sequence is determined to be an abnormal video sequence.

[0214] If the similarity threshold includes the video feature threshold, and the similarity between any video sequence and the video feature of the abnormal behavior category in the video feature base library is greater than the video feature threshold, the video sequence is determined to be an abnormal video sequence.

[0215] In one or more embodiments, the processing module 1402 is further configured to:

[0216] Perform object detection on the pre-acquired video stream to obtain the position information of each object in the video stream.

[0217] Each target is tracked based on the position information of each target to obtain multiple video sequences.

[0218] The video analysis device provided in the above embodiment is used to execute the video analysis method in any of the above method embodiments. Its implementation principle and technical effect are similar and will not be repeated here.

[0219] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0220] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Fig.15 As shown, the electronic device 1500 includes: a processor 1502, a memory 1501 and a display 1503 which are communicatively connected to the processor 1502;

[0221] The memory 1501 stores computer executable instructions;

[0222] The processor 1502 executes the computer-executable instructions stored in the memory 1501 to implement the video analysis method in any method embodiment.

[0223] Display 1503 is used to display alarm information of abnormal video to the user and to display text features and / or video feature visualization interface to the user.

[0224] Optionally, the above-mentioned components of the electronic device may be connected via a system bus.

[0225] The system bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The memory 1501 may include a random access memory 1501 (RAM), and may also include a non-volatile memory 1501 (NVM), such as at least one disk storage 1501.

[0226] All or part of the steps of the above-mentioned method embodiments can be completed by hardware related to program instructions. The above-mentioned program can be stored in a readable memory 1501. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the above-mentioned memory 1501 (storage medium) includes: read-only memory 1501 (ROM), RAM, flash memory 1501, hard disk, solid state hard disk, magnetic tape, floppy disk, optical disc and any combination thereof.

[0227] The electronic device provided in the embodiment of the present application can be used to execute the method provided in any of the above method embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0228] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to execute the video analysis methods provided in the above-mentioned various implementation modes.

[0229] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0230] Optionally, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0231] An embodiment of the present application also provides a computer program product, which includes a computer program. The computer program is stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, the technical solution provided by any of the above method embodiments can be implemented.

[0232] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship; in the formula, the character " / " indicates that the previous and next associated objects are in a "division" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0233] It is to be understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not intended to limit the scope of the embodiments of the present application. In the embodiments of the present application, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0234] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0235] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A video analysis method, characterized in that: include: Based on the pre-trained video-text multimodal model, the pre-acquired multiple video sequences and feature base database are inferred and calculated to obtain the similarity result corresponding to each video sequence. The features of the behavior categories in the feature base database can be dynamically updated. Based on a preset similarity threshold and a similarity result corresponding to each video sequence, determining at least one abnormal video sequence from the multiple video sequences; The at least one abnormal video sequence and the behavior feature corresponding to each abnormal video sequence are pushed to the user.

2. The method according to claim 1, characterized in that The feature base database includes a text feature base database and / or a video feature base database. The text feature base database includes text features of multiple behavior categories, and the video feature base database includes video features of multiple behavior categories.

3. The method according to claim 2, characterized in that The method further comprises: Based on the semantic information of each abnormal video sequence, cluster analysis is performed on the at least one abnormal video sequence to obtain a plurality of clusters, each of which includes a plurality of abnormal video sequences and video features corresponding to each abnormal video sequence; For each cluster, generate text features corresponding to the cluster; Pushing each cluster and the text feature corresponding to each cluster to the user; receiving a user calibration result sent by the user, wherein the user calibration result includes characteristics of a behavior type calibrated by the user; The feature base database is updated based on the user calibration result.

4. The method according to claim 3, characterized in that The updating of the feature base database based on the user calibration result includes: If the user calibration result includes the features of the behavior category of each clustering cluster calibrated by the user, then based on the text features corresponding to the behavior category of each clustering cluster calibrated by the user, the text feature base library is updated; If the user calibration result includes the features of the behavior category of each abnormal video sequence calibrated by the user, the video feature base library is updated based on the features of the behavior category of each abnormal video sequence calibrated by the user.

5. The method according to claim 3, characterized in that: The generating of text features corresponding to the clusters includes: For each cluster, based on a pre-trained visual language model, the cluster is analyzed to generate text information corresponding to the cluster, wherein the visual language model is trained based on a pre-built data set; Encoding the text information corresponding to the cluster and / or the text information of the cluster sent by the user to obtain initial text features corresponding to the cluster; Based on the correlation information between each initial text feature and the cluster and a preset text quantity threshold, the text feature corresponding to the cluster is determined.

6. The method according to claim 1, characterized in that The step of determining at least one abnormal video sequence from a plurality of video sequences based on a preset similarity threshold and a similarity result corresponding to each video sequence includes: If the similarity threshold includes a text feature threshold, and the similarity between any video sequence and the text feature of the abnormal behavior category in the text feature base library is greater than the text feature threshold, the video sequence is determined to be an abnormal video sequence; If the similarity threshold includes a video feature threshold, and the similarity between any video sequence and the video feature of the abnormal behavior category in the video feature base library is greater than the video feature threshold, the video sequence is determined to be an abnormal video sequence.

7. The method according to claim 1, characterized in that The method further comprises: Based on a pre-constructed video-text dataset, the pre-constructed initial video-text multimodal model is trained by a low-rank adaptation LoRA algorithm of a large language model to obtain the video-text multimodal model.

8. A video analysis device, characterized in that: include: An inference module is used to perform inference calculations on a plurality of pre-acquired video sequences and a feature base library based on a pre-trained video-text multimodal model, and obtain a similarity result corresponding to each video sequence. The features of the behavior categories in the feature base library can be dynamically updated; A processing module, configured to determine at least one abnormal video sequence from among the multiple video sequences based on a preset similarity threshold and a similarity result corresponding to each video sequence; The push module is used to push the at least one abnormal video sequence and the behavior characteristics corresponding to each abnormal video sequence to the user.

9. An electronic device, characterized in that: include: A processor, a memory and a display connected in communication with the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the video analysis method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the video analysis method according to any one of claims 1 to 7 when executed by a processor.

Images