Cross-modal data video retrieval and alarm method

Abstract

The invention relates to the technical field of cross-modal data, and particularly discloses a cross-modal data video retrieval and alarm method, which comprises the following steps of: S1, setting keywords, and dividing the keywords into normal keywords and abnormal keywords; obtaining image data of keywords, and establishing a keyword feature library; s2, acquiring a behavior sample video stream of the keyword, dividing the behavior sample video stream into an abnormal behavior video stream and a conventional behavior video stream, and training according to the behavior sample video stream to obtain a behavior recognition model; s3, analyzing the video stream uploaded by the user, and if a keyword to be analyzed is identified in an image frame and the keyword is marked as an abnormal keyword, directly prompting that the video stream is abnormal; if the identified keyword is a conventional keyword, acquiring a video segment, and identifying whether the video segment is an abnormal behavior video stream or not by using a behavior identification model; and counting the alarm quantity.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal data, and in particular to a cross-modal data video retrieval and alarm method. Background Art

[0002] In scenarios such as construction sites and security that require video surveillance, the traditional method is to manually observe the surveillance images for a long time to identify and determine whether there are abnormal situations. This method is not only inefficient, but also prone to errors, because it is difficult for people to maintain their attention for a long time, especially when faced with a large number of repetitive and monotonous surveillance images. In addition, due to the limitations of human vision and cognition, it is difficult to process multiple surveillance images at the same time or capture fast-occurring events. This manual-dependent video surveillance method has gradually shown its limitations in modern security management, especially in situations where real-time and accuracy are extremely high. Therefore, more and more monitoring platforms are beginning to need automated intelligent technology to assist or even replace manual monitoring to improve the efficiency and accuracy of monitoring. Therefore, the present invention proposes a cross-modal data video retrieval and alarm method. Summary of the invention

[0003] The purpose of the present invention is to provide a cross-modal data video retrieval and alarm method to solve the above technical problems.

[0004] The purpose of the present invention can be achieved through the following technical solutions:

[0005] A cross-modal data video retrieval and alarm method comprises the following steps:

[0006] Step S1: setting a number of keywords and dividing the keywords into abnormal keywords and regular keywords; obtaining image data of the keywords and extracting image features of the image data; obtaining a keyword feature library based on the image features of each keyword;

[0007] Step S2: Obtaining a behavior sample video stream of the conventional keyword, dividing the behavior sample video stream into an abnormal behavior video stream and a regular behavior video stream; establishing a convolutional neural network model, and training the convolutional neural network model through the behavior sample video stream to obtain a behavior recognition model;

[0008] Step S3: obtaining a video stream uploaded by a user, dividing the video stream into a number of image frames, obtaining all keywords in the image frames, and recording them as keywords to be analyzed; analyzing the image frames according to the image features to be analyzed, the keywords to be analyzed, and the keyword feature library;

[0009] If the keyword to be analyzed is identified in the image frame and the keyword to be analyzed is an abnormal keyword, it is directly prompted that an abnormality exists in the video stream once; if the keyword to be analyzed is identified in the image frame and the keyword to be analyzed is a regular keyword, the next N image frames of the image frame are obtained, where N is a preset image frame number threshold, and a video segment is obtained according to the image frame and the next N image frames; the behavior recognition model is used to identify whether the video segment is an abnormal behavior video stream, and if it is an abnormal behavior video stream, it is prompted that an abnormality exists in the video stream once;

[0010] All image frames in the video stream are analyzed in sequence, and the total number of abnormalities is statistically analyzed, and the total number is recorded as the alarm quantity, and the alarm quantity is displayed to the user.

[0011] As a further solution of the present invention: the keywords include animal names and object names.

[0012] As a further solution of the present invention: dividing the keywords into abnormal keywords and regular keywords includes:

[0013] Acquire the scene in the video stream, acquire the historical image of the scene, identify all the keywords appearing in the historical image through image recognition technology, and record the number of occurrences of each keyword; select the keywords whose number of occurrences exceeds a preset number of occurrences threshold and record them as regular keywords, and record the remaining keywords as abnormal keywords.

[0014] As a further solution of the present invention: the process of dividing the behavior sample video stream into abnormal behavior video stream and normal behavior video stream is based on manual division.

[0015] As a further solution of the present invention: the process of identifying whether there is a keyword to be analyzed in the image frame includes:

[0016] According to the keyword feature library, the similarity between the image feature to be analyzed and each image feature of the image frame is obtained, and the image feature with the highest similarity to the image feature to be analyzed and the similarity exceeds a preset similarity threshold is selected; the keyword corresponding to the image feature is obtained and recorded as a similar keyword; the semantic similarity between the similar keyword and the keyword to be analyzed is obtained through natural language processing technology, and if the semantic similarity exceeds the preset similarity threshold, the keyword to be analyzed is recorded as the similar keyword.

[0017] As a further solution of the present invention: if the image feature with the highest similarity to the image feature to be analyzed has a similarity less than or equal to a preset similarity threshold, the keyword to be analyzed does not belong to the keyword feature library; the keyword to be analyzed is directly recorded as an abnormal keyword.

[0018] As a further solution of the present invention: the process of setting the image frame number threshold includes:

[0019] The frame numbers of all behavior sample video streams are obtained, where the frame number is the total number of image frames in the behavior sample video streams; the average value of the frame numbers of all behavior sample video streams is obtained to obtain the frame number mean value, which is recorded as the image frame number threshold.

[0020] As a further solution of the present invention: if the video segment is identified as a regular behavior video stream through the behavior recognition model, then there is no abnormality in the current image frame.

[0021] Beneficial effects of the present invention:

[0022] In traditional video surveillance, manual continuous observation is often required, which is not only inefficient, but also prone to missed or misjudgment due to human factors. Compared with the prior art, the behavior recognition model in the present invention can automatically identify abnormal behaviors in video streams, greatly improving the real-time detection of abnormalities; it can analyze video content, and once abnormal behavior is detected, it can immediately trigger an alarm mechanism to ensure that security incidents are handled in a timely manner; the behavior recognition model can continuously learn and optimize the recognition algorithm through deep learning technology to improve the accuracy and reliability of recognition; this means that with the passage of time and the accumulation of usage data, the model will become more intelligent and can more accurately identify various complex abnormal behaviors; this self-evolution ability is difficult to match with traditional methods; the present invention also reduces the need for manual observation, reduces labor costs and the risk of human error. In automated video surveillance systems, manual intervention is usually limited and mainly focuses on system configuration and subsequent processing of abnormal events. Therefore, the implementation of the present invention not only improves the efficiency of the system, but also enhances its stability and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention will be further described below in conjunction with the accompanying drawings.

[0024] Figure 1 It is a flow chart of a cross-modal data video retrieval and alarm method of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] See also Figure 1As shown, the present invention is a cross-modal data video retrieval and alarm method, comprising the following steps:

[0027] Step S1: setting a number of keywords and dividing the keywords into abnormal keywords and regular keywords; obtaining image data of the keywords and extracting image features of the image data; obtaining a keyword feature library based on the image features of each keyword;

[0028] In a preferred embodiment of the present invention, the keywords include animal names and object names;

[0029] It should be noted that the abnormal keywords usually refer to objects or behaviors that should not appear in a specific scenario or require special attention, such as "illegal intruder" and "fire smoke". Regular keywords refer to objects or behaviors that often appear in the scene and usually do not cause alarms, such as "pedestrians" and "normal vehicles".

[0030] Step S2: Obtaining a behavior sample video stream of the conventional keyword, dividing the behavior sample video stream into an abnormal behavior video stream and a regular behavior video stream; establishing a convolutional neural network model, and training the convolutional neural network model through the behavior sample video stream to obtain a behavior recognition model;

[0031] It should be noted that, for example, at a construction site, the abnormal behavior video stream includes workers taking off their safety helmets; in an operating workshop, the abnormal behavior video stream includes workshop workers making operating errors;

[0032] It should be noted that the process of training the convolutional neural network model includes:

[0033] CNN is used to extract features from the preprocessed video stream. CNN automatically learns the hierarchical feature representation of the input data through multi-layer convolution and pooling operations, and uses the extracted features and corresponding label information to train CNN. During the training process, the weights and biases of the model are optimized through the back propagation algorithm so that the model can accurately predict the labels of the input data; the performance of the CNN model is evaluated on an independent test set, and the CNN model is optimized based on the evaluation results. Possible optimization methods include adjusting the network structure, changing the training strategy, and increasing the amount of data.

[0034] As a preferred embodiment of the present invention, dividing the keywords into abnormal keywords and regular keywords includes:

[0035] Obtain a scene in the video stream, obtain a historical image of the scene, identify all keywords appearing in the historical image by image recognition technology, and record the number of occurrences of each keyword; select keywords whose number of occurrences exceeds a preset number of occurrences threshold as regular keywords, and record the remaining keywords as abnormal keywords;

[0036] It is understandable that for each scene, its historical images are further obtained, and the historical images may include multiple views of the same scene at different time points or different angles; according to a preset threshold of the number of occurrences, it is determined which keywords are regular keywords; if the number of times a keyword appears in the historical images is large, it means that the keyword appears in the video stream frequently or normally;

[0037] As a preferred embodiment of the present invention, the process of dividing the behavior sample video stream into the abnormal behavior video stream and the normal behavior video stream is based on manual division;

[0038] Step S3: obtaining a video stream uploaded by a user, dividing the video stream into a number of image frames, obtaining all keywords in the image frames, and recording them as keywords to be analyzed; analyzing the image frames according to the image features to be analyzed, the keywords to be analyzed, and the keyword feature library;

[0039] If the keyword to be analyzed is identified in the image frame and the keyword to be analyzed is an abnormal keyword, it is directly prompted that an abnormality exists in the video stream once; if the keyword to be analyzed is identified in the image frame and the keyword to be analyzed is a regular keyword, the next N image frames of the image frame are obtained, where N is a preset image frame number threshold, and a video segment is obtained according to the image frame and the next N image frames; the behavior recognition model is used to identify whether the video segment is an abnormal behavior video stream, and if it is an abnormal behavior video stream, it is prompted that an abnormality exists in the video stream once;

[0040] Analyze all image frames in the video stream in sequence, statistically analyze the total number of abnormalities, record the total number as the alarm quantity, and display the alarm quantity to the user;

[0041] It can be understood that a video segment is generated based on the current image frame and the N subsequent image frames. This video segment contains richer temporal and spatial information, which helps to capture and understand the behavior pattern of the target object;

[0042] It is worth noting that all image frames in the video stream are analyzed in turn and the above steps are repeated; based on the comprehensive analysis, the total number of abnormalities in the video stream, i.e., the alarm quantity, is counted; finally, the counted alarm quantity is displayed to the user, and the user can understand the abnormal situation in the video stream according to the alarm quantity and take corresponding measures to deal with it;

[0043] As a preferred embodiment of the present invention, the process of identifying whether there is a keyword to be analyzed in the image frame includes:

[0044] According to the keyword feature library, the similarity between the image feature to be analyzed of the image frame and each image feature is obtained, and the image feature with the highest similarity to the image feature to be analyzed and the similarity exceeds a preset similarity threshold is selected; the keyword corresponding to the image feature is obtained and recorded as a similar keyword; the semantic similarity between the similar keyword and the keyword to be analyzed is obtained by natural language processing technology, and if the semantic similarity exceeds the preset similarity threshold, the keyword to be analyzed is recorded as the similar keyword;

[0045] If the image feature with the highest similarity to the image feature to be analyzed has a similarity less than or equal to a preset similarity threshold, the keyword to be analyzed does not belong to the keyword feature library; the keyword to be analyzed is directly recorded as an abnormal keyword;

[0046] It can be understood that if the keyword to be analyzed is an abnormal keyword, it means that it is unreasonable for the thing corresponding to the keyword to be analyzed to appear in the video scene, and the abnormality is directly recorded once; if the keyword to be analyzed is a regular keyword and the behavior in the video segment is abnormal, the abnormality is recorded once;

[0047] It is worth noting that if the keyword of the abnormality found in consecutive image frames is the same keyword, then the abnormality is recorded as once in total in these consecutive image frames; similarly, if the keyword of the abnormal behavior found in consecutive image frames is the same keyword and the behavior is the same, then the abnormality is recorded as once in total in these consecutive image frames;

[0048] As a preferred embodiment of the present invention, the process of setting the image frame number threshold includes:

[0049] Obtaining the number of frames of all behavior sample video streams, where the number of frames is the total number of image frames in the behavior sample video streams; obtaining an average value of the number of frames of all behavior sample video streams to obtain a frame number mean value, which is recorded as an image frame number threshold;

[0050] It is understandable that after obtaining the frame numbers of all behavior sample video streams, the average of these frame numbers is calculated to obtain the frame number mean. This step eliminates the influence of the length difference of a single video stream on the threshold setting through a statistical method, making the threshold more representative and universal.

[0051] As a preferred embodiment of the present invention, if the video segment is identified as a regular behavior video stream through the behavior recognition model, there is no abnormality in the current image frame.

[0052] It is worth noting that the present invention also includes the retrieval of video content, and the process includes: after the user uploads the video stream, the keywords input by the user are obtained and recorded as the keywords to be searched; all the keywords in all the image frames of the video stream are obtained, and when the keyword identical to the keyword to be searched is identified, the keyword is recorded as the target keyword, and the image frame where the target keyword is located is marked and displayed to the user.

[0053] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A cross-modal data video retrieval and alarm method, characterized in that: The following steps are involved: Step S1: setting a number of keywords and dividing the keywords into abnormal keywords and regular keywords; obtaining image data of the keywords and extracting image features of the image data; obtaining a keyword feature library based on the image features of each keyword; Step S2: Obtaining a behavior sample video stream of the conventional keyword, dividing the behavior sample video stream into an abnormal behavior video stream and a regular behavior video stream; establishing a convolutional neural network model, and training the convolutional neural network model through the behavior sample video stream to obtain a behavior recognition model; Step S3: obtaining a video stream uploaded by a user, dividing the video stream into a number of image frames, obtaining all keywords in the image frames, and recording them as keywords to be analyzed; analyzing the image frames according to the image features to be analyzed, the keywords to be analyzed, and the keyword feature library; If the keyword to be analyzed is identified in the image frame and the keyword to be analyzed is an abnormal keyword, it is directly prompted that an abnormality exists in the video stream once; If the keyword to be analyzed is identified in the image frame and the keyword to be analyzed is a regular keyword, then the next N image frames of the image frame are obtained, where N is a preset threshold of the number of image frames, and a video segment is obtained according to the image frame and the next N image frames; the behavior recognition model is used to identify whether the video segment is an abnormal behavior video stream, and if it is an abnormal behavior video stream, it is prompted that there is an abnormality in the video stream once; All image frames in the video stream are analyzed in sequence, and the total number of abnormalities is statistically analyzed, and the total number is recorded as the alarm quantity, which is displayed to the user.

2. A cross-modal data video retrieval and alarm method according to claim 1, characterized in that: In step S1, the keywords include animal names and object names.

3. A cross-modal data video retrieval and alarm method according to claim 2, characterized in that: In step S2, dividing the keywords into abnormal keywords and regular keywords includes: Acquire the scene in the video stream, acquire the historical image of the scene, identify all the keywords appearing in the historical image through image recognition technology, and record the number of occurrences of each keyword; select the keywords whose number of occurrences exceeds a preset number of occurrences threshold and record them as regular keywords, and record the remaining keywords as abnormal keywords.

4. A cross-modal data video retrieval and alarm method according to claim 1, characterized in that: In step S2, the process of dividing the behavior sample video stream into abnormal behavior video stream and normal behavior video stream is based on manual division.

5. A cross-modal data video retrieval and alarm method according to claim 1, characterized in that: In step S3, the process of identifying whether there is a keyword to be analyzed in the image frame includes: According to the keyword feature library, the similarity between the image feature to be analyzed and each image feature of the image frame is obtained, and the image feature with the highest similarity to the image feature to be analyzed and the similarity exceeds a preset similarity threshold is selected; the keyword corresponding to the image feature is obtained and recorded as a similar keyword; the semantic similarity between the similar keyword and the keyword to be analyzed is obtained through natural language processing technology, and if the semantic similarity exceeds the preset similarity threshold, the keyword to be analyzed is recorded as the similar keyword.

6. A cross-modal data video retrieval and alarm method according to claim 5, characterized in that: In step S3, if the image feature with the highest similarity to the image feature to be analyzed has a similarity less than or equal to a preset similarity threshold, the keyword to be analyzed does not belong to the keyword feature library; the keyword to be analyzed is directly recorded as an abnormal keyword.

7. A cross-modal data video retrieval and alarm method according to claim 1, characterized in that: In step S3, the process of setting the image frame number threshold includes: The frame numbers of all behavior sample video streams are obtained, where the frame number is the total number of image frames in the behavior sample video streams; the average value of the frame numbers of all behavior sample video streams is obtained to obtain the frame number mean value, which is recorded as the image frame number threshold.

8. A cross-modal data video retrieval and alarm method according to claim 1, characterized in that: In step S3, if the video segment is identified as a regular behavior video stream by the behavior recognition model, there is no abnormality in the current image frame.

Images