Ship lock area monitoring and early warning system and method based on non-training video anomaly detection
By using a method based on untrained video anomaly detection, a pre-trained model is used to analyze lock video data, generate anomaly scores and issue early warning signals, which solves the problem of high data dependence in lock detection and achieves efficient anomaly identification and low-cost monitoring.
Patent Information
- Application Number
- CN202510785733.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-12
- Publication Date
- 2025-09-19
AI Technical Summary
In ship lock detection, existing technologies require a large amount of training data support, resulting in high costs and poor generalization. It is difficult to identify anomalies outside the data set and is difficult to apply.
By acquiring lock video data, using pre-trained visual language models and large language models, subtitle sequence analysis and noise reduction are performed to generate initial and final anomaly scores, and ultimately generate early warning signals.
The accuracy and generalization ability of lock area monitoring can be improved without training, abnormal events can be effectively identified, and costs can be reduced.
Smart Images
Figure CN120673392A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of video anomaly detection, and specifically provides a ship lock area monitoring and early warning system and method based on untrained video anomaly detection. Background Art
[0002] Video anomaly detection involves locating anomalies within a given video. Examples include identifying home burglaries in surveillance footage or detecting people potentially in danger downstream before a dam is flooded. These tasks are difficult because anomalies typically occur infrequently, making it difficult to obtain sufficient training data. Furthermore, anomalies are often difficult to precisely define, and their detection and identification relies on context, placing high demands on the detection algorithm.
[0003] In application scenarios, most of them rely on a large amount of labeled data to train models. For example, given frame-level or video-level annotations, fully supervised or multi-instance learning algorithms are used to detect anomalies. In addition, data sets are provided to pre-built networks during the training phase to learn the distribution of normal data. After deployment, data outside the distribution after network processing is considered anomalies. The above training methods all have good performance, but in actual lock detection, obtaining relevant data on the locks and training the relevant data require a large amount of training data as support, which leads to high usage costs, poor generalization, and difficulty in obtaining the ability to identify anomalies outside the data set, making it difficult to apply. Summary of the Invention
[0004] The present application aims to solve at least one of the technical problems existing in the prior art; to this end, the present application proposes a lock area monitoring and early warning system and method based on untrained video anomaly detection, which is used to solve the technical problems that in actual lock detection, obtaining relevant data of the lock and training the relevant data require a large amount of training data as support, which leads to high cost of use, poor generalization, difficulty in obtaining the ability to identify anomalies outside the data set, and greater application difficulty.
[0005] To achieve the above objectives, the first aspect of the present application provides a lock area monitoring and early warning method based on untrained video anomaly detection, comprising:
[0006] Acquire video data of the target lock area;
[0007] Analyzing the video data to obtain a first subtitle sequence;
[0008] Performing noise reduction processing on the first subtitle sequence to obtain a second subtitle sequence;
[0009] The second subtitle sequence is analyzed based on a large language model to obtain a summary subtitle sequence for detecting the accuracy of the lock area information;
[0010] An initial anomaly score is obtained based on the analysis of the summary subtitle sequence to indicate the degree of anomaly.
[0011] An analysis is performed based on the initial anomaly score to obtain a final anomaly score indicating improved scoring accuracy; and an early warning signal is generated based on the final anomaly score.
[0012] Preferably, obtaining the first subtitle sequence based on the analysis of the video data includes:
[0013] The video data represents video data of monitoring the lock area;
[0014] Dividing the video data into various video data sets through a video acquisition device; extracting video frames from each video in the video data set; the video frames include various image frames;
[0015] Input each image frame into a subtitle generation model to obtain a text description for describing the image content; integrate each text description into a subtitle sequence, recorded as the first subtitle sequence;
[0016] Each video data set is denoted as V = {V1,…,V j ,…,V N},V j Represents the jth video, 1≤j≤N, N represents the total number of videos, and extracts video frames from each video in the video dataset V, that is, Represents the i-th frame image of the j-th video, 1≤i≤M, M represents the total number of frames of the current video;
[0017] Preferably, each image frame is input into a caption generation model to obtain a text description for describing the image content, including:
[0018] The caption generation model generates a corresponding text description for each image frame; the text description is obtained as follows:
[0019]
[0020] Among them, each text description is recorded as The subtitle generation model is denoted as φ C (·); Each image frame is recorded as
[0021] Analyze each text description to get the first subtitle sequence, which is recorded as The subtitle generation model is used to obtain image information of ships, water level marks, etc., and then generate corresponding text descriptions; wherein the subtitle generation model is obtained by pre-training the blip2 visual language model.
[0022] Preferably, performing noise reduction processing on the first subtitle sequence to obtain the second subtitle sequence includes:
[0023] After extracting the features of each image frame using the modal encoder, the features of the current image frame are compared with the features of all image frames in the first subtitle sequence corresponding to the frame set to which it belongs. The text description with the highest similarity to the image frame features is recorded as the denoised text description. The above formula is applied to all text descriptions in the first subtitle sequence to obtain a second subtitle sequence. The image frames and text descriptions are in a one-to-one correspondence, with one image frame corresponding to one text description.
[0024] Text description of the first subtitle sequence Perform noise reduction processing to obtain The specific formula is as follows:
[0025]
[0026] Among them, ε I , ε T They are image and text feature encoders respectively, and <·,·> represents cosine similarity. After extracting the features of each image frame using the modal encoder ε(·), the features of the current image frame are converted to All image frame features {ε T (k)|k∈C j} to compare and take the text description with the highest similarity to the image frame feature as the text description after denoising The argmax function represents the value of the independent variable that makes the function reach the maximum value; the second subtitle sequence is a denoised subtitle sequence, and the text description in the denoised subtitle sequence is recorded as the denoised text description;
[0027] The second subtitle sequence is obtained by calculating the above formula for all text descriptions in the first subtitle sequence
[0028] Preferably, the second subtitle sequence is analyzed based on a large language model to obtain a summary subtitle sequence for detecting the accuracy of the lock area information, including:
[0029] The second subtitle sequence is input into the large language model and combined with the time window sampling setting to perform time domain summarization to obtain a summarized subtitle sequence; the specific formula for summarizing subtitles is as follows:
[0030]
[0031] in, A summary text description of each image frame; Indicates that in the i-th frame image The video clip obtained by sampling N frames within the time window T set as the center The subtitle set corresponding to each frame in P S Represents the prompt word for the large language model temporal understanding and summarization task, ° represents the splicing operation; the large language model is recorded as Ω LLM (·), where large language models include LLAMA models, etc.
[0032] The above formula is calculated for all the denoised text descriptions in the second subtitle sequence to obtain the summary subtitle sequence recorded as
[0033] In this application, a large language model is used to complete the time domain summarization task, so that it can understand the meaning of the input subtitles, correctly summarize the content and give summary subtitles, thereby improving the accuracy of abnormal event detection in ship locks, and having the ability to understand the time sequence of consecutive events, so as to accurately grasp the dynamics of the event.
[0034] Preferably, the initial anomaly score for indicating the degree of abnormality of the event is obtained based on the analysis of the summary subtitle sequence, including:
[0035] Get the preset subtitle prompt words, which include prompt words that provide context information and format requirements; respectively denoted as P C 、P F ; Input the summary text description and subtitle prompt words into the large language model to obtain the initial anomaly score; The specific formula for the initial anomaly score is as follows:
[0036]
[0037] The above formula is calculated for all summary text descriptions in the summary subtitle sequence to obtain the initial anomaly score, which is recorded as Each initial anomaly score is recorded as
[0038] in, represents the initial anomaly score of each image frame; the obtained initial anomaly score only considers the information within the time window set for the current image frame;
[0039] Preferably, the analysis based on the initial anomaly score to obtain a final anomaly score for indicating improved scoring accuracy includes:
[0040] For each video frame, the corresponding window video clip is compared with the corresponding features through the video encoder, and the corresponding features of the subtitle sequence are extracted and summarized using the text encoder. Then, a cross-modal comparison is performed to obtain the subtitles with the closest video features.
[0041] Among them, each video frame is recorded as its corresponding window video segment The video encoder is denoted as ε V(·), the video encoder extracts the corresponding features and records them as The text encoder extracts the corresponding features of the summary subtitle sequence and records them as ε T (S j ); the text encoder is denoted as ε T (·); the number of subtitles closest to the given number is H;
[0042] The final anomaly score is calculated by weighted summing up the initial anomaly scores corresponding to the closest subtitles;
[0043] The above formula is calculated for each image frame to obtain the final abnormality score, which is recorded as
[0044] Preferably, a weighted sum of the initial anomaly scores corresponding to the closest subtitles is performed to obtain a final anomaly score, including:
[0045] The confidence coefficient of the initial anomaly score is obtained by summarizing the similarity between the corresponding features of the subtitle sequence and the corresponding features of the video clip. The confidence coefficient can be understood as the weight coefficient of the final anomaly score. The specific formula for the final anomaly score is as follows:
[0046]
[0047] in, represents each final anomaly score; Indicates S j The index number set corresponding to each of the h subtitles, is the initial anomaly score corresponding to the image frame with index h in the jth video, and <·,·> represents the cosine similarity;
[0048] Preferably, generating an early warning signal based on the final anomaly score includes:
[0049] Presetting a safety threshold for the final anomaly score; comparing the final anomaly score with its safety threshold; wherein the safety threshold is obtained from the database;
[0050] When the final anomaly score exceeds its corresponding safety threshold, an early warning signal is issued;
[0051] When the final anomaly score does not exceed the corresponding safety threshold, a safety signal is issued.
[0052] The second embodiment of the present application provides a lock area monitoring and early warning system based on untrained video anomaly detection, including:
[0053] Data acquisition module: acquires video data of the target lock area;
[0054] Subtitle processing module: Analyzes video data to obtain a first subtitle sequence; performs noise reduction processing on the first subtitle sequence to obtain a second subtitle sequence; and analyzes the second subtitle sequence based on a large language model to obtain a summary subtitle sequence for detecting the accuracy of lock area information.
[0055] Anomaly warning module: Analyzes the summary subtitle sequence to obtain an initial anomaly score to indicate the degree of anomaly; analyzes the initial anomaly score to obtain a final anomaly score to improve scoring accuracy; and generates an early warning signal based on the final anomaly score.
[0056] Compared with the prior art, the present invention has the following advantages:
[0057] 1. The present application obtains video data of a target ship lock area; obtains a first subtitle sequence based on analysis of the video data; obtains a second subtitle sequence by performing noise reduction processing on the first subtitle sequence; obtains a summary subtitle sequence for detecting the accuracy of ship lock area information based on analysis of the second subtitle sequence based on a large language model; obtains an initial anomaly score indicating the degree of abnormality of an event based on analysis of the summary subtitle sequence; obtains a final anomaly score based on analysis of the initial anomaly score; and generates an early warning signal based on the final anomaly score. The present invention utilizes a pre-trained visual language model and a large language model, and can work without any training. By utilizing the visual and semantic priors brought by pre-training, the present invention fundamentally solves the problems of data collection difficulties and poor network generalization in traditional training methods, thereby improving the accuracy of early warnings for ship lock areas.
[0058] 2. This application uses cross-modal feature extraction to map video frames, video clips, and text descriptions into a unified feature space. Through feature comparison, weighting, replacement, and other post-processing, it effectively improves the accuracy of information extraction, improves anomaly detection performance, and thus improves the accuracy of ship lock detection;
[0059] 3. This application effectively utilizes the inherent memory and logical reasoning capabilities of large language models by setting text prompts for them, achieves the summary of text descriptions and the scoring of anomaly in lock scene descriptions, and improves anomaly detection capabilities. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0061] Figure 1 This is a schematic diagram of the principles of this application;
[0062] Figure 2 This is a system diagram of this application. DETAILED DESCRIPTION
[0063] The following will clearly and completely describe the technical solutions of this application in conjunction with the embodiments. Obviously, the embodiments described are only a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0064] See also Figure 1 The first embodiment of the present application provides a lock area monitoring and early warning method based on untrained video anomaly detection, comprising:
[0065] Acquire video data of the target lock area;
[0066] Analyzing the video data to obtain a first subtitle sequence;
[0067] Performing noise reduction processing on the first subtitle sequence to obtain a second subtitle sequence;
[0068] The second subtitle sequence is analyzed based on a large language model to obtain a summary subtitle sequence for detecting the accuracy of the lock area information;
[0069] An initial anomaly score is obtained based on the analysis of the summary subtitle sequence to indicate the degree of anomaly.
[0070] An analysis is performed based on the initial anomaly score to obtain a final anomaly score indicating improved scoring accuracy; and an early warning signal is generated based on the final anomaly score.
[0071] Specifically, obtaining the first subtitle sequence based on the analysis of the video data includes:
[0072] The video data represents video data of monitoring the lock area;
[0073] Dividing the video data into various video data sets through a video acquisition device; extracting video frames from each video in the video data set; the video frames include various image frames;
[0074] Input each image frame into a subtitle generation model to obtain a text description for describing the image content; integrate each text description into a subtitle sequence, recorded as the first subtitle sequence;
[0075] Each video data set is denoted as V = {V1,…,V j ,…,V N},V jRepresents the jth video, 1≤j≤N, N represents the total number of videos, and extracts video frames from each video in the video dataset V, that is, represents the i-th frame image of the j-th video, 1≤i≤M, M represents the total number of frames of the current video; the first subtitle sequence is used to represent the text descriptions corresponding to the video frames;
[0076] In this embodiment, it should be specifically noted that tools such as OpenCV can be used to read and save video frames with serial numbers for subsequent processing. Prior to this, to facilitate batch processing by the machine, video editing tools such as ffmpeg can be used to unify the video frame sampling rate in advance. Alternatively, the frame rate of each video can be saved and an index file can be constructed to facilitate subsequent extraction of the corresponding frame rate by video name. The first subtitle sequence is a collection of integrated text descriptions.
[0077] Specifically, each image frame is input into the caption generation model to obtain a text description for describing the image content, including:
[0078] The caption generation model generates a corresponding text description for each image frame; the text description is obtained as follows:
[0079]
[0080] Among them, each text description is recorded as The subtitle generation model is denoted as φ C (·); Each image frame is recorded as
[0081] Analyze each text description to get the first subtitle sequence, which is recorded as The subtitle generation model is used to obtain image information of ships, water level marks, etc., and then generate corresponding text descriptions; wherein the subtitle generation model is obtained by using the blip2 pre-trained visual language model;
[0082] The text description is text used to describe the image content; the subtitle generation model is obtained by artificial intelligence.
[0083] Specifically, performing noise reduction processing on the first subtitle sequence to obtain the second subtitle sequence includes:
[0084] After extracting the features of each image frame using the modal encoder, the features of the current image frame are compared with the features of all image frames in the first subtitle sequence corresponding to the frame set to which it belongs. The text description with the highest similarity to the image frame features is recorded as the denoised text description. The above formula is applied to all text descriptions in the first subtitle sequence to obtain a second subtitle sequence. The image frames and text descriptions are in a one-to-one correspondence, with one image frame corresponding to one text description.
[0085] Text description of the first subtitle sequence Perform noise reduction processing to obtain The specific formula is as follows:
[0086]
[0087] Among them, ε I , ε T They are image and text feature encoders respectively, and <·,·> represents cosine similarity. After extracting the features of each image frame using the modal encoder ε(·), the features of the current image frame are converted to All image frame features {ε T (k)|k∈C j} to compare and take the text description with the highest similarity to the image frame feature as the text description after denoising The argmax function represents the value of the independent variable that makes the function reach the maximum value; the second subtitle sequence is a denoised subtitle sequence, and the text description in the denoised subtitle sequence is recorded as the denoised text description;
[0088] The second subtitle sequence is obtained by calculating the above formula for all text descriptions in the first subtitle sequence
[0089] In this embodiment, it should be specifically noted that the image and text feature encoders in the imagebind multimodal pre-training model can be used as ε I , ε T Implement cross-modal feature extraction to obtain a unified spatial feature vector, and then use tools such as Faiss to implement vector feature comparison and retrieval;
[0090] Specifically, the second subtitle sequence is analyzed based on a large language model to obtain a summary subtitle sequence for detecting the accuracy of the lock area information, including:
[0091] The second subtitle sequence is input into the large language model and combined with the time window sampling setting to perform time domain summarization to obtain a summarized subtitle sequence; the specific formula for summarizing subtitles is as follows:
[0092]
[0093] in, A summary text description of each image frame; Indicates that in the i-th frame image The video clip obtained by sampling N frames within the time window T set as the center The subtitle set corresponding to each frame in P S Represents the prompt word for the large language model temporal understanding and summarization task, ° represents the splicing operation; the large language model is recorded as Ω LLM (·), where large language models include LLAMA models, etc.
[0094] The above formula is calculated for all the denoised text descriptions in the second subtitle sequence to obtain the summary subtitle sequence recorded as
[0095] In this embodiment, it is specifically necessary to explain that a large language model such as LLAMA can be used and denoted as Ω LLM (·), T = N = 10. Prompt word P S , such as "Please briefly summarize the events based on the given timeline of the scene, omitting any irrelevant descriptions", which aims to enable the large language model to complete the time domain summary task and enable it to understand the input subtitles The significance of Contents give summary subtitles j ; Summary: The subtitle sequence is a subtitle sequence used to detect the accuracy of the lock area information.
[0096] In this embodiment, it is specifically necessary to explain that the video clip The subtitle set corresponding to each frame in the training dataset, such as "The vessel decelerates and approaches the lock gate", "The lockgate initiates the closing procedure", "The water level difference is detected within the normal range", etc.; the prompt words for summarizing the task, such as "Please briefly summarize the key events based on the following timeline description:"; finally, the summary subtitles output by the model It could be "After the ship slowed down and approached the gate, the gate began to close and the current water level difference is within a safe range."
[0097] In this application, a large language model is used to complete the time domain summarization task, so that it can understand the meaning of the input subtitles, correctly summarize the content and give summary subtitles, thereby improving the accuracy of abnormal event detection in ship locks, and having the ability to understand the time sequence of consecutive events, so as to accurately grasp the dynamics of the event.
[0098] Specifically, an initial anomaly score is obtained based on the analysis of the summary subtitle sequence to indicate the degree of abnormality of the event, including:
[0099] Get the preset subtitle prompt words, which include prompt words that provide context information and format requirements; respectively denoted as P C 、P F ; Input the summary text description and subtitle prompt words into the large language model to obtain the initial anomaly score; The specific formula for the initial anomaly score is as follows:
[0100]
[0101] The above formula is calculated for all summary text descriptions in the summary subtitle sequence to obtain the initial anomaly score, which is recorded as Each initial anomaly score is recorded as
[0102] in, represents the initial anomaly score of each image frame; the obtained initial anomaly score only considers the information within the time window set for the current image frame;
[0103] In this embodiment, it should be specifically noted that the prompt words providing contextual information, such as "On a scale from 0 to 1, how would you evaluate the scene, where 0 indicates a typical scene and 1 suggests the presence of suspicious or potentially criminal behavior", are specifically intended to enable the model to understand the meaning of the input subtitles and determine the degree of abnormality of the scene described by the input subtitles. The specific details can be changed according to actual conditions; prompt words in required formats are provided, such as "Please give the values in a list", which requires an initial anomaly score to be given for subsequent processing; the initial anomaly score indicates the degree of abnormality of the event.
[0104] Specifically, the final anomaly score used to improve the scoring accuracy is obtained by analyzing the initial anomaly score, including:
[0105] For each video frame, the corresponding window video clip is compared with the corresponding features through the video encoder, and the corresponding features of the subtitle sequence are extracted and summarized using the text encoder. Then, a cross-modal comparison is performed to obtain the subtitles with the closest video features.
[0106] Among them, each video frame is recorded as its corresponding window video segment The video encoder is denoted as ε V (·), the video encoder extracts the corresponding features and records them as The text encoder extracts the corresponding features of the summary subtitle sequence and records them as ε T (S j ); the text encoder is denoted as ε T (·); the number of subtitles closest to the given number is H;
[0107] The final anomaly score is calculated by weighted summing up the initial anomaly scores corresponding to the closest subtitles;
[0108] The above formula is calculated for each image frame to obtain the final abnormality score, which is recorded as
[0109] The final anomaly score represents an anomaly score that improves scoring accuracy;
[0110] Specifically, the final anomaly score is obtained by performing a weighted sum calculation on the initial anomaly scores corresponding to the closest subtitles, including:
[0111] The confidence coefficient of the initial anomaly score is obtained by summarizing the similarity between the corresponding features of the subtitle sequence and the corresponding features of the video clip. The confidence coefficient can be understood as the weight coefficient of the final anomaly score. The specific formula for the final anomaly score is as follows:
[0112]
[0113] in, represents each final anomaly score; Indicates S j The index number set corresponding to each of the h subtitles, is the initial anomaly score corresponding to the image frame with index h in the jth video, and <·,·> represents the cosine similarity;
[0114] In this embodiment, it should be specifically explained that the confidence coefficient of the initial anomaly score is obtained by summarizing the similarity between the corresponding features of the subtitles and the corresponding features of the video clips in the formula, and the initial anomaly score is weighted and summed to obtain the final anomaly score, which integrates the score information of multiple video frames; the number of subtitles is 10, that is, the initial score is corrected using the first 10 most similar subtitle features to obtain the final anomaly score; compared with the initial anomaly score, the final anomaly score is a combination of the score information of multiple video frames; that is, the entire video information of the lock is taken into consideration.
[0115] Specifically, warning signals are generated based on the final anomaly score, including:
[0116] Presetting a safety threshold for the final anomaly score; comparing the final anomaly score with its safety threshold; wherein the safety threshold is obtained from the database;
[0117] When the final anomaly score exceeds its corresponding safety threshold, an early warning signal is issued;
[0118] When the final anomaly score does not exceed the corresponding safety threshold, a safety signal is issued.
[0119] See also Figure 2 The second embodiment of the present application provides a ship lock area monitoring and early warning system based on untrained video anomaly detection, comprising:
[0120] Data acquisition module: acquires video data of the target lock area;
[0121] Subtitle processing module: Analyzes video data to obtain a first subtitle sequence; performs noise reduction processing on the first subtitle sequence to obtain a second subtitle sequence; and analyzes the second subtitle sequence based on a large language model to obtain a summary subtitle sequence for detecting the accuracy of lock area information.
[0122] Anomaly warning module: Analyzes the summary subtitle sequence to obtain an initial anomaly score to indicate the degree of anomaly; analyzes the initial anomaly score to obtain a final anomaly score to improve scoring accuracy; and generates an early warning signal based on the final anomaly score.
[0123] Some of the data in the above formula are calculated by removing the dimensions and taking their numerical values. The formula is a formula that is closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.
[0124] The working principle of the present application is as follows: video data of the target lock area is obtained; a first subtitle sequence is obtained based on analysis of the video data; a second subtitle sequence is obtained by noise reduction processing based on the first subtitle sequence; the second subtitle sequence is analyzed based on a large language model to obtain a summary subtitle sequence for detecting the accuracy of information in the lock area; an initial anomaly score is obtained based on analysis of the summary subtitle sequence to indicate the degree of abnormality of the event; a final anomaly score is obtained based on analysis of the initial anomaly score; and an early warning signal is generated based on the final anomaly score. The present invention can work without any training by utilizing pre-trained visual language models and large language models, and utilizes the visual priors and semantic priors brought by pre-training to fundamentally solve the problems of data collection difficulties and poor network generalization of traditional training methods, thereby improving the accuracy of early warning of the lock area.
[0125] The above embodiments are only used to illustrate the technical method of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present application.
Claims
1. A lock area monitoring and early warning method based on untrained video anomaly detection, characterized in that: Acquire video data of the target lock area; Analyzing the video data to obtain a first subtitle sequence; Performing noise reduction processing on the first subtitle sequence to obtain a second subtitle sequence; The second subtitle sequence is analyzed based on a large language model to obtain a summary subtitle sequence for detecting the accuracy of the lock area information; Analyze the summary subtitle sequence to obtain an initial anomaly score indicating the degree of anomaly. The initial anomaly score is analyzed to obtain a final anomaly score that is used to improve the scoring accuracy, and an early warning signal is generated based on the final anomaly score.
2. The method for monitoring and early warning of ship lock areas based on untrained video anomaly detection according to claim 1 is characterized in that: The first subtitle sequence is obtained by analyzing the video data, including: The video data represents video data of monitoring the lock area; Dividing the video data into a plurality of video data sets by a video acquisition device; extracting a video frame from each video in the video data set; the video frame includes each image frame; Multiple video datasets, denoted as V = {V1,…,V j ,…,V N },V j Represents the jth video, 1≤j≤N, N represents the total number of videos, and extracts video frames from each video in the video dataset V, that is, Represents the i-th frame image of the j-th video, 1≤i≤M, M represents the total number of frames of the current video; A text description for describing the corresponding image content is obtained according to each image frame; and each text description is integrated into a subtitle sequence, which is recorded as a first subtitle sequence.
3. The method for monitoring and early warning of ship lock areas based on untrained video anomaly detection according to claim 2 is characterized in that: The method for obtaining the text description is: A corresponding text description is generated for each image frame by a caption generation model, wherein the caption generation model is obtained by pre-training a visual language model.
4. The method for monitoring and early warning of ship lock areas based on untrained video anomaly detection according to claim 1 is characterized in that: Performing noise reduction processing on the first subtitle sequence to obtain the second subtitle sequence includes: Text description of the first subtitle sequence Perform noise reduction processing to obtain The specific formula is as follows: Among them, ε I , ε T They are image and text feature encoders respectively, and <·,·> represents cosine similarity. After extracting the features of each image frame using the modal encoder ε(·), the features of the current image frame are converted to All image frame features {ε T (k)|k∈C j } to compare and take the text description with the highest similarity to the image frame feature as the text description after denoising The argmax function represents the independent variable value that makes the function reach the maximum value; the second subtitle sequence is obtained by performing the above calculation on all text descriptions in the first subtitle sequence.
5. The method for monitoring and early warning of ship lock areas based on untrained video anomaly detection according to claim 1 is characterized in that: The method for obtaining the summary subtitle sequence includes: The second subtitle sequence is input into the large language model and combined with the time window sampling setting to perform time domain summarization to obtain a summarized subtitle sequence.
6. The method for monitoring and early warning of ship lock areas based on untrained video anomaly detection according to claim 1 is characterized in that: The initial anomaly score for indicating the degree of abnormality of the event is obtained by analyzing the summary subtitle sequence, including: Obtain preset subtitle prompt words, including prompt words that provide contextual information and format requirements; input the summary text description and subtitle prompt words into the large language model to obtain an initial anomaly score.
7. The method for monitoring and early warning of ship lock areas based on untrained video anomaly detection according to claim 1 is characterized in that: The final anomaly score for improving the scoring accuracy is obtained by analyzing the initial anomaly score, including: For each video frame, the corresponding window video clip is compared with the corresponding features through the video encoder, and the corresponding features of the subtitle sequence are extracted and summarized using the text encoder. Then, a cross-modal comparison is performed to obtain the subtitles with the closest video features. The final anomaly score is calculated by weighted summing up the initial anomaly scores corresponding to the closest subtitles; The above formula is calculated for each image frame to obtain the final anomaly score.
8. The method for monitoring and early warning of ship lock areas based on untrained video anomaly detection according to claim 7 is characterized in that: The final anomaly score is obtained as follows: in, Indicates the final abnormality score of each; the number of subtitles closest to it is H; Indicates S j The index number set corresponding to each of the h subtitles, is the initial anomaly score corresponding to the image frame with index h in the jth video, <·,·> represents the cosine similarity; each video frame is recorded as The video encoder is denoted as ε V (·), the text encoder is denoted as ε T (·), the video encoder extracts the corresponding features and records them as The text encoder extracts the corresponding features of the summary subtitle sequence and records them as ε T (S j ).
9. The method for monitoring and early warning of ship lock areas based on untrained video anomaly detection according to claim 1, characterized in that: Generating an early warning signal based on the final anomaly score includes: Preset a safety threshold for the final anomaly score; compare the final anomaly score with its safety threshold; When the final anomaly score exceeds its corresponding safety threshold, an early warning signal is issued; When the final anomaly score does not exceed the corresponding safety threshold, a safety signal is issued.
10. A ship lock area monitoring and early warning system based on untrained video anomaly detection, applied to the ship lock area monitoring and early warning method based on untrained video anomaly detection according to any one of claims 1 to 9; characterized in that: include: Data acquisition module: acquires video data of the target lock area; Subtitle processing module: obtains a first subtitle sequence based on analysis of video data; Performing noise reduction processing on the first subtitle sequence to obtain a second subtitle sequence; The second subtitle sequence is analyzed based on a large language model to obtain a summary subtitle sequence for detecting the accuracy of the lock area information; Abnormal warning module: Analyzes the subtitle sequence to obtain an initial abnormality score to indicate the degree of abnormality; Analyze the initial anomaly score to obtain a final anomaly score that represents the improvement in scoring accuracy; Generate early warning signals based on the final anomaly score.