Intelligent potential risk event analysis method based on comparative learning

Abstract

The invention provides a potential risk event intelligent analysis method based on comparative learning, and belongs to the technical field of risk event intelligent analysis. A memory unit is regarded as a prototype of a normal video mode, and on the basis of memorizing video data features, contrast learning is introduced into the memory unit for training. A data enhancement strategy is adopted: different video sections in the same video belong to positive samples; data enhancement is performed on a video, and spatio-temporal information of the video, such as disorder, random rotation of video frames, continuous repetition of single video frames and the like, is destroyed as a negative sample sequence. Reconstruction errors are used as a standard for judging the drowning event, the similarity between an input video stream and a memory unit is used as an auxiliary standard for judging the drowning event, and the specific mixed decision-making method is completed in a weighting mode. According to the method, the general problem of network reconstruction and the problem of a memory model in anomaly detection are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent analysis of risk events, and in particular relates to an intelligent analysis method of potential risk events based on contrastive learning. Background Art

[0002] Potential risk events are abnormal events in videos, specifically sporadic events. Effective detection of abnormal events in videos is first of all a necessary function of the monitoring system. Accurate monitoring of abnormal events can also better promote the construction of safe cities and smart cities. In computer vision and video analysis, abnormal points or abnormal events in videos refer specifically to distribution points and video events that do not conform to most normal data in terms of probability distribution. Effective probability modeling of large-scale videos is an important issue that needs to be solved urgently and is conducive to video anomaly detection.

[0003] At present, there are a lot of studies on video anomaly detection, and scholars are committed to continuously improving the method of anomaly determination to improve detection efficiency. For systems deployed in reality, semi-supervised training is a relatively effective and mature solution, among which the reconstruction-based model is one of the widely used methods. Reconstruct and model the video, and input a large number of known normal video sequences into the model during the test phase. The reconstruction model will reconstruct and model the normal video sequence. If normal data is input into the model during the practical test phase, the model will reconstruct the video sequence well; if a video sequence with an abnormal event appears, the reconstruction error of the model will increase, so this can be used as a sign of the appearance of an abnormal video sequence. Because normal videos can be reconstructed better, if the reconstruction error is large, it is regarded as an abnormal event. Based on the memory network MemAE model, a memory module is introduced into the abnormal event detection network to memorize the knowledge of normal events seen in the training phase. If the event encountered in the test phase is a normal event with a normal pattern, it can fit well with the memory module, so that it can be better reconstructed. On the contrary, if the model encounters an abnormal event and the memory module does not have this pattern, the reconstruction error will be greatly increased, thereby detecting the occurrence of an abnormal event.

[0004] It can be seen that the problems of the memory reconstruction model for anomaly detection solutions include: (1) The memory module is a deterministic feature space and cannot well reflect the law of normal events in probability distribution; (2) The acquisition and iteration methods of memory unit knowledge are not mature enough. At present, the general attention mechanism method cannot achieve the optimal result. Summary of the invention

[0005] The present invention proposes an intelligent analysis scheme for potential risk events based on contrastive learning. It proposes a solution to the general problem of reconstructing the network and the problem of memory model in anomaly detection. The scheme is based on the introduction of contrastive learning method, combined with a novel hybrid decision-making mode, and on this basis proposes an abnormal event detection algorithm in the video, which provides a scheme for video abnormal event detection technology with high data utilization, low data labeling requirements, and is conducive to the deployment of real-life scenarios and good scalability. In particular, the feature of good scalability can better facilitate the long-term research and development of the project, algorithm iteration, and large-scale deployment of potential risk event scenarios.

[0006] The first aspect of the present invention provides a potential risk event intelligent analysis method based on contrastive learning, the method discovers specific incidental events in a video through video abnormal event detection, and the method comprises: Step S1, pre-training the detection model; Among them, the memory unit is used as the prototype of the normal video mode, and on the basis of memorizing the video data features, contrast learning is introduced into the memory unit for training; Step S2: using data enhancement strategy to enhance video data, including positive samples and negative samples; Among them, different video clips in the same video belong to the positive sample sequence; the temporal and spatial information of the video data is processed, including disorder, random rotation of video frames, and repetition of a single video frame, to obtain a negative sample sequence; Step S3, based on the enhancement result of step S2, using the trained detection model obtained in step S1, detecting drowning events in the video; In addition to using the reconstruction error as a criterion for determining a drowning event, the similarity between the input video stream and the memory unit is also used as a criterion for determining a drowning event.

[0007] According to the method of the first aspect of the present invention, in step S1: Step S1-1, define memory units and select contrastive learning tasks; wherein: Determine the structure and function of memory cells, which are components that store short-term or long-term information, including long short-term memory networks (LSTMs); determine contrastive learning tasks, where the detection model extracts effective features by comparing two or more samples; Step S1-2: construct training data for contrastive learning tasks; wherein: Select sample pairs from the data set, and select a positive sample and a negative sample for each sample pair, where the similarity between the positive sample and the target sample is higher than a first target threshold, and the difference between the negative sample and the target sample is higher than a second target threshold; Step S1-3, feature extraction and contrast loss calculation; wherein: Connect the input of the memory unit to the input of the contrastive learning task, extract features through the memory unit, and calculate the contrastive loss; Step S1-4, back propagation and optimization; wherein: Use contrastive loss for backpropagation and update the weights of the detection model according to the gradient, so that the detection model captures the similarities between positive samples and maintains the differences between negative samples; Step S1-5, repeat iteration and adjustment; Repeat the above steps until the detection model converges, and use the detection model to extract features and compare features; use the validation set to evaluate the performance of the detection model and perform hyperparameter adjustment and optimization.

[0008] According to the method of the first aspect of the present invention, in step S2, enhancing the video data includes: Shuffle: Randomly change the order of video frames to increase the robustness of the detection model to frame order; Random rotation of video frames: Randomly rotate each video frame to increase the model's adaptability to rotation changes; Repetition of a single video frame: Select a specific frame in the video and insert it repeatedly into different positions to increase the adaptability of the detection model to static scenes; Temporal truncation: randomly or regularly intercept a part of the time in the video to increase the adaptability of the detection model to videos of different lengths; Random speed adjustment: randomly adjust the video playback speed to increase the adaptability of the detection model to videos of different speeds; Color space transformation: Randomly adjust the color of video frames, including hue, saturation, and brightness; Random cropping: randomly cropping video frames in time and space to increase the detection model's ability to perceive local and temporal changes; Adding noise: Introducing random noise into the video to improve the robustness of the detection model in noisy environments; Object occlusion: simulate object occlusion in the video to increase the adaptability of the detection model to partial occlusion; Mix videos: Mix two or more videos to generate a new video.

[0009] According to the method of the first aspect of the present invention, in step S3, the similarity between the video stream and the memory unit is introduced as a criterion for determining a drowning event, including: Reconstruction error criteria; specifically, using an autoencoder for training and comparing the reconstruction error between input and output to detect drowning events; Calculating the similarity between the video stream and the memory unit; specifically, calculating the similarity between the current frame and the learned normal mode by combining the spatiotemporal information of the video frame and the normal mode learned by the detection model, wherein the similarity refers to the similarity between the feature representations of the video stream and the memory unit; Perform weighted hybrid decision-making; specifically, it includes: combining the reconstruction error, the similarity of the video stream and the memory unit, using a weighted method to make a hybrid decision, assigning a weight to each criterion, calculating the weighted sum to obtain the final anomaly score, and determining the drowning event based on the score.

[0010] A second aspect of the present invention provides a potential risk event intelligent analysis system based on contrastive learning, wherein the system detects specific incidental events in a video through video abnormal event detection, and the system includes a processing unit, wherein the processing unit is configured to perform the following steps: Step S1, pre-training the detection model; Among them, the memory unit is used as the prototype of the normal video mode, and on the basis of memorizing the video data features, contrast learning is introduced into the memory unit for training; Step S2: using data enhancement strategy to enhance video data, including positive samples and negative samples; Among them, different video clips in the same video belong to the positive sample sequence; the temporal and spatial information of the video data is processed, including disorder, random rotation of video frames, and repetition of a single video frame, to obtain a negative sample sequence; Step S3, based on the enhancement result of step S2, using the trained detection model obtained in step S1, detecting drowning events in the video; In addition to using the reconstruction error as a criterion for determining a drowning event, the similarity between the input video stream and the memory unit is also used as a criterion for determining a drowning event.

[0011] According to the system of the second aspect of the present invention, when the processing unit is configured to execute step S1, specifically the following steps are performed: Step S1-1, define memory units and select contrastive learning tasks; wherein: Determine the structure and function of memory cells, which are components that store short-term or long-term information, including long short-term memory networks (LSTMs); determine contrastive learning tasks, where the detection model extracts effective features by comparing two or more samples; Step S1-2: construct training data for contrastive learning tasks; wherein: Select sample pairs from the data set, and select a positive sample and a negative sample for each sample pair, where the similarity between the positive sample and the target sample is higher than a first target threshold, and the difference between the negative sample and the target sample is higher than a second target threshold; Step S1-3, feature extraction and contrast loss calculation; wherein: Connect the input of the memory unit to the input of the contrastive learning task, extract features through the memory unit, and calculate the contrastive loss; Step S1-4, back propagation and optimization; wherein: Use contrastive loss for backpropagation and update the weights of the detection model according to the gradient, so that the detection model captures the similarities between positive samples and maintains the differences between negative samples; Step S1-5, repeat iteration and adjustment; Repeat the above steps until the detection model converges, and use the detection model to extract features and compare features; use the validation set to evaluate the performance of the detection model and perform hyperparameter adjustment and optimization.

[0012] According to the system of the second aspect of the present invention, when the processing unit is configured to perform step S2, enhancing the video data specifically includes: Shuffle: Randomly change the order of video frames to increase the robustness of the detection model to frame order; Random rotation of video frames: Randomly rotate each video frame to increase the model's adaptability to rotation changes; Repetition of a single video frame: Select a specific frame in the video and insert it repeatedly into different positions to increase the adaptability of the detection model to static scenes; Temporal truncation: randomly or regularly intercept a part of the time in the video to increase the adaptability of the detection model to videos of different lengths; Random speed adjustment: randomly adjust the video playback speed to increase the adaptability of the detection model to videos of different speeds; Color space transformation: Randomly adjust the color of video frames, including hue, saturation, and brightness; Random cropping: randomly cropping video frames in time and space to increase the detection model's ability to perceive local and temporal changes; Adding noise: Introducing random noise into the video to improve the robustness of the detection model in noisy environments; Object occlusion: simulate object occlusion in the video to increase the adaptability of the detection model to partial occlusion; Mix videos: Mix two or more videos to generate a new video.

[0013] According to the system of the second aspect of the present invention, when the processing unit is configured to execute step S3, the similarity between the video stream and the memory unit is introduced as a criterion for determining a drowning event, including: Reconstruction error criteria; specifically, using an autoencoder for training and comparing the reconstruction error between input and output to detect drowning events; Calculating the similarity between the video stream and the memory unit; specifically, calculating the similarity between the current frame and the learned normal mode by combining the spatiotemporal information of the video frame and the normal mode learned by the detection model, wherein the similarity refers to the similarity between the feature representations of the video stream and the memory unit; Perform weighted hybrid decision-making; specifically, it includes: combining the reconstruction error, the similarity of the video stream and the memory unit, using a weighted method to make a hybrid decision, assigning a weight to each criterion, calculating the weighted sum to obtain the final anomaly score, and determining the drowning event based on the score.

[0014] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the potential risk event intelligent analysis method based on contrastive learning described in the first aspect of the present disclosure is implemented.

[0015] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for intelligently analyzing potential risk events based on contrastive learning described in the first aspect of the present disclosure is implemented.

[0016] In summary, the present invention introduces contrastive learning into the memory module training process and uses the similarity between the video stream and the memory module as an additional decision-making basis to assist in drowning judgment. It aims to develop a new potential risk event detection algorithm model which is innovative, can be actually deployed, has high evaluation accuracy, high execution efficiency, fast response speed and can be continuously iterated and optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 Flow chart of a method for intelligent analysis of potential risk events based on contrastive learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0020] The first aspect of the present invention proposes a potential risk event intelligent analysis method based on contrastive learning, such as Figure 1 As shown, the method detects specific incidental events in a video through abnormal video event detection, and the method includes: Step S1, pre-training the detection model; Among them, the memory unit is used as the prototype of the normal video mode, and on the basis of memorizing the video data features, contrast learning is introduced into the memory unit for training; Step S2: using data enhancement strategy to enhance video data, including positive samples and negative samples; Among them, different video clips in the same video belong to the positive sample sequence; the temporal and spatial information of the video data is processed, including disorder, random rotation of video frames, and repetition of a single video frame, to obtain a negative sample sequence; Step S3, based on the enhancement result of step S2, using the trained detection model obtained in step S1, detecting drowning events in the video; In addition to using the reconstruction error as a criterion for determining a drowning event, the similarity between the input video stream and the memory unit is also used as a criterion for determining a drowning event.

[0021] According to the method of the first aspect of the present invention, in step S1: Step S1-1, define memory units and select contrastive learning tasks; wherein: Determine the structure and function of memory cells, which are components that store short-term or long-term information, including long short-term memory networks (LSTMs); determine contrastive learning tasks, where the detection model extracts effective features by comparing two or more samples; Step S1-2: construct training data for contrastive learning tasks; wherein: Select sample pairs from the data set, and select a positive sample and a negative sample for each sample pair, where the similarity between the positive sample and the target sample is higher than a first target threshold, and the difference between the negative sample and the target sample is higher than a second target threshold; Step S1-3, feature extraction and contrast loss calculation; wherein: Connect the input of the memory unit to the input of the contrastive learning task, extract features through the memory unit, and calculate the contrastive loss; Step S1-4, back propagation and optimization; wherein: Use contrastive loss for backpropagation and update the weights of the detection model according to the gradient, so that the detection model captures the similarities between positive samples and maintains the differences between negative samples; Step S1-5, repeat iteration and adjustment; Repeat the above steps until the detection model converges, and use the detection model to extract features and compare features; use the validation set to evaluate the performance of the detection model and perform hyperparameter adjustment and optimization.

[0022] According to the method of the first aspect of the present invention, in step S2, enhancing the video data includes: Shuffle: Randomly change the order of video frames to increase the robustness of the detection model to frame order; Random rotation of video frames: Randomly rotate each video frame to increase the model's adaptability to rotation changes; Repetition of a single video frame: Select a specific frame in the video and insert it repeatedly into different positions to increase the adaptability of the detection model to static scenes; Temporal truncation: randomly or regularly intercept a part of the time in the video to increase the adaptability of the detection model to videos of different lengths; Random speed adjustment: randomly adjust the video playback speed to increase the adaptability of the detection model to videos of different speeds; Color space transformation: Randomly adjust the color of video frames, including hue, saturation, and brightness; Random cropping: randomly cropping video frames in time and space to increase the detection model's ability to perceive local and temporal changes; Adding noise: Introducing random noise into the video to improve the robustness of the detection model in noisy environments; Object occlusion: simulate object occlusion in the video to increase the adaptability of the detection model to partial occlusion; Mix videos: Mix two or more videos to generate a new video.

[0023] According to the method of the first aspect of the present invention, in step S3, the similarity between the video stream and the memory unit is introduced as a criterion for determining a drowning event, including: Reconstruction error criteria; specifically, using an autoencoder for training and comparing the reconstruction error between input and output to detect drowning events; Calculating the similarity between the video stream and the memory unit; specifically, calculating the similarity between the current frame and the learned normal mode by combining the spatiotemporal information of the video frame and the normal mode learned by the detection model, wherein the similarity refers to the similarity between the feature representations of the video stream and the memory unit; Perform weighted hybrid decision-making; specifically, it includes: combining the reconstruction error, the similarity of the video stream and the memory unit, using a weighted method to make a hybrid decision, assigning a weight to each criterion, calculating the weighted sum to obtain the final anomaly score, and determining the drowning event based on the score.

[0024] First embodiment

[0025] This embodiment uses a video anomaly event detection system to timely discover specific incidental events in surveillance videos, i.e., potential risk events. The algorithm mainly consists of two parts: video anomaly detection and self-supervised contrastive learning. Video anomaly detection mainly uses semi-supervised training to realize drowning event video detection. It selects a widely used method based on reconstruction model, memorizes the normal video stream in the training stage, extracts the "normal video" mode, and models the probability prototype, so as to better detect drowning events. Contrastive learning achieves the goal of self-supervised learning by setting up instance discrimination proxy tasks. The image pairs used in the training process are image pairs obtained by respectively augmenting the same image (positive sample pairs), or image pairs formed by respectively augmenting different images (negative sample pairs). Through a large number of comparison calculations between positive and negative sample pairs, the neural network model can automatically extract better feature expressions from the data.

[0026] 1. First, introduce contrastive learning into the memory unit for training.

[0027] First of all, the memory module is a unit that reflects the probability distribution and pattern rules of normal data, so the learning and iteration of the memory module is extremely important. Previous studies have only simply accumulated and memorized data features, which cannot fully reflect the pattern rules of data features. Therefore, it is proposed to use an additional loss function to train the memory module. The memory unit is regarded as a prototype of the normal video mode. These prototypes should be able to reflect the rules of the normal video mode. Therefore, on the basis of memorizing the video data features, contrastive learning is introduced to train the memory unit.

[0028] 2. Secondly, data enhancement is performed.

[0029] First, in contrastive learning, data augmentation is an important module. It is related to the selection of positive and negative samples in the contrastive learning loss function. Adopting such a data augmentation strategy, different video clips in the same video are positive samples.

[0030] Data enhancement is performed on the video to destroy the temporal and spatial information of the video, such as disorder, random rotation of video frames, and continuous repetition of a single video frame, as a negative sample sequence. Through such a comparative learning strategy, the contextual relationship of the video can be better modeled (because different video clips of the same video are regarded as positive samples), and the model is more sensitive to video sequences with abnormal temporal and spatial relationships, thereby better predicting drowning events.

[0031] 3. Finally, make a drowning incident determination.

[0032] For drowning event judgment, reconstruction-based methods usually directly use reconstruction error as the criterion for drowning event judgment. This scheme is still one of the reconstruction-based methods and adopts the basic criterion based on reconstruction error. Since the memory module of the normal mode is also effectively learned and modeled, a hybrid decision-making method is adopted to determine whether the video contains a drowning event. The similarity between the video stream and the memory module is used as an additional decision basis to assist in the drowning event judgment.

[0033] Second embodiment

[0034] This embodiment involves three aspects of research: (1) First, the memory unit is regarded as the prototype of the normal video mode. On the basis of memorizing the video data features, we introduce contrast learning to the memory unit for training; (2) Adopt the following data enhancement strategy: different video clips in the same video belong to positive samples; perform data enhancement on the video, destroy the temporal and spatial information of the video, such as disorder, random rotation of video frames, continuous repetition of a single video frame, etc., as a negative sample sequence; (3) Finally, in addition to using the reconstruction error as the criterion for determining drowning events, the similarity between the input video stream and the memory unit is also used as an auxiliary criterion for determining drowning events. The specific hybrid decision method is completed using a weighted approach. A semi-supervised potential risk event detection scheme based on a reconstructed memory network is obtained. The specific techniques are as follows: 1. Introduce contrastive learning into memory module training.

[0035] Contrastive learning originates from self-supervised learning, and achieves the goal of self-supervised learning by setting up instance discrimination proxy tasks. First, different image enhancements are performed on the same image, and then the similarity between the features of the obtained image pairs is measured, so that the similarity between the features of the image pairs after the same image enhancement increases, while the similarity between the features of different images decreases. Contrastive learning can be introduced into the training process of memory units to improve the performance of the model. Memory units are usually used for long-term memory and information storage of the model. The specific implementation method is: (1) Define memory units and select contrastive learning tasks. First, determine the structure and function of a memory unit, which can be a component that stores short-term or long-term information, such as an LSTM (Long Short-Term Memory) network or other similar structures. Determine contrastive learning tasks, in which the model needs to learn to extract meaningful features by comparing two or more samples.

[0036] (2) Constructing training data. Constructing training data for the contrastive learning task involves selecting sample pairs from the dataset and selecting a positive sample and a negative sample for each pair. The positive sample should be similar to the target sample, while the negative sample should have sufficient differences.

[0037] (3) Feature extraction and contrastive loss calculation. The input of the memory unit is connected to the input of the contrastive learning task, features are extracted through the memory unit, and then the contrastive loss is calculated. The contrastive loss is usually defined by measuring the similarity between two samples, such as cosine similarity or Euclidean distance.

[0038] (4) Back propagation and optimization. Back propagation is performed using contrastive loss and the model weights are updated according to the gradient. This step is to enable the model to better capture the similarities between positive samples and maintain sufficient differences between negative samples.

[0039] (5) Repeat iteration and adjustment. Repeat the above steps until the model converges and can accurately extract and compare features. Use the validation set to evaluate the performance of the model and perform hyperparameter adjustment or other optimization as needed.

[0040] 2. Adopt data enhancement strategies to model video contextual relationships.

[0041] Video data enhancement is a method to increase data diversity and richness by applying various transformations and processing techniques on video datasets. Similar to static image data enhancement, video data enhancement aims to improve the generalization ability of the model so that it can better adapt to different scenarios and changes. The following methods can be used: (1) Shuffle. Randomly change the order of video frames to increase the model’s robustness to frame order.

[0042] (2) Random rotation of video frames. Each video frame is randomly rotated to increase the model’s adaptability to rotation changes.

[0043] (3) Continuous repetition of a single video frame. A specific frame is selected in the video and then repeatedly inserted into different positions to increase the adaptability of the model to static scenes.

[0044] (4) Temporal truncation: Randomly or regularly truncate a portion of the video to increase the model’s adaptability to videos of different lengths.

[0045] (5) Random speed adjustment: Randomly adjust the video playback speed to increase adaptability to videos of different speeds.

[0046] (6) Color gamut transformation: Randomly adjust the color of video frames, including hue, saturation, brightness, etc.

[0047] (7) Random cropping. Randomly crop video frames in time and space to increase the model’s ability to perceive local and temporal changes.

[0048] (8) Adding noise: Introducing random noise into the video to improve the robustness of the model in noisy environments.

[0049] (9) Object occlusion: Simulate object occlusion in the video to increase the model’s adaptability to partial occlusion.

[0050] (10) Video Mixing: Mix two or more videos together to generate a new video.

[0051] These technologies can be used alone or in combination, and the specific choice of technology depends on the characteristics of the data. By applying these technologies, the model can be better learned and understood in different scenes, and the robustness and generalization ability of the model can be improved, so that it can better adapt to different video scenes.

[0052] 3. Drowning incidents are judged by introducing the similarity between video stream and memory unit as an auxiliary criterion.

[0053] Through effective learning of the memory module, the model can learn a memory pattern of the normal video data stream. In the test deployment phase, in addition to using the reconstruction error as the criterion for determining drowning events, the similarity between the input video stream and the memory unit is also used as an auxiliary criterion for determining drowning events. The specific hybrid decision method is completed using a weighted approach. The specific steps for implementation are as follows: (1) Reconstruction error criterion: Use an autoencoder or similar model for training and compare the reconstruction error between input and output to detect drowning events.

[0054] (2) Similarity between video stream and memory unit: Combining the spatiotemporal information of the video frame and the normal pattern learned by the model, the similarity between the current frame and the normal pattern learned in the past is calculated. This can be achieved by considering the similarity between the feature representation of the video stream (optical flow, etc.) and the memory unit.

[0055] (3) Weighted hybrid decision: Combine the reconstruction error, the similarity between the video stream and the memory unit, and use a weighted approach to make hybrid decisions. Each criterion can be assigned a weight, and the final anomaly score is obtained by calculating the weighted sum.

[0056] In some embodiments, in an actual video surveillance system, a large amount of normal surveillance data will be continuously collected by the system. This solution can make the most of this normal surveillance data, and use the modeling (reconstruction) of the normal surveillance data to alarm the video stream when a drowning incident occurs. Therefore, this solution has the advantages of high data utilization, extremely low annotation cost, good performance and high efficiency in actual deployment.

[0057] In some embodiments, the method of detecting drowning events in videos based on reconstruction can effectively mine the spatiotemporal information of drowning events. Through the reconstruction of video frames, combined with auxiliary information such as spatiotemporal context, the time node and spatial location of drowning events can be effectively located using advanced video feature extraction models. Moreover, the model of this solution does not require any drowning event annotation information during the training stage, and can achieve spatiotemporal location of drowning events, which has the advantages of low cost and good effect.

[0058] In some embodiments, the abnormal data enhancement in the model training process proposed in the scheme is currently formed based on the mechanism of the occurrence of general drowning events, based on the disturbance of the spatiotemporal information of normal data, to generate an enhanced version of the drowning event data, to assist in the generation and training of the memory module. In actual deployment, if the actual situation requires the detection of predefined specific drowning events, only using the disturbed version of normal data often cannot cover the specific drowning events, so specific drowning event samples can be generated in the data enhancement process of the drowning event samples to assist in the training of the memory module for the detection of specific drowning event scenarios.

[0059] A second aspect of the present invention provides a potential risk event intelligent analysis system based on contrastive learning, wherein the system detects specific incidental events in a video through video abnormal event detection, and the system includes a processing unit, wherein the processing unit is configured to perform the following steps: Step S1, pre-training the detection model; wherein, the memory unit is used as a prototype of a normal video mode, and on the basis of memorizing the video data features, contrast learning is introduced into the memory unit for training; Step S2: using data enhancement strategy to enhance video data, including positive samples and negative samples; Among them, different video clips in the same video belong to the positive sample sequence; the temporal and spatial information of the video data is processed, including disorder, random rotation of video frames, and repetition of a single video frame, to obtain a negative sample sequence; Step S3, based on the enhancement result of step S2, using the trained detection model obtained in step S1, detecting drowning events in the video; In addition to using the reconstruction error as a criterion for determining a drowning event, the similarity between the input video stream and the memory unit is also used as a criterion for determining a drowning event.

[0060] According to the system of the second aspect of the present invention, when the processing unit is configured to execute step S1, specifically the following steps are performed: Step S1-1, define memory units and select contrastive learning tasks; wherein: Determine the structure and function of memory cells, which are components that store short-term or long-term information, including long short-term memory networks (LSTMs); determine contrastive learning tasks, where the detection model extracts effective features by comparing two or more samples; Step S1-2: construct training data for contrastive learning tasks; wherein: Select sample pairs from the data set, and select a positive sample and a negative sample for each sample pair, where the similarity between the positive sample and the target sample is higher than a first target threshold, and the difference between the negative sample and the target sample is higher than a second target threshold; Step S1-3, feature extraction and contrast loss calculation; wherein: Connect the input of the memory unit to the input of the contrastive learning task, extract features through the memory unit, and calculate the contrastive loss; Step S1-4, back propagation and optimization; wherein: Use contrastive loss for backpropagation and update the weights of the detection model according to the gradient, so that the detection model captures the similarities between positive samples and maintains the differences between negative samples; Step S1-5, repeat iteration and adjustment; Repeat the above steps until the detection model converges, and use the detection model to extract features and compare features; use the validation set to evaluate the performance of the detection model and perform hyperparameter adjustment and optimization.

[0061] According to the system of the second aspect of the present invention, when the processing unit is configured to perform step S2, enhancing the video data specifically includes: Shuffle: Randomly change the order of video frames to increase the robustness of the detection model to frame order; Random rotation of video frames: Randomly rotate each video frame to increase the model's adaptability to rotation changes; Repetition of a single video frame: Select a specific frame in the video and insert it repeatedly into different positions to increase the adaptability of the detection model to static scenes; Temporal truncation: randomly or regularly intercept a part of the time in the video to increase the adaptability of the detection model to videos of different lengths; Random speed adjustment: randomly adjust the video playback speed to increase the adaptability of the detection model to videos of different speeds; Color space transformation: Randomly adjust the color of video frames, including hue, saturation, and brightness; Random cropping: randomly cropping video frames in time and space to increase the detection model's ability to perceive local and temporal changes; Adding noise: Introducing random noise into the video to improve the robustness of the detection model in noisy environments; Object occlusion: simulate object occlusion in the video to increase the adaptability of the detection model to partial occlusion; Mix videos: Mix two or more videos to generate a new video.

[0062] According to the system of the second aspect of the present invention, when the processing unit is configured to execute step S3, the similarity between the video stream and the memory unit is introduced as a criterion for determining a drowning event, including: Reconstruction error criteria; specifically, using an autoencoder for training and comparing the reconstruction error between input and output to detect drowning events; Calculating the similarity between the video stream and the memory unit; specifically, calculating the similarity between the current frame and the learned normal mode by combining the spatiotemporal information of the video frame and the normal mode learned by the detection model, wherein the similarity refers to the similarity between the feature representations of the video stream and the memory unit; Perform weighted hybrid decision-making; specifically, it includes: combining the reconstruction error, the similarity of the video stream and the memory unit, using a weighted method to make a hybrid decision, assigning a weight to each criterion, calculating the weighted sum to obtain the final anomaly score, and determining the drowning event based on the score.

[0063] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the potential risk event intelligent analysis method based on contrastive learning described in the first aspect of the present disclosure is implemented.

[0064] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for intelligently analyzing potential risk events based on contrastive learning described in the first aspect of the present disclosure is implemented.

[0065] In summary, the present invention introduces contrastive learning into the memory module training process and uses the similarity between the video stream and the memory module as an additional decision-making basis to assist in drowning judgment. It aims to develop a new potential risk event detection algorithm model which is innovative, can be actually deployed, has high evaluation accuracy, high execution efficiency, fast response speed and can be continuously iterated and optimized.

[0066] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above-mentioned embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. A potential risk event intelligent analysis method based on contrastive learning, characterized in that: The method discovers specific incidental events in a video through video abnormal event detection, and the method comprises: Step S1, pre-training the detection model; Among them, the memory unit is used as the prototype of the normal video mode, and on the basis of memorizing the video data features, contrast learning is introduced into the memory unit for training; Step S2: using data enhancement strategy to enhance video data, including positive samples and negative samples; Among them, different video clips in the same video belong to the positive sample sequence; the temporal and spatial information of the video data is processed, including disorder, random rotation of video frames, and repetition of a single video frame, to obtain a negative sample sequence; Step S3, based on the enhancement result of step S2, using the trained detection model obtained in step S1, detecting drowning events in the video; In addition to using the reconstruction error as a criterion for determining a drowning event, the similarity between the input video stream and the memory unit is also used as a criterion for determining a drowning event.

2. According to claim 1, a potential risk event intelligent analysis method based on contrastive learning is characterized in that: In step S1: Step S1-1, define a memory unit and select a contrastive learning task; wherein: Determine the structure and function of memory cells, which are components that store short-term or long-term information, including long short-term memory networks (LSTMs); determine contrastive learning tasks, where the detection model extracts effective features by comparing two or more samples; Step S1-2: construct training data for contrastive learning tasks; wherein: Select sample pairs from the data set, and select a positive sample and a negative sample for each sample pair, where the similarity between the positive sample and the target sample is higher than a first target threshold, and the difference between the negative sample and the target sample is higher than a second target threshold; Step S1-3, feature extraction and contrast loss calculation; wherein: Connect the input of the memory unit to the input of the contrastive learning task, extract features through the memory unit, and calculate the contrastive loss; Step S1-4, back propagation and optimization; wherein: Use contrastive loss for backpropagation and update the weights of the detection model according to the gradient, so that the detection model captures the similarities between positive samples and maintains the differences between negative samples; Step S1-5, repeat iteration and adjustment; Repeat the above steps until the detection model converges, and use the detection model to extract features and compare features; use the validation set to evaluate the performance of the detection model and perform hyperparameter adjustment and optimization.

3. The method for intelligent analysis of potential risk events based on contrastive learning according to claim 2 is characterized in that: In step S2, enhancing the video data includes: Shuffle: Randomly change the order of video frames to increase the robustness of the detection model to frame order; Random rotation of video frames: Randomly rotate each video frame to increase the model's adaptability to rotation changes; Repetition of a single video frame: Select a specific frame in the video and insert it repeatedly into different positions to increase the adaptability of the detection model to static scenes; Temporal truncation: randomly or regularly intercept a part of the time in the video to increase the adaptability of the detection model to videos of different lengths; Random speed adjustment: randomly adjust the video playback speed to increase the adaptability of the detection model to videos of different speeds; Color space transformation: Randomly adjust the color of video frames, including hue, saturation, and brightness; Random cropping: randomly cropping video frames in time and space to increase the detection model's ability to perceive local and temporal changes; Adding noise: Introducing random noise into the video to improve the robustness of the detection model in noisy environments; Object occlusion: simulate object occlusion in the video to increase the adaptability of the detection model to partial occlusion; Mix videos: Mix two or more videos to generate a new video.

4. The method for intelligent analysis of potential risk events based on contrastive learning according to claim 3 is characterized in that: In step S3, the similarity between the video stream and the memory unit is introduced as a criterion for determining a drowning event, including: Reconstruction error criteria; specifically, using an autoencoder for training and comparing the reconstruction error between input and output to detect drowning events; Calculating the similarity between the video stream and the memory unit; specifically, calculating the similarity between the current frame and the learned normal mode by combining the spatiotemporal information of the video frame and the normal mode learned by the detection model, wherein the similarity refers to the similarity between the feature representations of the video stream and the memory unit; Perform weighted hybrid decision-making; specifically, it includes: combining the reconstruction error, the similarity of the video stream and the memory unit, using a weighted method to make a hybrid decision, assigning a weight to each criterion, calculating the weighted sum to obtain the final anomaly score, and determining the drowning event based on the score.

5. A potential risk event intelligent analysis system based on contrastive learning, characterized in that: The system detects specific incidental events in a video through video abnormal event detection, and the system includes a processing unit, which is configured to perform the following steps: Step S1, pre-training the detection model; Among them, the memory unit is used as the prototype of the normal video mode, and on the basis of memorizing the video data features, contrast learning is introduced into the memory unit for training; Step S2: using data enhancement strategy to enhance video data, including positive samples and negative samples; Among them, different video clips in the same video belong to the positive sample sequence; the temporal and spatial information of the video data is processed, including disorder, random rotation of video frames, and repetition of a single video frame, to obtain a negative sample sequence; Step S3, based on the enhancement result of step S2, using the trained detection model obtained in step S1, detecting drowning events in the video; In addition to using the reconstruction error as a criterion for determining a drowning event, the similarity between the input video stream and the memory unit is also used as a criterion for determining a drowning event.

6. The potential risk event intelligent analysis system based on contrastive learning according to claim 5 is characterized in that: When the processing unit is configured to perform step S1, the processing unit specifically performs the following steps: Step S1-1, define a memory unit and select a contrastive learning task; wherein: Determine the structure and function of memory cells, which are components that store short-term or long-term information, including long short-term memory networks (LSTMs); determine contrastive learning tasks, where the detection model extracts effective features by comparing two or more samples; Step S1-2: construct training data for contrastive learning tasks; wherein: Select sample pairs from the data set, and select a positive sample and a negative sample for each sample pair, where the similarity between the positive sample and the target sample is higher than a first target threshold, and the difference between the negative sample and the target sample is higher than a second target threshold; Step S1-3, feature extraction and contrast loss calculation; wherein: Connect the input of the memory unit to the input of the contrastive learning task, extract features through the memory unit, and calculate the contrastive loss; Step S1-4, back propagation and optimization; wherein: Use contrastive loss for backpropagation and update the weights of the detection model according to the gradient, so that the detection model captures the similarities between positive samples and maintains the differences between negative samples; Step S1-5, repeat iteration and adjustment; Repeat the above steps until the detection model converges, and use the detection model to extract features and compare features; use the validation set to evaluate the performance of the detection model and perform hyperparameter adjustment and optimization.

7. The potential risk event intelligent analysis system based on contrastive learning according to claim 6 is characterized in that: When the processing unit is configured to perform step S2, enhancing the video data specifically includes: Shuffle: Randomly change the order of video frames to increase the robustness of the detection model to frame order; Random rotation of video frames: Randomly rotate each video frame to increase the model's adaptability to rotation changes; Repetition of a single video frame: Select a specific frame in the video and insert it repeatedly into different positions to increase the adaptability of the detection model to static scenes; Temporal truncation: randomly or regularly intercept a part of the time in the video to increase the adaptability of the detection model to videos of different lengths; Random speed adjustment: randomly adjust the video playback speed to increase the adaptability of the detection model to videos of different speeds; Color space transformation: Randomly adjust the color of video frames, including hue, saturation, and brightness; Random cropping: randomly cropping video frames in time and space to increase the detection model's ability to perceive local and temporal changes; Adding noise: Introducing random noise into the video to improve the robustness of the detection model in noisy environments; Object occlusion: simulate object occlusion in the video to increase the adaptability of the detection model to partial occlusion; Mix videos: Mix two or more videos to generate a new video.

8. The potential risk event intelligent analysis system based on contrastive learning according to claim 7 is characterized in that: When the processing unit is configured to execute step S3, the similarity between the video stream and the memory unit is introduced as a criterion for determining a drowning event, including: Reconstruction error criteria; specifically, using an autoencoder for training and comparing the reconstruction error between input and output to detect drowning events; Calculating the similarity between the video stream and the memory unit; specifically, calculating the similarity between the current frame and the learned normal mode by combining the spatiotemporal information of the video frame and the normal mode learned by the detection model, wherein the similarity refers to the similarity between the feature representations of the video stream and the memory unit; Perform weighted hybrid decision-making; specifically, it includes: combining the reconstruction error, the similarity of the video stream and the memory unit, using a weighted method to make a hybrid decision, assigning a weight to each criterion, calculating the weighted sum to obtain the final anomaly score, and determining the drowning event based on the score.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the intelligent analysis method for potential risk events based on contrastive learning as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the method for intelligent analysis of potential risk events based on contrastive learning as described in any one of claims 1 to 4 is implemented.

Images