Video anomaly detection method and device, equipment and storage medium
By segmenting medical videos into video clips and video abnormality detection methods based on feature comparison and tag training, the problem of low accuracy of abnormal behavior detection in medical videos is solved, and efficient abnormal object behavior recognition is achieved.
Patent Information
- Application Number
- CN202411095360.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-22
AI Technical Summary
In medical videos, existing neural network models are difficult to effectively detect the behavior of multiple abnormal objects due to the small amount of training data and limited annotation samples, resulting in low detection accuracy.
A video anomaly detection method is proposed. By segmenting medical videos into video clips, based on the comparison of the characteristics of the first and second video clips and the training of predicted video tags, a combined model of convolutional layer, pooling layer and prediction layer is adopted. Perform feature extraction and anomaly detection to improve the model's feature extraction and classification prediction capabilities.
The accuracy of detection of abnormal behavior of objects in medical videos is improved. Through model training at feature level and classification prediction level, the ability to recognize abnormal object behavior of medical videos is enhanced and the false positive rate is reduced.
Smart Images

Figure CN119007079A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of digital medicine, and in particular to a video anomaly detection method and apparatus, device and storage medium. Background Art
[0002] At present, in medical scenarios, it is often necessary to detect the behavior of various objects in different time periods and different areas. In this regard, related technologies often detect abnormal behavior of objects in medical videos based on neural network models. However, abnormal object behaviors in medical videos are often diverse (for example, abnormal object behaviors include abnormal operation of equipment, sudden cramps of objects, abnormal behaviors in operating rooms, etc.), and the amount of training data in medical scenarios is relatively small. The neural network model trained by limited labeled samples is often difficult to fully detect these abnormal object behaviors, resulting in low accuracy in detecting abnormal behavior of objects in medical videos. Summary of the invention
[0003] The main purpose of the embodiments of the present application is to propose a video anomaly detection method and apparatus, a device and a storage medium, aiming to improve the accuracy of detecting abnormal behavior of objects in medical videos.
[0004] To achieve the above object, a first aspect of an embodiment of the present application provides a video anomaly detection method, the method comprising:
[0005] Acquire a target medical video, where the target medical video is used to indicate an object behavior of a target object in a target medical scene;
[0006] Performing scaling processing on the target medical video to obtain a scaled medical video;
[0007] Inputting the zoomed medical video into a video anomaly detection model for anomaly detection to obtain an anomaly detection result of the target medical video, wherein the anomaly detection result is used to indicate that the target object in the target medical video has an abnormal object behavior or indicates that the target object in the target medical video does not have an abnormal object behavior;
[0008] Among them, the video anomaly detection model is obtained by dividing the sample medical video into sample video clips, and training a preset first model based on the comparison of the first video clip features of the sample video clips with the second video clip features, and the comparison of the predicted video labels of the sample video clips with the sample video labels; the first video clip features are obtained by inputting the sample video clips into the first model for feature extraction; the predicted video labels are predicted based on the first video clip features; the second video clip features are obtained by inputting the sample video clips into the preset second model for feature extraction.
[0009] In some embodiments, the video anomaly detection model includes a first convolutional layer, a maximum pooling layer, a second convolutional layer, an average pooling layer, and a prediction layer;
[0010] The step of inputting the zoomed medical video into a video anomaly detection model for anomaly detection to obtain an anomaly detection result of the target medical video includes:
[0011] Extracting a first feature of the scaled medical video based on the first convolutional layer to obtain a first medical video convolutional feature;
[0012] Performing maximum pooling processing on the first medical video convolution feature based on the maximum pooling layer to obtain a first medical video pooling feature;
[0013] Performing a second feature extraction on the first medical video pooling feature based on the second convolutional layer to obtain a second medical video convolutional feature;
[0014] Performing average pooling processing on the second medical video convolution feature based on the average pooling layer to obtain a second medical video pooling feature;
[0015] Based on the prediction layer, feature dimension reduction is performed on the second medical video pooling feature to obtain a reduced dimension medical video feature, and anomaly detection is performed based on the reduced dimension medical video feature to obtain the anomaly detection result.
[0016] In some embodiments, the video anomaly detection model is trained in the following manner:
[0017] Obtaining a sample medical video and a reference video label of the sample medical video;
[0018] Segmenting the sample medical video to obtain a plurality of sample video segments, and determining the reference video label as the sample video label of the sample video segment;
[0019] For each of the sample video clips, scaling the sample video clip, and inputting the scaled sample video clip into the first model for feature extraction to obtain the first video clip feature of the sample video clip, and performing anomaly detection based on the first video clip feature to obtain a predicted video label of the sample video clip;
[0020] Inputting the sample video segment into the second model for feature extraction to obtain the second video segment features of the sample video segment;
[0021] Based on the comparison of the first video segment features and the second video segment features of a plurality of the sample video segments, and the comparison of the predicted video labels and the sample video labels, the first model is trained to obtain the video anomaly detection model.
[0022] In some embodiments, the second model includes a third convolutional layer, a pooling layer, and a fourth convolutional layer;
[0023] The step of inputting the sample video segment into the second model for feature extraction to obtain the second video segment features of the sample video segment includes:
[0024] Performing a first convolution process on the sample video segment based on the third convolution layer to obtain a sample video convolution feature;
[0025] Based on the pooling layer, a maximum pooling process is performed on the sample video convolution feature to obtain a sample video pooling feature;
[0026] Based on the fourth convolution layer, a second convolution process is performed on the sample video pooling feature to obtain the second video clip feature.
[0027] In some embodiments, the step of training the first model based on the comparison of the first video segment features and the second video segment features of the plurality of sample video segments and the comparison of the predicted video labels and the sample video labels to obtain the video anomaly detection model includes:
[0028] Determining a first loss function based on a comparison of the first video segment features and the second video segment features of a plurality of the sample video segments;
[0029] Determining a second loss function based on a comparison of the predicted video labels and the sample video labels of the plurality of sample video clips;
[0030] Adding the first loss function and the second loss function to obtain a total loss function;
[0031] Based on the total loss function, the first model is trained to obtain the video anomaly detection model.
[0032] In some embodiments, the first video segment feature includes a first abnormal probability feature of each video element in the sample video segment, and the second video segment feature includes a second abnormal probability feature of each video element in the sample video segment;
[0033] The determining of a first loss function based on a comparison of the first video segment feature and the second video segment feature of the plurality of sample video segments comprises:
[0034] For each of the sample video clips, performing divergence calculation on the first video clip feature and the second video clip feature to obtain a first sub-loss function of the sample video clip;
[0035] For each of the sample video clips, taking the logarithm of the quotient of the first abnormal probability feature and the second abnormal probability feature of each of the video elements to obtain a logarithmic result, multiplying the logarithmic result and the first abnormal probability feature to obtain a weighted abnormal probability feature, and adding the weighted abnormal probability features of multiple video elements to obtain a second sub-loss function of the sample video clip;
[0036] Performing weighted calculation on the first sub-loss function and the second sub-loss function to obtain a weighted loss function;
[0037] The weighted loss functions of the plurality of sample video clips are integrated to obtain the first loss function.
[0038] In some embodiments, the sample medical video is generated by:
[0039] Get original medical videos;
[0040] Scaling the original medical video to obtain an intermediate medical video;
[0041] The intermediate medical video is subjected to noise reduction processing to obtain the sample medical video.
[0042] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a video anomaly detection device, the device comprising:
[0043] an acquisition unit, configured to acquire a target medical video, wherein the target medical video is used to indicate an object behavior of a target object in a target medical scene;
[0044] A scaling unit, used for scaling the target medical video to obtain a scaled medical video;
[0045] a detection unit, configured to input the zoomed medical video into a video anomaly detection model for anomaly detection, and obtain an anomaly detection result of the target medical video, wherein the anomaly detection result is used to indicate that the target object in the target medical video has an abnormal object behavior or indicates that the target object in the target medical video does not have an abnormal object behavior;
[0046] Among them, the video anomaly detection model is obtained by dividing the sample medical video into sample video clips, and training a preset first model based on the comparison of the first video clip features of the sample video clips with the second video clip features, and the comparison of the predicted video labels of the sample video clips with the sample video labels; the first video clip features are obtained by inputting the sample video clips into the first model for feature extraction; the predicted video labels are predicted based on the first video clip features; the second video clip features are obtained by inputting the sample video clips into the preset second model for feature extraction.
[0047] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the video anomaly detection method described in the first aspect is implemented.
[0048] To achieve the above objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the video anomaly detection method described in the first aspect is implemented.
[0049] The video anomaly detection method and device, electronic device and storage medium proposed in the present application obtain a target medical video, which is used to indicate the object behavior of the target object in the target medical scene; scale the target medical video to obtain a scaled medical video; input the scaled medical video into a video anomaly detection model for anomaly detection to obtain an anomaly detection result of the target medical video, and can use the video anomaly detection model to identify the abnormal object behavior features in the low-dimensional scaled medical video, and can improve the detection accuracy of abnormal behavior of objects in medical videos. Among them, the video anomaly detection model is obtained by dividing the sample medical video into sample video clips, and training a preset first model based on the comparison of the first video clip feature of the sample video clip with the second video clip feature, and the comparison of the predicted video label of the sample video clip with the sample video label. The first video clip feature is obtained by inputting the sample video clip into the first model for feature extraction; the predicted video label is predicted based on the first video clip feature; the second video clip feature is obtained by inputting the sample video clip into the preset second model for feature extraction. The embodiment of the present application adopts a method combining feature comparison and label comparison during model training, which can realize model training at the feature level and model training at the classification prediction level, and is beneficial to training the first model's video feature information analysis capabilities in the feature extraction stage and the anomaly detection stage, thereby improving the model training effect and improving the detection accuracy of the trained video anomaly detection model for abnormal object behaviors in medical videos. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 is a flow chart of a video anomaly detection method provided by an embodiment of the present application;
[0051] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.
[0052] Figure 3 is a flow chart of a training video anomaly detection model provided in an embodiment of the present application;
[0053] Figure 4 yes Figure 3 Flow chart of step S304 in FIG.
[0054] Figure 5 yes Figure 3 Flow chart of step S305 in FIG.
[0055] Figure 6 yes Figure 5 Flow chart of step S501 in FIG.
[0056] Figure 7 Schematic diagram of the implementation process of the training video anomaly detection model provided in the embodiment of the present application;
[0057] Figure 8 is a structural schematic diagram of a video anomaly detection device provided in an embodiment of the present application;
[0058] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0060] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0062] The video anomaly detection method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the video anomaly detection method in the embodiments of the present application is described.
[0063] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0064] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0065] The video anomaly detection method provided in the embodiment of the present application relates to the field of digital medical technology. The video anomaly detection method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the video anomaly detection method, etc., but is not limited to the above forms.
[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0067] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the identity or characteristics of the object, such as object information, object behavior data, object historical data, and object location information, the permission or consent of the object will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the personal information of the object, the separate permission or separate consent of the object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the object, the necessary object-related data for the normal operation of the embodiment of the present application will be obtained.
[0068] Figure 1 is an optional flowchart of the video anomaly detection method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S103.
[0069] Step S101, obtaining a target medical video;
[0070] Step S102, scaling the target medical video to obtain a scaled medical video;
[0071] Step S103: input the scaled medical video into a video anomaly detection model for anomaly detection to obtain an anomaly detection result of the target medical video.
[0072] Among them, the target medical video is used to indicate the object behavior of the target object in the target medical scene; the abnormality detection result is used to indicate that the target object in the target medical video has abnormal object behavior or indicates that the target object in the target medical video does not have abnormal object behavior.
[0073] It should be noted that the video anomaly detection model is obtained by dividing the sample medical video into sample video clips, and training a preset first model based on the comparison of the first video clip features of the sample video clips with the second video clip features, and the comparison of the predicted video labels of the sample video clips with the sample video labels; the first video clip features are obtained by inputting the sample video clips into the first model for feature extraction; the predicted video label is predicted based on the first video clip features; the second video clip features are obtained by inputting the sample video clips into the preset second model for feature extraction.
[0074] Steps S101 to S106 shown in the embodiment of the present application are as follows: obtaining a target medical video, the target medical video is used to indicate the object behavior of the target object in the target medical scene; scaling the target medical video to obtain a scaled medical video; inputting the scaled medical video into a video anomaly detection model for anomaly detection to obtain an anomaly detection result of the target medical video, and being able to use the video anomaly detection model to identify the abnormal object behavior features in the low-dimensional scaled medical video, and being able to improve the accuracy of detecting abnormal behavior of objects in medical videos. Among them, the video anomaly detection model is obtained by segmenting the sample medical video into sample video clips, and training a preset first model based on the comparison of the first video clip features of the sample video clips with the second video clip features, and the comparison of the predicted video labels of the sample video clips with the sample video labels. The first video clip features are obtained by inputting the sample video clips into the first model for feature extraction; the predicted video labels are predicted based on the first video clip features; and the second video clip features are obtained by inputting the sample video clips into the preset second model for feature extraction. The embodiment of the present application adopts a method combining feature comparison and label comparison during model training, which can realize model training at the feature level and model training at the classification prediction level, and is beneficial to training the first model's video feature information analysis capabilities in the feature extraction stage and the anomaly detection stage, thereby improving the model training effect and improving the detection accuracy of the trained video anomaly detection model for abnormal object behaviors in medical videos.
[0075] In step S101 of some embodiments, a target medical video is acquired.
[0076] The target medical video is some video data generated in various medical scenarios.
[0077] The target medical video is used to indicate the behavior of the target object in the target medical scene. For example, the target medical video is used to indicate the activities of medical staff in a certain period of time (communication behaviors in various medical departments, use behaviors of certain medical equipment, etc.).
[0078] In the specific implementation of this embodiment, with authorization, a video that meets the requirements is extracted from the video library of the target medical scene as the target medical video. The video that meets the requirements in the video library refers to a video containing the target object collected in a fixed area at different time periods.
[0079] In step S102 of some embodiments, the target medical video is scaled to obtain a scaled medical video.
[0080] The scaled medical video refers to a video formed after resizing each video frame of the target medical video. The scaled medical video and the target medical video have the same video content, but there are certain differences in video resolution, clarity, etc.
[0081] When this embodiment is implemented, the target medical video can be scaled by using a proportional scaling method. Specifically, while keeping the aspect ratio of the target medical video unchanged, the width and height of the target medical video are changed at the same time, thereby changing the resolution of the target medical video to obtain a scaled medical video, wherein the aspect ratio of the scaled medical video is the same as that of the target medical video.
[0082] In an embodiment of the present application, the video anomaly detection model can be a neural network model based on the ResNet18 structure, wherein the video anomaly detection model includes a first convolutional layer, a maximum pooling layer, a second convolutional layer, an average pooling layer, and a prediction layer.
[0083] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S205:
[0084] Step S201, extracting a first feature of the scaled medical video based on the first convolutional layer to obtain a first medical video convolution feature;
[0085] Step S202, performing maximum pooling processing on the first medical video convolution feature based on the maximum pooling layer to obtain the first medical video pooling feature;
[0086] Step S203, performing a second feature extraction on the first medical video pooling feature based on the second convolutional layer to obtain a second medical video convolutional feature;
[0087] Step S204, performing average pooling processing on the second medical video convolution feature based on the average pooling layer to obtain a second medical video pooling feature;
[0088] Step S205, performing feature dimensionality reduction on the second medical video pooling feature based on the prediction layer to obtain a reduced-dimensional medical video feature, and performing anomaly detection based on the reduced-dimensional medical video feature to obtain anomaly detection results.
[0089] Steps S201 to S205 are described in detail below.
[0090] In step S201 of some embodiments, a first feature extraction is performed on the scaled medical video based on the first convolutional layer to extract video content information in the scaled medical video and obtain a first medical video convolution feature.
[0091] In step S202 of some embodiments, maximum pooling processing is performed on the first medical video convolution feature based on the maximum pooling layer, so as to achieve downsampling processing of the first medical video convolution feature while retaining important feature information of the first medical video convolution feature, so as to obtain the first medical video pooling feature.
[0092] In step S203 of some embodiments, a second feature extraction is performed on the first medical video pooling feature based on the second convolutional layer to capture important video information in the first medical video pooling feature to obtain a second medical video convolutional feature.
[0093] In step S204 of some embodiments, average pooling processing is performed on the second medical video convolution feature based on the average pooling layer to reduce the dimension of each feature element in the second medical video convolution feature, and each feature element in the second medical video convolution feature that has undergone dimension reduction is averaged to obtain the second medical video pooling feature.
[0094] In step S205 of some embodiments, first, the second medical video pooling feature is input into the prediction layer, and the prediction layer inputs the second medical video pooling feature into the low-dimensional vector space to achieve feature dimensionality reduction of the second medical video pooling feature, and obtain a one-dimensional reduced-dimensional medical video feature. Then, the reduced-dimensional medical video feature is input into the preset function of the prediction layer, and the preset function performs abnormality detection on the reduced-dimensional medical video feature, generates a probability distribution for the target medical video, and outputs the abnormality detection result according to the probability distribution. Among them, the preset function is a softmax function, and the preset function is used to perform binary classification on the target medical video according to the reduced-dimensional medical video feature. The probability distribution is used to indicate the first probability that the target medical video has abnormal object behavior, and the second probability that the target medical video does not have abnormal object behavior. When the first probability in the probability distribution is greater than the second probability, the abnormal detection result is that the target object in the target medical video has abnormal object behavior; when the first probability in the probability distribution is less than the second probability, the abnormal detection result is that the target object in the target medical video does not have abnormal object behavior.
[0095] The benefit of this embodiment is that, for the scaled medical video, after the scaled medical video is input into the video anomaly detection model, after two convolutions and two poolings, the important video information in the scaled medical video can be effectively extracted, and feature extraction in a low-dimensional space can be achieved. The prediction layer is used for anomaly detection, and the possibility of abnormal object behavior in the target object in the target medical video can be quantified by probability, which is conducive to improving the accuracy of anomaly detection in medical scenarios.
[0096] See also Figure 3 In some embodiments, the specific process of training the video anomaly detection model may include but is not limited to steps S301 to S305:
[0097] Step S301, obtaining a sample medical video and a reference video label of the sample medical video;
[0098] Step S302, segmenting the sample medical video to obtain a plurality of sample video segments, and determining the reference video label as the sample video label of the sample video segment;
[0099] Step S303: for each sample video segment, scaling the sample video segment, and inputting the scaled sample video segment into the first model for feature extraction to obtain a first video segment feature of the sample video segment, and performing anomaly detection based on the first video segment feature to obtain a predicted video label of the sample video segment;
[0100] Step S304, inputting the sample video segment into the second model for feature extraction to obtain second video segment features of the sample video segment;
[0101] Step S305 , based on the comparison of the first video segment features and the second video segment features of the plurality of sample video segments, and the comparison of the predicted video labels and the sample video labels, a first model is trained to obtain a video anomaly detection model.
[0102] Steps S301 to S305 are described in detail below.
[0103] In step S301 of some embodiments, the sample medical video is a medical video containing a sample object for training anomaly detection model, and the reference video label of the sample medical video is used to indicate whether the sample object in the sample medical video has abnormal behavior.
[0104] In the specific implementation of this embodiment, the reference video label of the sample medical video can be obtained by manual labeling or machine labeling. The sample medical video is generated by:
[0105] Get original medical videos;
[0106] Scaling the original medical video to obtain an intermediate medical video;
[0107] The intermediate medical video is subjected to noise reduction processing to obtain a sample medical video.
[0108] Specifically, first, with authorization, extract the medical videos containing each sample object from the video library corresponding to the medical scene, and annotate the medical videos for abnormalities to obtain the original medical video and the video label of the original medical video. Then, scale the original medical video proportionally to adjust the resolution of the original medical video without changing the aspect ratio of the original medical video to obtain an intermediate medical video. Furthermore, since there is common random noise in the intermediate medical video, the intermediate medical video can be denoised by Gaussian filtering to obtain a sample medical video, and the video label of the original medical video is directly used as the reference video label of the sample medical video.
[0109] Among them, the noise reduction processing of the intermediate medical video can be expressed as follows:
[0110]
[0111] Where I refers to the sample medical video, k is the template size of Gaussian filtering, K = 3 × 3; G(i, j) is a two-dimensional Gaussian kernel, i is the horizontal coordinate of the two-dimensional Gaussian kernel G(i,j) in the two-dimensional space, and j is the vertical coordinate of the two-dimensional Gaussian kernel G(i,j) in the two-dimensional space. 0 Refers to the Intermediate Medical Video, I 0 (m,n) refers to the element value in the intermediate medical video with coordinates (m,n) in the two-dimensional space, where m is the horizontal coordinate of an element value of the intermediate medical video, and n is the vertical coordinate of the element value of the intermediate medical video.
[0112]
[0113] In step S302 of some embodiments, first, when segmenting the sample medical video, the sample medical video is segmented into a plurality of sample video segments of equal length, with three video frames as one segment. Next, the reference video label of the sample medical video is determined as the sample video label of each sample video segment corresponding to the sample medical video. This method can achieve effective expansion of the sample data for training the video anomaly detection model, and can obtain more positive and negative sample video segments, thereby achieving the effect of using more sample examples for model training. Among them, the positive sample video segment refers to the sample video segment whose sample video label is that the sample object does not have abnormal object behavior, and the negative sample video segment refers to the sample video segment whose sample video label is that the sample object has abnormal object behavior.
[0114] In step S303 of some embodiments, first, for each sample video clip, the sample video clip can be scaled in a proportional manner. Specifically, while keeping the aspect ratio of the sample video clip unchanged, the width and height of the sample video clip are changed at the same time, thereby changing the resolution of the sample video clip. Next, the scaled sample video clip is input into the first model for feature extraction, and the convolution layer and pooling layer (first convolution layer, maximum pooling layer, second convolution layer, average pooling layer) of the first model capture the video clip information of the scaled sample video clip to obtain the first video clip feature. Furthermore, the prediction layer of the first model performs anomaly detection on the first video clip feature to obtain the predicted video label of the sample video clip, and its specific implementation process is similar to the above-mentioned steps S102 and S103. To save space, it will not be repeated.
[0115] In step S304 of some embodiments, the sample video segment is input into the second model for feature extraction, and the important video segment information of the sample video segment is captured by the second model to obtain the second video segment feature of the sample video segment.
[0116] In step S305 of some embodiments, first, the first video clip features and the second video clip features of multiple sample video clips are compared to determine the feature differences between the first model and the second model at the feature level for the same sample video clip. Next, the predicted video label output by the second model is compared with the real sample video label to determine the label difference between the prediction result of the first model and the real label. Further, the first model is trained according to the feature differences and label differences of the multiple sample video clips to implement supervised training and comparative training of the first model, and the model parameters of the first model are continuously adjusted. The above steps S301-S305 are repeated to perform cyclic iterative training on the first model. When the feature differences and label differences of the multiple sample video clips meet the training requirements, the iterative training is stopped, and the first model that enables the feature differences and label differences of the multiple sample video clips to meet the training requirements is used as a video anomaly detection model.
[0117] The benefit of this embodiment is that after scaling and denoising the original medical video as a sample medical video, irrelevant video information in the sample medical video can be effectively reduced, thereby improving the video quality of the sample medical video. Furthermore, the sample medical video is also divided into a plurality of video clips of equal length, which can effectively increase the amount of sample data input to the first model and the second model. After the sample video clips are respectively input to the first model and the second model to obtain the first video clip features and the second video clip features, the first model is trained by comparing the feature differences between the first video clip features and the second video clip features; the first model is supervised by comparing the labels of the predicted video labels and the sample video labels. This method integrates multiple training forms (comparative training and supervised training) in model training, and can train the feature extraction ability and video anomaly detection ability of the first model at the feature level and the classification prediction level, thereby effectively improving the accuracy of the model in detecting anomalies in medical videos.
[0118] In an embodiment of the present application, the second model may be a neural network model based on the ResNet50 structure, and the second model includes a third convolutional layer, a pooling layer, and a fourth convolutional layer.
[0119] See also Figure 4 In some embodiments, step S304 may include but is not limited to steps S401 to S403:
[0120] Step S401, performing a first convolution process on the sample video segment based on the third convolution layer to obtain a convolution feature of the sample video;
[0121] Step S402, performing maximum pooling processing on the sample video convolution features based on the pooling layer to obtain the sample video pooling features;
[0122] Step S403: Perform a second convolution process on the sample video pooling features based on the fourth convolution layer to obtain a second video clip feature.
[0123] Steps S401 to S403 are described in detail below.
[0124] In step S401 of some embodiments, a first convolution process is performed on the sample video segment based on the third convolution layer to extract video segment content information in the sample video segment to obtain a sample video convolution feature.
[0125] In step S402 of some embodiments, maximum pooling processing is performed on the sample video convolution features based on the pooling layer to capture global video feature information in the sample video convolution features to obtain the sample video pooling features.
[0126] In step S403 of some embodiments, a second convolution process is performed on the sample video pooling features based on the fourth convolution layer to capture important video feature information in the sample video pooling features to obtain a second video clip feature.
[0127] The advantage of this embodiment is that, for the sample video clip, after the sample video clip is input into the second model, after two convolutions and one pooling, the second model can fully extract the important video feature information contained in the sample video clip, thereby extracting the second video clip features, and can use the second video clip features as reference features during model training. The first model can be trained at the feature level to learn and extract the feature information of the sample video clip, which is conducive to improving the accuracy of the model in detecting abnormal video features in medical scenarios.
[0128] See also Figure 5 In some embodiments, step S305 may include but is not limited to steps S501 to S504:
[0129] Step S501, determining a first loss function based on a comparison of a first video segment feature and a second video segment feature of a plurality of sample video segments;
[0130] Step S502, determining a second loss function based on a comparison between the predicted video labels of the plurality of sample video clips and the sample video labels;
[0131] Step S503, adding the first loss function and the second loss function to obtain a total loss function;
[0132] Step S504: Based on the total loss function, train the first model to obtain a video anomaly detection model.
[0133] Steps S501 to S504 are described in detail below.
[0134] In step S501 of some embodiments, for each sample video clip, the first video clip feature and the second video clip feature are compared to achieve a difference comparison of the sample video clips at the feature level, and the second video clip feature can be used as a reference to construct a feature loss function of the sample video clip by using a comparative training method, wherein the feature loss function is used to indicate the degree of feature difference between the first video clip feature and the second video clip feature of a single sample video clip. Furthermore, the feature loss functions of multiple sample video clips are averaged to obtain a first loss function, wherein the first loss function is used to indicate the overall degree of feature difference between the first video clip feature and the second video clip feature of multiple sample video clips.
[0135] In step S502 of some embodiments, for each sample video clip, the predicted video label and the sample video label are compared to determine the degree of difference between the prediction result of the first model and the true label of the sample video clip, to obtain the classification loss function of the sample video clip, and to sum the classification loss functions of multiple sample video clips to obtain a second loss function.
[0136] Specifically, for each sample video clip, if the sample video label of the sample video clip is that the sample object in the sample video clip does not have abnormal object behavior, the sample video clip is considered to be a positive sample video clip, and the negative logarithm of the predicted video label is multiplied by the predicted video label to obtain the classification loss function of the sample video clip. If the sample video label of the sample video clip is that the sample object in the sample video clip has abnormal object behavior, the sample video clip is considered to be a negative sample video clip, and the negative logarithm of the result of 1 minus the predicted video label is taken to obtain a first logarithm result, and then the result of 1 minus the predicted video label is multiplied by the first logarithm result to obtain the classification loss function of the sample video clip.
[0137] Among them, the second loss function can be expressed as shown in formula (1):
[0138]
[0139] Among them, l 1 is the second loss function, i is the i-th sample video clip, is the predicted video label of the i-th sample video clip, and the predicted video label is a probability value; y(i) is the sample video label of the i-th sample video clip, wherein the sample video label y(i) is 0 or 1. When the sample video label indicates that the sample object in the sample video clip does not have abnormal object behavior, the sample video label y(i) is 1; when the sample video label indicates that the sample object in the sample video clip has abnormal object behavior, the sample video label y(i) is 0.
[0140] In step S503 of some embodiments, the first loss function and the second loss function may be directly added to obtain a total loss function, wherein the total loss function is used to indicate the overall anomaly detection accuracy of the first model at the feature level and the classification prediction level.
[0141] Among them, the total loss function can be expressed as shown in formula (2):
[0142] L = l 1 +l 2 Formula (2);
[0143] Among them, L is the total loss function, l 2 is the first loss function.
[0144] In step S504 of some embodiments, in each iteration round, the output value of the total loss function is compared with the preset loss value. If the output value of the total loss function is greater than or equal to the preset loss value, the model parameters of the first model are adjusted, and the above steps S301 to S305 are repeated until the output value of the total loss function is less than the preset loss value in a certain iteration round, the training of the first model is stopped, and the first model of the current iteration round is used as the final video anomaly detection model.
[0145] The advantage of this embodiment is that comparative training of the first model is achieved by comparing the feature differences between the features of the first video clip and the features of the second video clip; supervised training of the first model is achieved by comparing the labels of the predicted video labels and the sample video labels, and the feature extraction capability and video anomaly detection capability of the first model can be iteratively trained at the feature level and the classification prediction level, thereby effectively improving the accuracy of the model in detecting anomalies in medical videos.
[0146] In the embodiment of the present application, the first video clip feature includes a first abnormal probability feature of each video element in the sample video clip, and the second video clip feature includes a second abnormal probability feature of each video element in the sample video clip.
[0147] Video elements refer to characteristic elements of sample video clips at various positions in two-dimensional space.
[0148] The first abnormal probability feature is used to indicate the abnormal probability size of the video element in the first video segment feature predicted by the first model.
[0149] The second abnormal probability feature is used to indicate the abnormal probability size of the video element in the second video segment feature predicted by the second model.
[0150] See also Figure 6 In some embodiments, step S601 includes but is not limited to steps S601 to S604:
[0151] Step S601, for each sample video clip, performing divergence calculation on the first video clip feature and the second video clip feature to obtain a first sub-loss function of the sample video clip;
[0152] Step S602: for each sample video clip, taking the logarithm of the quotient of the first abnormal probability feature and the second abnormal probability feature of each video element to obtain a logarithmic result, multiplying the logarithmic result and the first abnormal probability feature to obtain a weighted abnormal probability feature, and adding the weighted abnormal probability features of multiple video elements to obtain a second sub-loss function of the sample video clip;
[0153] Step S603, performing weighted calculation on the first sub-loss function and the second sub-loss function to obtain a weighted loss function;
[0154] Step S604: integrating the weighted loss functions of the multiple sample video clips to obtain a first loss function.
[0155] Steps S601 to S604 are described in detail below.
[0156] In step S601 of some embodiments, when performing divergence calculation on the first video segment feature and the second video segment feature for each sample video segment, firstly perform feature difference calculation on the same video element in the first video segment feature and the second video segment feature to obtain the feature difference of the video element, and then square the feature difference to obtain the square result of the feature difference. Then, sum the square results of the feature differences of all video elements to obtain the total square result, and finally, take the square root of the total square result to obtain the first sub-loss function of the sample video segment.
[0157] Among them, the first sub-loss function of the sample video clip can be expressed as shown in formula (3):
[0158]
[0159] Among them, l loss1 is the first sub-loss function, is the feature of the first video clip; s indicates that the first model can be regarded as a student model, and s is the abbreviation of the student model; is the feature of the second video clip; t indicates that the second model can be regarded as the teacher model, and t is the abbreviation of the teacher model.
[0160] In step S602 of some embodiments, for each sample video clip, the logarithm of the quotient of the first abnormality probability feature and the second abnormality probability feature of each video element is taken to obtain a logarithmic result, the logarithmic result and the first abnormality probability feature are multiplied to obtain a weighted abnormality probability feature, and the weighted abnormality probability features of multiple video elements are added to obtain a second sub-loss function of the sample video clip.
[0161] Among them, the second sub-loss function of the sample video clip is shown in formula (4):
[0162]
[0163] in, Indicates the first video segment features The first abnormal probability feature of the video element with coordinate position (p, q); Represents the second video segment feature The second abnormal probability feature of the video element with coordinate position (p, q); loss2 is the second sub-loss function.
[0164] In step S603 of some embodiments, the first sub-loss function and the second sub-loss function are weighted to obtain a weighted loss function, where the weighted loss function can be expressed as λ 1 l loss1 +λ 2 l loss2 , where λ 1 and λ 2 is a hyperparameter used to balance the first and second sub-loss functions, λ 1 and λ 2 All are set to 0.001. The weighted loss function can be understood as the contrast loss of each sample video clip.
[0165] In step S604 of some embodiments, when the weighted loss functions of multiple sample video clips are integrated, the weighted loss functions of the multiple sample video clips are summed to obtain a first loss function.
[0166] The advantage of this embodiment is that, for each sample video clip, the divergence calculation and variant cross entropy calculation methods are adopted to construct the first sub-loss function and the second sub-loss function respectively, and the contrast loss of the sample video clip is constructed by the weighted results of the first sub-loss function and the second sub-loss function. The contrast training and knowledge distillation strategies are used to train the first model, which can effectively make up for the defect of low accuracy of model training caused by insufficient label data. In addition, the knowledge distillation method is used to realize the contrast training of the first model at the feature level, which can better train the first model's feature extraction ability for effective video information, which is conducive to the accuracy of video anomaly detection of training models in multiple dimensions.
[0167] like Figure 7 The figure is a schematic diagram of the implementation process of training a video anomaly detection model. Specifically, first, the sample video segment X is input into the second model, and the feature of the second video segment is obtained after the feature extraction of the second model. At the same time, the sample video segment X is scaled, and the scaled sample video segment X is input into the first model, and the first model performs feature extraction and anomaly prediction to obtain the first video segment feature and predict video labels Furthermore, based on the comparison between the predicted video label and the sample video label, the second loss function l is calculated. 1 ; According to the first video clip features and the second video clip features Compare the first sub-loss function l and calculate it respectively.loss1 , and the second sub-loss function l loss2 Finally, the first model is trained by constructing a total loss function based on the first sub-loss function, the second sub-loss function, and the second loss function, and the model parameters of the first model are adjusted. The above process is repeated continuously to implement iterative training of the first model until the training requirements are met. The training is stopped and the first model that meets the training requirements is used as the video anomaly detection model.
[0168] See also Figure 8 The embodiment of the present application further provides a video anomaly detection device, which can implement the above-mentioned video anomaly detection method, and the device includes:
[0169] An acquisition unit 801 is used to acquire a target medical video, where the target medical video is used to indicate an object behavior of a target object in a target medical scene;
[0170] A scaling unit 802 is used to scale the target medical video to obtain a scaled medical video;
[0171] A detection unit 803 is used to input the zoomed medical video into a video anomaly detection model for anomaly detection, and obtain an anomaly detection result of the target medical video, wherein the anomaly detection result is used to indicate that an abnormal object behavior exists in a target object in the target medical video or to indicate that an abnormal object behavior does not exist in the target medical video;
[0172] Among them, the video anomaly detection model is obtained by dividing the sample medical video into sample video clips, and training a preset first model based on the comparison of the first video clip features of the sample video clips with the second video clip features, and the comparison of the predicted video labels of the sample video clips with the sample video labels; the first video clip features are obtained by inputting the sample video clips into the first model for feature extraction; the predicted video label is predicted based on the first video clip features; the second video clip features are obtained by inputting the sample video clips into the preset second model for feature extraction.
[0173] In some embodiments, the video anomaly detection model includes a first convolutional layer, a maximum pooling layer, a second convolutional layer, an average pooling layer, and a prediction layer;
[0174] The detection unit 803 is used for:
[0175] Extracting a first feature of the scaled medical video based on the first convolutional layer to obtain a first medical video convolutional feature;
[0176] Performing maximum pooling processing on the first medical video convolution feature based on the maximum pooling layer to obtain the first medical video pooling feature;
[0177] Performing a second feature extraction on the first medical video pooling feature based on the second convolutional layer to obtain a second medical video convolution feature;
[0178] Performing average pooling processing on the second medical video convolution feature based on the average pooling layer to obtain the second medical video pooling feature;
[0179] Based on the prediction layer, feature dimension reduction is performed on the second medical video pooling feature to obtain a reduced dimension medical video feature, and anomaly detection is performed based on the reduced dimension medical video feature to obtain anomaly detection results.
[0180] In some embodiments, the video anomaly detection model is trained in the following manner:
[0181] A sample acquisition unit (not shown), used to acquire a sample medical video and a reference video tag of the sample medical video;
[0182] a segmentation unit (not shown), configured to segment the sample medical video to obtain a plurality of sample video segments, and determine the reference video label as the sample video label of the sample video segment;
[0183] a first feature extraction unit (not shown), configured to scale the sample video segment for each sample video segment, input the scaled sample video segment into a first model for feature extraction, obtain first video segment features of the sample video segment, and perform anomaly detection based on the first video segment features to obtain a predicted video label of the sample video segment;
[0184] A second feature extraction unit (not shown), configured to input the sample video segment into a second model for feature extraction to obtain a second video segment feature of the sample video segment;
[0185] A training unit (not shown) is used to train a first model based on comparison of first video segment features and second video segment features of multiple sample video segments, and comparison of predicted video labels and sample video labels to obtain a video anomaly detection model.
[0186] In some embodiments, the second model includes a third convolutional layer, a pooling layer, and a fourth convolutional layer;
[0187] The second feature extraction unit (not shown) is used to:
[0188] Performing a first convolution process on the sample video segment based on the third convolution layer to obtain a convolution feature of the sample video;
[0189] Based on the pooling layer, the sample video convolution features are processed by maximum pooling to obtain the sample video pooling features;
[0190] Based on the fourth convolution layer, a second convolution process is performed on the sample video pooling features to obtain a second video clip feature.
[0191] In some embodiments, a training unit (not shown) is used to:
[0192] Determining a first loss function based on a comparison of a first video segment feature and a second video segment feature of the plurality of sample video segments;
[0193] Determining a second loss function based on a comparison of the predicted video labels of the plurality of sample video clips and the sample video labels;
[0194] Add the first loss function and the second loss function to obtain the total loss function;
[0195] Based on the total loss function, the first model is trained to obtain a video anomaly detection model.
[0196] In some embodiments, the first video segment feature includes a first abnormal probability feature of each video element in the sample video segment, and the second video segment feature includes a second abnormal probability feature of each video element in the sample video segment;
[0197] Determining a first loss function based on a comparison between a first video segment feature and a second video segment feature of a plurality of sample video segments includes:
[0198] For each sample video clip, performing divergence calculation on the first video clip feature and the second video clip feature to obtain a first sub-loss function of the sample video clip;
[0199] For each sample video clip, taking the logarithm of the quotient of the first abnormal probability feature and the second abnormal probability feature of each video element to obtain a logarithmic result, multiplying the logarithmic result and the first abnormal probability feature to obtain a weighted abnormal probability feature, and adding the weighted abnormal probability features of multiple video elements to obtain a second sub-loss function of the sample video clip;
[0200] Performing weighted calculation on the first sub-loss function and the second sub-loss function to obtain a weighted loss function;
[0201] The weighted loss functions of multiple sample video clips are integrated to obtain a first loss function.
[0202] In some embodiments, the sample medical video is generated by:
[0203] Get original medical videos;
[0204] Scaling the original medical video to obtain an intermediate medical video;
[0205] The intermediate medical video is subjected to noise reduction processing to obtain a sample medical video.
[0206] The specific implementation of the video anomaly detection device is basically the same as the specific implementation of the above-mentioned video anomaly detection method, and will not be repeated here.
[0207] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned video anomaly detection method. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.
[0208] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0209] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0210] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the video anomaly detection method of the embodiment of this application;
[0211] Input / output interface 903, used to implement information input and output;
[0212] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);
[0213] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);
[0214] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .
[0215] An embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the above-mentioned video anomaly detection method.
[0216] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0217] The video anomaly detection method, video anomaly detection device, electronic device and computer-readable storage medium provided by the embodiment of the present application obtain a target medical video, the target medical video is used to indicate the object behavior of the target object in the target medical scene; the target medical video is scaled to obtain a scaled medical video; the scaled medical video is input into a video anomaly detection model for anomaly detection to obtain an anomaly detection result of the target medical video, and the video anomaly detection model can be used to identify the abnormal object behavior features in the low-dimensional scaled medical video, and the detection accuracy of the abnormal behavior of the object in the medical video can be improved. Among them, the video anomaly detection model is obtained by dividing the sample medical video into sample video clips, and training a preset first model based on the comparison of the first video clip feature of the sample video clip with the second video clip feature, and the comparison of the predicted video label of the sample video clip with the sample video label. The first video clip feature is obtained by inputting the sample video clip into the first model for feature extraction; the predicted video label is predicted based on the first video clip feature; the second video clip feature is obtained by inputting the sample video clip into the preset second model for feature extraction. The embodiment of the present application adopts a method combining feature comparison and label comparison during model training, which can realize model training at the feature level and model training at the classification prediction level, and is conducive to training the video feature information analysis capability of the first model in the feature extraction stage and the anomaly detection stage, thereby improving the model training effect and improving the detection accuracy of the trained video anomaly detection model for abnormal object behavior in medical videos. This method realizes the increase of sample diversity and data volume by multiple examples through the method of sample video segmentation. At the same time, the first video clip feature and the second video clip feature are compared during model training, and the prediction effect of the first model on abnormal behavior is improved by comparing low-resolution feature maps, which can effectively reduce the false alarm rate of the video anomaly detection model for abnormal object behavior. In addition, deploying the video anomaly detection model in medical scenarios can effectively reduce the cost of manual detection, realize automatic detection of abnormal behavior in the video area, and improve the accuracy and convenience of video abnormal behavior detection in the medical field.
[0218] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0219] It can be understood by those skilled in the art that Figure 1-7 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.
[0220] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0221] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0222] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0223] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0224] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0225] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0226] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0227] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0228] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A video anomaly detection method, characterized in that: The method comprises: Acquire a target medical video, where the target medical video is used to indicate an object behavior of a target object in a target medical scene; Performing scaling processing on the target medical video to obtain a scaled medical video; Inputting the zoomed medical video into a video anomaly detection model for anomaly detection to obtain an anomaly detection result of the target medical video, wherein the anomaly detection result is used to indicate that the target object in the target medical video has an abnormal object behavior or indicates that the target object in the target medical video does not have an abnormal object behavior; Among them, the video anomaly detection model is obtained by dividing the sample medical video into sample video clips, and training a preset first model based on the comparison of the first video clip features of the sample video clips with the second video clip features, and the comparison of the predicted video labels of the sample video clips with the sample video labels; the first video clip features are obtained by inputting the sample video clips into the first model for feature extraction; the predicted video labels are predicted based on the first video clip features; the second video clip features are obtained by inputting the sample video clips into the preset second model for feature extraction.
2. The method according to claim 1, characterized in that: The video anomaly detection model includes a first convolutional layer, a maximum pooling layer, a second convolutional layer, an average pooling layer, and a prediction layer; The step of inputting the zoomed medical video into a video anomaly detection model for anomaly detection to obtain an anomaly detection result of the target medical video includes: Extracting a first feature of the scaled medical video based on the first convolutional layer to obtain a first medical video convolutional feature; Performing maximum pooling processing on the first medical video convolution feature based on the maximum pooling layer to obtain a first medical video pooling feature; Performing a second feature extraction on the first medical video pooling feature based on the second convolutional layer to obtain a second medical video convolutional feature; Performing average pooling processing on the second medical video convolution feature based on the average pooling layer to obtain a second medical video pooling feature; Based on the prediction layer, feature dimension reduction is performed on the second medical video pooling feature to obtain a reduced dimension medical video feature, and anomaly detection is performed based on the reduced dimension medical video feature to obtain the anomaly detection result.
3. The method according to claim 1, characterized in that: The video anomaly detection model is trained in the following way: Obtaining a sample medical video and a reference video label of the sample medical video; Segmenting the sample medical video to obtain a plurality of sample video segments, and determining the reference video label as the sample video label of the sample video segment; For each of the sample video clips, scaling the sample video clip, and inputting the scaled sample video clip into the first model for feature extraction to obtain the first video clip feature of the sample video clip, and performing anomaly detection based on the first video clip feature to obtain a predicted video label of the sample video clip; Inputting the sample video segment into the second model for feature extraction to obtain the second video segment features of the sample video segment; Based on the comparison of the first video segment features and the second video segment features of a plurality of the sample video segments, and the comparison of the predicted video labels and the sample video labels, the first model is trained to obtain the video anomaly detection model.
4. The method according to claim 3, characterized in that The second model includes a third convolutional layer, a pooling layer, and a fourth convolutional layer; The step of inputting the sample video segment into the second model for feature extraction to obtain the second video segment features of the sample video segment includes: Performing a first convolution process on the sample video segment based on the third convolution layer to obtain a sample video convolution feature; Based on the pooling layer, a maximum pooling process is performed on the sample video convolution feature to obtain a sample video pooling feature; Based on the fourth convolution layer, a second convolution process is performed on the sample video pooling feature to obtain the second video clip feature.
5. The method according to claim 3, characterized in that: The step of training the first model based on the comparison of the first video segment features and the second video segment features of the plurality of sample video segments and the comparison of the predicted video labels and the sample video labels to obtain the video anomaly detection model includes: Determining a first loss function based on a comparison of the first video segment features and the second video segment features of a plurality of the sample video segments; Determining a second loss function based on a comparison of the predicted video labels and the sample video labels of the plurality of sample video clips; Adding the first loss function and the second loss function to obtain a total loss function; Based on the total loss function, the first model is trained to obtain the video anomaly detection model.
6. The method according to claim 5, characterized in that The first video segment feature includes a first abnormal probability feature of each video element in the sample video segment, and the second video segment feature includes a second abnormal probability feature of each video element in the sample video segment; The determining of a first loss function based on a comparison of the first video segment feature and the second video segment feature of the plurality of sample video segments comprises: For each of the sample video clips, performing divergence calculation on the first video clip feature and the second video clip feature to obtain a first sub-loss function of the sample video clip; For each of the sample video clips, taking the logarithm of the quotient of the first abnormal probability feature and the second abnormal probability feature of each of the video elements to obtain a logarithmic result, multiplying the logarithmic result and the first abnormal probability feature to obtain a weighted abnormal probability feature, and adding the weighted abnormal probability features of multiple video elements to obtain a second sub-loss function of the sample video clip; Performing weighted calculation on the first sub-loss function and the second sub-loss function to obtain a weighted loss function; The weighted loss functions of the plurality of sample video clips are integrated to obtain the first loss function.
7. The method according to any one of claims 1 to 6, characterized in that: The sample medical video is generated in the following way: Get original medical videos; Scaling the original medical video to obtain an intermediate medical video; The intermediate medical video is subjected to noise reduction processing to obtain the sample medical video.
8. A video anomaly detection device, characterized in that: The device comprises: an acquisition unit, configured to acquire a target medical video, wherein the target medical video is used to indicate an object behavior of a target object in a target medical scene; A scaling unit, used for scaling the target medical video to obtain a scaled medical video; a detection unit, configured to input the zoomed medical video into a video anomaly detection model for anomaly detection, and obtain an anomaly detection result of the target medical video, wherein the anomaly detection result is used to indicate that the target object in the target medical video has an abnormal object behavior or indicates that the target object in the target medical video does not have an abnormal object behavior; Among them, the video anomaly detection model is obtained by dividing the sample medical video into sample video clips, and training a preset first model based on the comparison of the first video clip features of the sample video clips with the second video clip features, and the comparison of the predicted video labels of the sample video clips with the sample video labels; the first video clip features are obtained by inputting the sample video clips into the first model for feature extraction; the predicted video labels are predicted based on the first video clip features; the second video clip features are obtained by inputting the sample video clips into the preset second model for feature extraction.
9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the video anomaly detection method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video anomaly detection method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Abnormal behavior detection method and device, equipment and medium
CN114581836A
Industrial control network intrusion detection method and device based on knowledge distillation
CN118233222A
Unsupervised efficient video retrieval method, device and equipment based on comparative learning
CN118410206A
Fast anomaly detection method and system based on contrastive representation distillation
US20230368372A1
