A method for analyzing and warning of illegal operation video for bright kitchen and bright stove scene

Abstract

The present application relates to the field of video monitoring and artificial intelligence technology, and more particularly to a method for analyzing and warning against illegal operation video in a bright kitchen and bright stove scene.A method for analyzing and warning against illegal operation video in a bright kitchen and bright stove scene includes the following steps: S1: collecting video data of normal operation of the back kitchen in the bright kitchen and bright stove scene and the spatial position of each functional area, and dividing the video data based on the functional area to obtain regional normal video data; S2: generating pseudo-abnormal sample data according to the regional normal video data, the pseudo-abnormal sample data including spatial pseudo-abnormal samples, motion distortion pseudo-abnormal samples, shielding pseudo-abnormal samples and rhythm pseudo-abnormal samples.The present application effectively enhances the feature perception and response capability of the mouse infestation and other super-small targets by constructing a pseudo-abnormal generation strategy based on a spatiotemporal attention heat map and a grid-based spatial discretization, and significantly improves the detection recall rate under low-light and low-pixel ratio conditions.

Description

Technical Field

[0001] This invention relates to the fields of video surveillance and artificial intelligence technology, and in particular to a method for video analysis and early warning of violations in open kitchen scenarios. Background Technology

[0002] The "transparent kitchen" project, by deploying video surveillance systems in the kitchen, makes the food processing and preparation process transparent, serving as a crucial technological means to ensure food safety in catering services. In recent years, with the rapid development of artificial intelligence technology, intelligent monitoring systems based on video analytics have been gradually applied to kitchen scenarios. Some systems can already automatically identify and issue alarms for high-frequency violations such as chefs not wearing masks, improper work clothes, and smoking in non-smoking areas. JD.com's food delivery platform has also upgraded its AI patrol system to identify food safety hazards such as unauthorized personnel entering the kitchen, smoking in the kitchen, improper attire, and rodent infestations, improving monitoring efficiency to some extent. However, existing technologies still face many challenges in the practical application of transparent kitchens, and the monitoring effect is far from ideal. First, the kitchen environment is complex, cramped, and has variable lighting conditions. Especially at night under low light conditions, small targets like rodents occupy very few pixels in the monitoring image, and existing target detection algorithms have a significantly low recall rate for such ultra-small targets, making them difficult to effectively capture. Second, there is significant mutual occlusion between personnel, stoves, shelves, and steamers in the kitchen. Chefs are frequently obscured by other objects during operations, leading to interruptions in target detection, loss of identity, and difficulty in continuously determining violations. Third, violations are low-probability events, and obtaining and labeling real violation video samples is extremely difficult, resulting in a weak data foundation for supervised training. Existing systems mostly rely on a small number of manually labeled samples, severely limiting the model's generalization ability. Fourth, existing methods are mostly based on single-frame target detection plus simple temporal voting rules, lacking a deep understanding of the temporal logic of behavior. They struggle to distinguish between accidental actions and continuous violations, resulting in low accuracy in recognizing complex behaviors such as the multi-stage actions of smoking and the continuous judgment of using a mobile phone.

[0003] Therefore, there is an urgent need to develop a video analysis and early warning method for violations in open kitchen scenarios to overcome the above-mentioned shortcomings and achieve efficient, accurate and intelligent early warning of violations in the kitchen. Summary of the Invention

[0004] To overcome the shortcomings of insufficient small target detection capability, lack of occlusion scene processing, difficulty in obtaining violation samples, and weak behavior temporal discrimination capability in open kitchen scenarios, this invention provides a video analysis and early warning method for violations in open kitchen scenarios.

[0005] The technical implementation scheme of the present invention is: a video analysis and early warning method for violations in open kitchen scenarios, comprising the following steps:

[0006] S1: Collect video data of normal kitchen operations and the spatial location of each functional area in the open kitchen scenario; and divide the video data based on the functional areas to obtain normal video data of the area.

[0007] S2: Generate pseudo-anomaly sample data based on the normal video data of the region. The pseudo-anomaly sample data includes spatial pseudo-anomaly samples, motion distortion pseudo-anomaly samples, occlusion pseudo-anomaly samples, and rhythm pseudo-anomaly samples.

[0008] S3: Obtain the confidence score of each pseudo-anomaly sample based on the pseudo-anomaly sample data, and obtain the pseudo-anomaly training set based on the confidence score;

[0009] S4: Construct a dual-channel encoder model and train the dual-channel encoder model based on the pseudo-anomaly training set;

[0010] S5: Obtain the deviation based on the trained dual-channel encoder model, and trigger an early warning based on the deviation.

[0011] Preferably, the step of collecting video data of normal kitchen operations and the spatial location of each functional area in an open kitchen scenario, and dividing the video data based on the functional areas to obtain regional normal video data, includes: collecting video data of the kitchen through a high-definition network camera; determining the spatial location of each functional area based on the video data; preprocessing the video data; and then dividing the video data into regional normal video data according to the functional areas covered by the video screen. The regional normal video data includes video data of the cooking area, video data of the cleaning area, video data of the storage area, and video data of the food preparation area.

[0012] Preferably, the step of generating pseudo-anomaly sample data based on the normal video data of the region, wherein the pseudo-anomaly sample data includes spatial pseudo-anomaly samples, motion distortion pseudo-anomaly samples, occlusion pseudo-anomaly samples, and rhythmic pseudo-anomaly samples, includes: extracting each frame of the normal video data of the region, segmenting each frame of the image into image blocks, and calculating the self-attention weights between each image block; and constructing a spatiotemporal attention heatmap based on the self-attention weights. Based on the spatiotemporal attention heatmap, low-attention areas are identified; the kitchen space is discretized into a grid diagram. ,in For a set of nodes, The edge set is used; based on the mesh graph, a minimum normal response path is planned between low-interest regions, and a motion trajectory is synthesized based on the minimum normal response path to generate spatial pseudo-anomaly samples. .

[0013] Preferably, the motion distortion pseudo-anomaly samples include: for each extracted frame image The RAFT optical flow estimation algorithm is used to obtain the dense optical flow field between the image and its adjacent frames. The optical flow field divergence is obtained through the optical flow field divergence function. In the low-interest area Inside, an abnormal motion field is constructed. By solving the Poisson equation with boundary conditions Injection target divergence distribution The pixels within the region are distorted during rendering to generate pseudo-anomaly samples of motion distortion. ,in For the Laplace operator, Let be the stream function to be solved. The target optical flow field includes a high-divergence mode and a low-divergence mode.

[0014] Preferably, the occlusion pseudo-anomaly sample includes: performing target detection and instance segmentation on each extracted image frame to obtain identifiable objects in the image; for each identifiable object, determining its depth level to obtain the front-to-back occlusion relationship between objects; and constructing an occlusion relationship graph by establishing directed edges based on the identifiable objects as nodes and the front-to-back occlusion relationship between objects. The occlusion relationship graph is sorted using a topological sorting algorithm to obtain the occlusion level of each identifiable object; based on the occlusion level, new occlusion relationships are generated using a level insertion strategy or a level swapping strategy, and video frames are re-rendered to generate occlusion pseudo-anomaly samples. .

[0015] Preferably, the rhythm pseudo-anomaly sample includes: extracting features from normal video data in the region to obtain a multi-dimensional time-series signal, and performing a short-time Fourier transform on the multi-dimensional time-series signal to obtain a normal operating rhythm spectrum. Based on the normal operating rhythm spectrum, the multi-dimensional temporal signal is distorted by dynamic time warping to deviate the alignment path from the identity mapping, thereby obtaining multi-scale rhythm anomalies; the multi-dimensional temporal signal is edited according to the multi-scale rhythm anomalies, and the edited multi-dimensional temporal signal is mapped back to video frames to generate rhythm pseudo-anomaly samples. .

[0016] Preferably, obtaining the confidence score of each pseudo-anomaly sample based on the pseudo-anomaly sample data includes: for any pseudo-anomaly sample Obtain the corresponding set of spatial locations of pseudo-anomalies. Based on the set of spatial locations of the pseudo-anomalies and the low-interest region Obtain a spatial rationality score Based on the pseudo-anomaly samples in each frame of the image Corresponding optical flow field Obtain the optical flow difference value between adjacent frames A temporal coherence score is obtained based on the optical flow difference value. Based on the normal video data and pseudo-abnormal sample data of the region, the embedded feature set of normal samples is obtained through a pre-trained feature extraction network. Embedding features of pseudo-anomaly samples The semantic deviation score is obtained based on the embedding feature set of the normal samples and the embedding features of the pseudo-abnormal samples. The confidence score is obtained through a confidence scoring function based on the spatial rationality score, temporal coherence score, and semantic deviation score. .

[0017] Preferably, obtaining the pseudo-anomaly training set based on the confidence score includes: retaining pseudo-anomaly samples with confidence scores greater than a preset first confidence threshold in the pseudo-anomaly training set; marking pseudo-anomaly samples with confidence scores greater than a preset second confidence threshold in the pseudo-anomaly training set as high-confidence samples; and performing quantity balance control on the pseudo-anomaly samples in the pseudo-anomaly training set to make the number of samples of each type tend to be balanced.

[0018] Preferably, the construction of the dual-channel encoder model, training the dual-channel encoder model based on the pseudo-anomaly training set, includes: constructing a dual-channel encoder model, wherein the first encoder is used to learn the feature representation of normal data and possess video reconstruction capabilities, and a pre-trained feature extraction network is used as the initial weights for training; the second encoder is used to learn the discriminative features between normal and anomalies; the two encoders share the underlying convolutional parameters; using different time segments of normal video data in the same region as positive sample pairs, using pseudo-anomaly samples (not labeled with high confidence) in the pseudo-anomaly training set as negative sample pairs, and using pseudo-anomaly samples (labeled as high confidence) in the pseudo-anomaly training set as hard negative sample pairs, the dual-channel encoder model is trained using a dynamic difficulty scheduling strategy and an adversarial training strategy; the loss function of the dual-channel encoder model is: ,in For the total loss, To rebuild the losses, To compare the losses, For consistency loss, This is the loss weighting coefficient.

[0019] Preferably, the step of obtaining the deviation based on the trained dual-channel encoder model and triggering an early warning based on the deviation further includes: extracting features from all normal video segments of the normal video data in the region using the trained first-channel encoder, calculating the mean and covariance matrix of the feature distribution, and establishing a normal mode basis based on the mean and covariance matrix; setting a fixed duration window for the real-time video stream. Extract real-time video segments within a fixed duration window, and extract window features from the real-time video segments using the trained first encoder. The deviation is obtained based on the window features and the normal mode base. Define dynamic early warning thresholds. ,when An alert is triggered at any time.

[0020] This invention effectively enhances the feature perception and response capabilities for ultra-small targets such as rodents by constructing a pseudo-anomaly generation strategy based on spatiotemporal attention heatmaps and gridded spatial discretization, significantly improving the detection recall rate under low-light and low-pixel-ratio conditions. For complex occlusion scenarios in kitchens, this method introduces an occlusion relationship graph construction and hierarchical reasoning mechanism, which can maintain the continuity of target identity and the integrity of behavior judgment even when people and kitchen utensils are mutually occluded, greatly improving robustness in occlusion scenarios. Addressing the industry pain point of extremely scarce real violation samples, this method innovatively sets... This paper proposes a scheme to generate four types of pseudo-anomaly samples: spatial, motion distortion, occlusion, and rhythm. A confidence scoring and high-confidence screening mechanism is introduced to construct a high-quality pseudo-anomaly training set, effectively alleviating the problem of insufficient supervised training data and significantly improving the model's generalization ability. Simultaneously, this method, through a dual-encoder model combined with contrastive learning and consistency constraints, fully explores the inherent distribution patterns of normal data and achieves dynamic early warning based on temporal window feature deviation calculation. This effectively distinguishes between accidental actions and continuous violations, endowing the system with a deep semantic understanding of complex temporal behaviors. In summary, this method comprehensively overcomes multiple shortcomings of existing technologies in terms of detection accuracy, occlusion robustness, sample dependence, and temporal discrimination, providing an efficient, accurate, and intelligent early warning solution for violations in transparent kitchen scenarios. Attached Figure Description

[0021] Figure 1 A flowchart of a video analysis and early warning method for violations in open kitchen scenarios;

[0022] Figure 2 This is a flowchart of a kitchen supervision method based on spatiotemporal pseudo-anomaly generation and collaborative training of a dual-channel twin encoder. Detailed Implementation

[0023] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Example: A video analysis and early warning method for violations in open kitchen scenarios, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0025] S1: Collect video data of normal kitchen operations and the spatial location of each functional area in the open kitchen scenario; and divide the video data based on the functional areas to obtain normal video data of the area.

[0026] High-definition network cameras are used to collect video data from the kitchen, and the spatial location of each functional area is determined based on the video data. The video data is preprocessed, and then divided into normal video data for each area according to the functional area covered by the video screen. The normal video data for each area includes video data of the cooking area, the cleaning area, the storage area, and the food preparation area.

[0027] It should also be noted that the high-definition network cameras cover breakfast, lunch, dinner, and off-peak hours. These cameras, with a resolution of at least 1920×1080 pixels, are installed in various functional areas of the kitchen, including a cooking area, a cleaning area, a storage area, and a food preparation area. The cooking area covers the stove and wok, capturing video footage of the cooking process. The cleaning area covers the sink and dishwasher, capturing video footage of food washing and utensil disinfection. The storage area covers the staple food storage, non-staple food storage, and refrigeration / freezing equipment, capturing video footage of the food storage environment. The food preparation area covers the cutting and preparation stations, capturing video footage of food cutting and preparation. Based on the kitchen video footage captured by the high-definition network cameras, the pixel coordinate ranges of the cooking, cleaning, storage, and food preparation areas are identified through functional area boundary calibration to determine the spatial location of each functional area.

[0028] The preprocessing includes: using a spatial filtering method to perform Gaussian filtering or median filtering on each frame of the image to remove sensor noise and transmission noise; and using a temporal filtering method to perform motion-adaptive noise reduction by utilizing the pixel correlation between adjacent frames to suppress temporal noise while maintaining clear motion edges.

[0029] S2: Generate pseudo-anomaly sample data based on the normal video data of the region. The pseudo-anomaly sample data includes spatial pseudo-anomaly samples, motion distortion pseudo-anomaly samples, occlusion pseudo-anomaly samples, and rhythm pseudo-anomaly samples.

[0030] Extract each frame of normal video data from the region, segment each frame into image blocks, and calculate the self-attention weights between each image block; construct a spatiotemporal attention heatmap based on the self-attention weights. Based on the spatiotemporal attention heatmap, low-attention areas are identified; the kitchen space is discretized into a grid diagram. ,in For a set of nodes, The edge set is used; based on the mesh graph, a minimum normal response path is planned between low-interest regions, and a motion trajectory is synthesized based on the minimum normal response path to generate spatial pseudo-anomaly samples. .

[0031] For each extracted frame image The RAFT optical flow estimation algorithm is used to obtain the dense optical flow field between the image and its adjacent frames. The optical flow field divergence is obtained through the optical flow field divergence function. In the low-interest area Inside, an abnormal motion field is constructed. By solving the Poisson equation with boundary conditions Injection target divergence distribution The pixels within the region are distorted during rendering to generate pseudo-anomaly samples of motion distortion. ,in For the Laplace operator, Let be the stream function to be solved. The target optical flow field includes a high-divergence mode and a low-divergence mode.

[0032] For each extracted image frame, object detection and instance segmentation are performed to obtain identifiable objects in the image. For each identifiable object, its depth level is determined to obtain the front-to-back occlusion relationship between objects. Using the identifiable objects as nodes, directed edges are established for the front-to-back occlusion relationship between objects to construct an occlusion relationship graph. The occlusion relationship graph is sorted using a topological sorting algorithm to obtain the occlusion level of each identifiable object; based on the occlusion level, new occlusion relationships are generated using a level insertion strategy or a level swapping strategy, and video frames are re-rendered to generate occlusion pseudo-anomaly samples. .

[0033] Feature extraction is performed on the normal video data of the region to obtain a multi-dimensional time-series signal, and a short-time Fourier transform is performed on the multi-dimensional time-series signal to obtain the normal operation rhythm spectrum. Based on the normal operating rhythm spectrum, the multi-dimensional temporal signal is distorted by dynamic time warping to deviate the alignment path from the identity mapping, thereby obtaining multi-scale rhythm anomalies; the multi-dimensional temporal signal is edited according to the multi-scale rhythm anomalies, and the edited multi-dimensional temporal signal is mapped back to video frames to generate rhythm pseudo-anomaly samples. .

[0034] It should also be noted that each frame of the image is divided into N×N image blocks. Each image block is converted into a feature vector through linear mapping, and positional encoding is added to preserve spatial location information. The feature vectors of each image block are input into the Transformer encoder to calculate the multi-head self-attention weights. For the , Layer The first point of attention, the first The image patch for the first Attention weights for each image patch ,in For the query vector, by position The vector obtained by linear transformation of image patch features; This is the matrix transpose operator; Let the key vector be the position. The vector obtained by linear transformation of image patch features; The dimensions of the query vector and key vector; To normalize the exponential function, Min-Max normalization is used; for each frame, the attention weights of each layer and head are aggregated according to spatial location to obtain the spatial attention distribution of that frame. The spatial attention distribution represents location. The degree of attention received in this frame; the attention distribution of all frames within the same video segment is summed over time and normalized to obtain the spatiotemporal attention heatmap of that segment. ,in The total number of frames; satisfying the spatiotemporal attention heatmap. Spatial locations with values ​​below a preset attention threshold are marked as low-attention areas. This preset attention threshold is determined by statistically analyzing the spatiotemporal attention heatmap distribution of a large number of normal kitchen videos, and the heatmap quantile value at which the low-attention area accounts for 5% of the total pixels is taken as the threshold. The kitchen space is discretized into a grid map. Node set The coordinates of each node are corresponding to its spatial location. edge set Connect adjacent nodes using four-neighbor connections; edge weights. Represents the distance between nodes. ,in The Euclidean distance from each node to the nearest obstacle, where the obstacle is an impassable area such as a stove, shelf, or equipment; To balance the coefficients, a grid search is employed. On a validation set labeled with normal motion trajectories, the goal is to minimize the average distance between the minimum normal response path and the actual trajectory of a person / object. Cross-validation is used to determine the optimal value. The process of planning the minimum normal response path is as follows: the distance of the starting node is set to 0, and the distances of all other nodes are set to infinity. All nodes are marked as unvisited. The node with the smallest distance among all unvisited nodes is selected as the current node. All its neighboring nodes are traversed. If the distance to a neighboring node through the current node is less than the current distance of the neighboring node, the distance of the neighboring node is updated. The algorithm terminates when the target node is visited, and the minimum normal response path is obtained by backtracking along the parent node pointer. The process of generating spatial pseudo-anomaly samples is as follows: a rectangular block is generated on the minimum normal response path, with a size set to... The image is set to 5×5 to 15×15 pixels and moves frame by frame along the path at a speed of 1 to 3 pixels per frame. At the same time, random scale jitter and directional jitter are applied to the rectangular block to simulate the randomness of the movement of real small targets. At certain nodes of the path, a sequence of human key points is synthesized. The behavior of simulating a person staying in a non-operational area for a long time is set to 5 to 30 seconds. If the time exceeds the preset reasonable time threshold, it constitutes a pseudo-anomaly. The synthesized motion trajectory is superimposed on the corresponding position of the original normal video frame. The Poisson image editing method is used to ensure the natural integration of the synthesized target and the background to generate a pseudo-anomaly video sample with spatial anomalies, which is the spatial pseudo-anomaly sample. The preset reasonable time threshold is based on the distribution of the continuous time of personnel in each functional area in the normal operation video of the kitchen, and the upper quartile of the distribution is taken as the threshold.

[0035] For each extracted frame image Obtain its image and adjacent frame images The dense optical flow field between them is estimated using the RAFT optical flow algorithm. This algorithm outputs a high-precision dense optical flow field by constructing a 4D correlation volume and a cyclic update module. ,in The horizontal optical flow component. The optical flow component in the vertical direction. and All represent pixels from arrive Displacement; optical flow field divergence This reflects the convergence and divergence of the motion field; positive values ​​indicate divergence, and negative values ​​indicate convergence. The optical flow field divergence function is... In the selected low-concern areas Construct an abnormal motion field inside. This causes its divergence distribution to deviate significantly from the normal distribution. Solve the Poisson equation with Dirichlet boundary conditions. ,in These are constraints; The value of the stream function on the boundary; This represents the original boundary optical flow value. For the region The boundary conditions ensure continuity with the original optical flow at the region boundary; target divergence distribution. It includes a high-divergence mode and a low-divergence mode. The high-divergence mode has a divergence value 3 to 5 times the normal divergence mean, simulating abnormal motion such as sudden turns and stops. The low-divergence mode suppresses the divergence value to near zero, simulating an abnormal state of motion stagnation. The low-interest region is discretized into a pixel grid, and the Poisson equation is applied using the finite difference method. Transform it into a large sparse linear system of equations and change the boundary conditions. Substituting this system of equations as Dirichlet boundary constraints, the linear equations are then numerically solved using a multigrid iterative method: low-frequency residuals are calculated on a coarse grid to obtain the overall trend, high-frequency details are corrected on a fine grid, and a V-shaped iterative process is used until the residuals are less than [a certain value]. Thus, the flow function corresponding to each pixel position within the local editing region is obtained. For the obtained stream function Calculate the spatial gradient to obtain the edited abnormal motion field. Using the obtained abnormal motion field, the reverse mapping is performed on... The pixels within the target frame are distorted during rendering. Find its corresponding position in the source frame based on the motion field. The pixel value of the source frame at this position is obtained by bilinear interpolation and filled into the target frame. The pseudo-anomaly video sequence with local motion distortion is obtained by processing frame by frame, which is the motion distortion pseudo-anomaly sample.

[0036] For each frame of a normal video, object detection and instance segmentation are performed using Mask R-CNN or YOLACT algorithms to identify identifiable objects in the frame. These identifiable objects include, but are not limited to, people, cooking utensils, countertops, shelves, containers, and food. For each identifiable object, its depth level is determined based on its relative position in the frame: at occlusion boundaries, objects with complete outlines appear in front, while objects with truncated outlines appear behind. The process of sorting the occlusion graph using a topological sorting algorithm to obtain the occlusion level of each identifiable object is as follows: calculate the in-degree of each node in the graph, add all nodes with an in-degree of 0 to a queue, assign them level 1, remove nodes from the queue, delete all their outgoing edges, update the in-degree of neighboring nodes, and if a neighboring node's in-degree becomes 0, add it to the queue and assign it to the next level. Repeat this process of removing nodes from the queue until all nodes are assigned a level. The level insertion strategy is to insert the node into the occlusion graph at the first... A pseudo-object block is inserted into each layer to simulate small targets or foreign objects such as mice and cockroaches. The size of the pseudo-object block is set to 5×5 to 30×30 pixels. The occlusion relationship between the pseudo-object block and each object is automatically determined using topological sorting. The pseudo-object block is assigned a layer k, and its occlusion relationship with objects in the same layer and adjacent layers is automatically determined based on the degree of spatial overlap. The layer swapping strategy involves randomly selecting two adjacent layer objects A and B and swapping their layer positions to create unreasonable occlusion. After the swap, during re-rendering, B will occlude A, producing a visual anomaly. The video frames are re-rendered according to the new occlusion relationship. During rendering, each object is drawn sequentially from the bottom layer to the top layer, with the later drawn object covering the earlier drawn object to achieve the correct occlusion effect. Frame-by-frame processing generates pseudo-anomaly videos with hierarchical pseudo-occlusion as pseudo-anomaly samples.

[0037] The multi-dimensional timing signal includes an action frequency signal. Position change rate signal and hand movement speed signal The aforementioned action frequency signal reflects the level of activity in the kitchen, and is collected through... The number of atomic actions occurring within 5 seconds before and after a given moment; the position change rate signal reflects the overall movement activity of the kitchen. The average rate of position change of all trackable targets in the real-time frame, where the trackable targets are people or hands; the hand movement speed signal reflects the rhythm of manual operation. The average motion speed of all hand key points detected at any given time; the window size of the short-time Fourier transform is 2s, with an overlap rate of 50%; the normal operation rhythm spectrum is obtained by performing a short-time Fourier transform on the multi-dimensional temporal signal, and then the signal is nonlinearly distorted in a short window of 3-5 frames and a long window of 30-50 frames respectively, so that the alignment path deviates from the identity mapping, generating multi-scale rhythm anomalies; the multi-scale rhythm anomalies include micro-rhythm anomalies in a short time window of 3 to 5 frames and macro-rhythm anomalies in a long time window of 30 to 50 frames; the micro-rhythm anomalies are single actions compressed to 50% to 70% of the original duration or stretched to 130% to 200% of the original duration; the macro-rhythm anomalies are entire operation segments compressed to 60% to 80% of the original duration or stretched to 120% to 150% of the original duration; the edited multi-dimensional temporal signal is mapped back to video frames, and pseudo-anomaly videos with rhythm anomalies are generated by non-uniform sampling of the original video frames as rhythm pseudo-anomaly samples.

[0038] S3: Obtain the confidence score of each pseudo-anomaly sample based on the pseudo-anomaly sample data, and obtain the pseudo-anomaly training set based on the confidence score;

[0039] For any pseudo-anomaly sample Obtain the corresponding set of spatial locations of pseudo-anomalies. Based on the set of spatial locations of the pseudo-anomalies and the low-interest region Obtain a spatial rationality score Based on the pseudo-anomaly samples in each frame of the image Corresponding optical flow field Obtain the optical flow difference value between adjacent frames A temporal coherence score is obtained based on the optical flow difference value. Based on the normal video data and pseudo-abnormal sample data of the region, the embedded feature set of normal samples is obtained through a pre-trained feature extraction network. Embedding features of pseudo-anomaly samples The semantic deviation score is obtained based on the embedding feature set of the normal samples and the embedding features of the pseudo-abnormal samples. The confidence score is obtained through a confidence scoring function based on the spatial rationality score, temporal coherence score, and semantic deviation score. .

[0040] The pseudo-anomaly samples with confidence scores greater than a preset first confidence threshold are retained in the pseudo-anomaly training set, and the pseudo-anomaly samples with confidence scores greater than a preset second confidence threshold in the pseudo-anomaly training set are marked as high-confidence samples; the pseudo-anomaly samples in the pseudo-anomaly training set are subject to quantity balance control so that the number of samples of each type tends to be balanced.

[0041] It should also be noted that for spatial pseudo-anomaly samples, the spatial locations traversed by their synthetic motion trajectories are directly extracted and aggregated as a set of pseudo-anomaly change spatial locations; for motion distortion pseudo-anomaly samples, the spatial locations where they are distorted and rendered within low-interest areas are aggregated as a set of pseudo-anomaly change spatial locations; for occlusion pseudo-anomaly samples, the spatial locations where they are inserted into pseudo-object blocks or swapped with other objects are aggregated as a set of pseudo-anomaly change spatial locations; for rhythmic pseudo-anomaly samples, the spatial locations of their edited regions are aggregated as a set of pseudo-anomaly change spatial locations. The spatial rationality score is used to evaluate whether the location of the pseudo-anomaly conforms to the abnormal spatial distribution pattern of the kitchen scene. The higher the proportion of pseudo-anomalies appearing in low-interest areas, the better. The larger, through Obtain, among which Indicates the area of ​​the region. Indicates intersection; optical flow difference value ,in Total number of pixels The distance is Euclidean; the temporal coherence score is used to evaluate the smoothness of the inter-frame transition of pseudo-anomalies in the time dimension, reducing visual jumps caused by pseudo-anomalies. The smoother the inter-frame transition, the better. The larger, through Obtain, among which Total number of frames The average optical flow difference value of normal videos is used. Normal video segments and pseudo-abnormal video segments are input into the video Transformer model respectively, and the global average pooling feature vector of the penultimate layer is extracted as the embedding feature. The embedding features of all normal samples constitute a set. Embedded features of each pseudo-anomaly sample ; Calculate the mean of the set based on the embedded feature set of the normal samples. Covariance Matrix The semantic deviation score is used to evaluate the semantic distance between pseudo-abnormal samples and normal samples in the feature space, through... Obtain the Mahalanobis distance between the embedded features of pseudo-abnormal samples and the embedded features of normal samples. Normalize it: ,in To calculate the 95th percentile of the Mahalanobis distance distribution between all pairwise normal samples; ultimately, through... Obtain semantic deviation score ,in To determine the optimal deviation, the Mahalanobis distance distribution between all pairs of normal samples is statistically analyzed, and the normalized deviation value corresponding to the mean of this distribution is taken as the optimal deviation. For bandwidth parameters, cross-validation or rules of thumb are used to determine the bandwidth. Set to 1.5-2 times the standard deviation of the normalized deviation of normal samples; the confidence score function is... ,in The optimal value of the confidence weight coefficient is determined by using grid search on the validation set with the goal of maximizing the consistency between the confidence scores of pseudo-anomaly samples and the score distribution of manually labeled real anomaly samples.

[0042] The preset first confidence threshold is based on the curve of pseudo-anomaly sample retention rate and model anomaly detection recall rate corresponding to different confidence cutoff values ​​on the validation set. The confidence value at the inflection point of the recall rate is taken as the preset first confidence threshold. The preset second confidence threshold aims to maximize the contribution of high-confidence samples as difficult negative samples to the improvement of the model's discrimination ability in contrastive learning. The upper quartile of the confidence score distribution is selected as the preset second confidence threshold through grid search on the validation set. The process of quantity balance control is as follows: the number of retained pseudo-anomaly samples, motion distortion pseudo-anomaly samples, occlusion pseudo-anomaly samples, and rhythm pseudo-anomaly samples are statistically analyzed. Based on the median number of samples in the four categories, random undersampling is performed on categories with a number exceeding 1.5 times the median, and supplementary generation is performed on categories with a number less than 0.5 times the median, so that the number of retained pseudo-anomaly samples of each category tends to be balanced.

[0043] S4: Construct a dual-channel encoder model and train the dual-channel encoder model based on the pseudo-anomaly training set;

[0044] A dual-encoder model is constructed. The first encoder learns the feature representation of normal data and possesses video reconstruction capabilities, using a pre-trained feature extraction network as initial weights for training. The second encoder learns the discriminative features between normal and abnormal data. Both encoders share the underlying convolutional parameters. Different time segments of normal video data from the same region are used as positive sample pairs. Pseudo-abnormal samples (unlabeled high-confidence samples in the pseudo-abnormal training set) are used as negative sample pairs. Hard negative sample pairs are used between normal video data from the region and pseudo-abnormal samples (labeled as high-confidence samples in the pseudo-abnormal training set). A dynamic difficulty scheduling strategy and an adversarial training strategy are employed to train the dual-encoder model. The loss function of the dual-encoder model is: ,in For the total loss, To rebuild the losses, To compare the losses, For consistency loss, This is the loss weighting coefficient.

[0045] It should also be noted that the first encoder uses a 3D convolutional neural network as its backbone, containing five 3D convolutional blocks and a global average pooling layer. It outputs a fixed-length feature vector and includes a decoder branch for reconstructing video frames. This decoder branch consists of 3D transposed convolutional layers and uses a pre-trained feature extraction network as initial weights during training. The second encoder shares the weight parameters of the first four 3D convolutional blocks with the first encoder, branches after the last convolutional block, and additionally includes a fully connected layer and a binary classification output layer. The dynamic difficulty scheduling strategy primarily uses simple negative samples in the early stages of training. As the number of iterations increases, the sampling ratio of difficult negative samples is gradually increased, and the model's discrimination accuracy at each difficulty level is periodically evaluated. The sampling weights at each level are adaptively adjusted. Simultaneously, the temperature coefficient of the contrast loss is dynamically adjusted based on the current training loss. When the loss decreases slowly, the temperature is reduced, forcing the model to increase its feature discrimination strength for difficult samples. Through progressive learning from easy to difficult, the model achieves stable convergence and continuously improves its discrimination ability for boundary samples. The adversarial training strategy introduces a lightweight discriminator to form a generative adversarial game with the first encoder to distinguish between reconstructed frames and real normal frames, forcing the reconstructed features to approximate normal features. The distribution utilizes the discriminant output of the second encoder as an adversarial signal. A gradient inversion layer enhances the robustness of the first encoder against pseudo-anomaly perturbations. During training, the encoder and discriminator parameters are alternately updated. When the discriminator can accurately distinguish between normal and pseudo-anomalies, the adversarial loss weight of the encoder is increased simultaneously, prompting the model to discover a more generalized normal pattern base, thereby effectively suppressing overfitting. The reconstruction loss uses mean squared error or L1 loss, calculating the pixel-by-pixel difference between the video frame reconstructed by the first encoder and the original input frame to constrain the encoder to retain details of the normal video. The contrast loss uses InfoNCE or... The NT-Xent loss uses different time segments of the same normal segment as positive sample pairs and normal segments and pseudo-abnormal segments as negative sample pairs, maximizing the mutual information of positive sample pairs and minimizing the mutual information of negative sample pairs. The consistency loss calculates the mean square error or cosine similarity loss between the output features of the first and second encoders for the same video segment, prompting the two encoders to align in the feature space and enhancing the consistency of representation. The loss weight coefficients are optimized using Bayesian optimization on the validation set, with the optimization objective of maximizing the F1-score or AUC-ROC of anomaly detection, to determine the optimal value of the loss weight coefficients.

[0046] S5: Obtain the deviation based on the trained dual-channel encoder model, and trigger an early warning based on the deviation.

[0047] The trained first encoder is used to extract features from all normal video segments of the region's normal video data. The mean and covariance matrix of the feature distribution are calculated, and a normal pattern basis is established based on the mean and covariance matrix. A fixed duration window is set for the real-time video stream. Extract real-time video segments within a fixed duration window, and extract window features from the real-time video segments using the trained first encoder. The deviation is obtained based on the window features and the normal mode base. Define dynamic early warning thresholds. ,when An alert is triggered at any time.

[0048] It should also be noted that the normal videos in the training set are divided into several video segments, each containing 16 consecutive frames with 8 overlapping frames between adjacent segments. Each video segment is input into the first encoder after training, and the output of the global average pooling layer is extracted as the embedding feature vector for that segment, thus obtaining the normal sample embedding feature set. The mean vector of the embedding feature set is then calculated. Covariance Matrix Then establish a normal pattern base. Set a fixed duration window for the real-time video stream. With sliding step size Extract video segments within a window and extract window features using the first encoder. The fixed-duration window is based on the statistical distribution of the duration of typical violations in the kitchen, and the median of this distribution is taken as the window duration; the sliding step size is set to 1 / 3 to 1 / 2 of the fixed-duration window; the deviation degree indicates the degree to which the behavior pattern of the current window deviates from the normal baseline. The larger the value, the further the current window's behavior deviates from the normal baseline. The dynamic warning threshold is obtained by using the statistical distribution of deviations of all normal video segments in the training set, combined with the 5% quantile on the validation set as a benchmark, and is updated adaptively using the mean deviation of the recent normal windows in the real-time video stream. ,in The mean of the statistical distribution. Let be the standard deviation of the statistical distribution. The parameters are dynamically adjusted, with a value range of 2-5. After triggering an alert, a structured alarm record containing a heat map of the abnormal area location and a confidence score is generated. Key video clips and typical frames before and after the abnormal window are automatically extracted as evidence for archiving. At the same time, pop-up notifications, audio-visual prompts, or SMS messages are immediately pushed to the monitoring terminal, and the violation timeline and spatial coordinates are highlighted on the monitoring screen for manual review and handling.

[0049] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that variations may be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A video analysis and early warning method for violations in open kitchen scenarios, characterized in that, Includes the following steps: S1: Collect video data of normal kitchen operations and the spatial location of each functional area in the open kitchen scenario; and divide the video data based on the functional areas to obtain normal video data of the area. S2: Generate pseudo-anomaly sample data based on the normal video data of the region. The pseudo-anomaly sample data includes spatial pseudo-anomaly samples, motion distortion pseudo-anomaly samples, occlusion pseudo-anomaly samples, and rhythm pseudo-anomaly samples. S3: Obtain the confidence score of each pseudo-anomaly sample based on the pseudo-anomaly sample data, and obtain the pseudo-anomaly training set based on the confidence score; S4: Construct a dual-channel encoder model and train the dual-channel encoder model based on the pseudo-anomaly training set; S5: Obtain the deviation based on the trained dual-channel encoder model, and trigger an early warning based on the deviation.

2. The video analysis and early warning method for violations in open kitchen scenarios according to claim 1, characterized in that, The process of collecting video data of normal kitchen operations and the spatial location of each functional area under a transparent kitchen scenario, and then dividing the video data based on the functional areas to obtain regional normal video data, includes: collecting video data of the kitchen through a high-definition network camera; determining the spatial location of each functional area based on the video data; preprocessing the video data; and then dividing the video data into regional normal video data according to the functional areas covered by the video screen. The regional normal video data includes video data of the cooking area, the cleaning area, the storage area, and the food preparation area.

3. The video analysis and early warning method for violations in open kitchen scenarios as described in claim 1, characterized in that, The step of generating pseudo-anomaly sample data based on the normal video data of the region, wherein the pseudo-anomaly sample data includes spatial pseudo-anomaly samples, motion distortion pseudo-anomaly samples, occlusion pseudo-anomaly samples, and rhythmic pseudo-anomaly samples, includes: extracting each frame of the normal video data of the region, segmenting each frame of the image into image blocks, and calculating the self-attention weights between each image block; and constructing a spatiotemporal attention heatmap based on the self-attention weights. Based on the spatiotemporal attention heatmap, low-attention areas are identified; the kitchen space is discretized into a grid diagram. ,in For a set of nodes, The edge set is used; based on the mesh graph, a minimum normal response path is planned between low-interest regions, and a motion trajectory is synthesized based on the minimum normal response path to generate spatial pseudo-anomaly samples. .

4. A video analysis and early warning method for violations in open kitchen scenarios as described in claim 3, characterized in that, The motion distortion pseudo-anomaly samples include: for each extracted frame image The RAFT optical flow estimation algorithm is used to obtain the dense optical flow field between the image and its adjacent frames. The optical flow field divergence is obtained through the optical flow field divergence function. In the low-interest area Inside, an abnormal motion field is constructed. By solving the Poisson equation with boundary conditions Injection target divergence distribution The pixels within the region are distorted during rendering to generate pseudo-anomaly samples of motion distortion. ,in For the Laplace operator, Let be the stream function to be solved. The target optical flow field includes a high-divergence mode and a low-divergence mode.

5. A video analysis and early warning method for violations in open kitchen scenarios as described in claim 3, characterized in that, The occlusion pseudo-anomaly samples include: performing object detection and instance segmentation on each extracted image frame to obtain identifiable objects in the image; for each identifiable object, determining its depth level to obtain the front-to-back occlusion relationship between objects; and constructing an occlusion relationship graph by establishing directed edges for the front-to-back occlusion relationship between objects, using the identifiable objects as nodes. The occlusion relationship graph is sorted using a topological sorting algorithm to obtain the occlusion level of each identifiable object; based on the occlusion level, new occlusion relationships are generated using a level insertion strategy or a level swapping strategy, and video frames are re-rendered to generate occlusion pseudo-anomaly samples. .

6. A video analysis and early warning method for violations in open kitchen scenarios as described in claim 3, characterized in that, The rhythm pseudo-anomaly sample includes: extracting features from normal video data in the region to obtain a multi-dimensional time-series signal, and performing a short-time Fourier transform on the multi-dimensional time-series signal to obtain a normal operating rhythm spectrum. Based on the normal operating rhythm spectrum, the multi-dimensional temporal signal is distorted by dynamic time warping to deviate the alignment path from the identity mapping, thereby obtaining multi-scale rhythm anomalies; the multi-dimensional temporal signal is edited according to the multi-scale rhythm anomalies, and the edited multi-dimensional temporal signal is mapped back to video frames to generate rhythm pseudo-anomaly samples. .

7. A video analysis and early warning method for violations in open kitchen scenarios as described in claim 1, characterized in that, The step of obtaining the confidence score of each pseudo-anomaly sample based on the pseudo-anomaly sample data includes: for any pseudo-anomaly sample Obtain the corresponding set of spatial locations of pseudo-anomaly changes. Based on the set of spatial locations of the pseudo-anomalies and the low-interest region Obtain a spatial rationality score Based on the pseudo-anomaly samples in each frame of the image Corresponding optical flow field Obtain the optical flow difference value between adjacent frames A temporal coherence score is obtained based on the optical flow difference value. Based on the normal video data and pseudo-abnormal sample data of the region, the embedded feature set of normal samples is obtained through a pre-trained feature extraction network. Embedding features of pseudo-anomaly samples The semantic deviation score is obtained based on the embedding feature set of the normal samples and the embedding features of the pseudo-abnormal samples. The confidence score is obtained through a confidence scoring function based on the spatial rationality score, temporal coherence score, and semantic deviation score. .

8. A video analysis and early warning method for violations in open kitchen scenarios as described in claim 1, characterized in that, The step of obtaining the pseudo-anomaly training set based on the confidence score includes: retaining pseudo-anomaly samples with confidence scores greater than a preset first confidence threshold into the pseudo-anomaly training set; marking pseudo-anomaly samples with confidence scores greater than a preset second confidence threshold in the pseudo-anomaly training set as high-confidence samples; and performing quantity balance control on the pseudo-anomaly samples in the pseudo-anomaly training set to make the number of samples of each type tend to be balanced.

9. A video analysis and early warning method for violations in open kitchen scenarios as described in claim 1, characterized in that, The construction of the dual-channel encoder model, based on the pseudo-anomaly training set, includes: constructing a dual-channel encoder model, wherein the first encoder is used to learn the feature representation of normal data and possess video reconstruction capabilities, using a pre-trained feature extraction network as initial weights for training; the second encoder is used to learn the discriminative features between normal and anomalies; the two encoders share the underlying convolutional parameters; using different time segments of normal video data from the same region as positive sample pairs; using normal video data from the region and pseudo-anomaly samples (not labeled with high confidence) from the pseudo-anomaly training set as negative sample pairs; and using normal video data from the region and pseudo-anomaly samples (labeled with high confidence) from the pseudo-anomaly training set as hard negative sample pairs; and employing a dynamic difficulty scheduling strategy and an adversarial training strategy to train the dual-channel encoder model; the loss function of the dual-channel encoder model is: ,in For the total loss, To rebuild the losses, To compare the losses, For consistency loss, This is the loss weighting coefficient.

10. A video analysis and early warning method for violations in open kitchen scenarios as described in claim 1, characterized in that, The method of obtaining deviation based on the trained dual-channel encoder model and triggering an early warning based on the deviation also includes: extracting features from all normal video segments of the normal video data in the region using the trained first-channel encoder, calculating the mean and covariance matrix of the feature distribution, and establishing a normal mode basis based on the mean and covariance matrix; setting a fixed duration window for the real-time video stream. Extract real-time video segments within a fixed duration window, and extract window features from the real-time video segments using the trained first encoder. The deviation is obtained based on the window features and the normal mode base. Define dynamic early warning thresholds ,when An alert is triggered at any time.

Images