Food business operator abnormal behavior detection method and system based on visual detection

Abstract

This invention relates to the technical field of visual inspection, and discloses a method and system for detecting abnormal behavior in food service establishments based on visual inspection. The method includes: identifying multiple operation markers in each functional area based on panoramic image recognition. These operation markers represent the operation area and operation content. Based on each operation marker, the corresponding operation location, and adjacent environmental images, corresponding abnormal operation events are determined, achieving visual inspection of multiple functional areas to improve the accuracy of abnormal operation events. The abnormality level of each abnormal operation event is determined based on multiple abnormal behaviors, the functional type of the functional area, and the corresponding operator. Based on the abnormality level of each abnormal operation event, the corresponding operation area, and the distribution map of the food service establishment, a dynamic monitoring map of the abnormality of the food service establishment is constructed, improving the accuracy of the dynamic monitoring map and dynamically optimizing the visual monitoring capabilities of the food service establishment.

Description

Technical Field

[0001] This invention relates to the field of visual inspection technology, and in particular to a method and system for detecting abnormal behavior in food business premises based on visual inspection. Background Technology

[0002] With increasingly stringent food safety regulations, video surveillance systems have been widely used in the daily supervision of food establishments (such as restaurant kitchens and food processing workshops). However, existing monitoring methods typically use independent cameras to capture images at fixed locations. The lack of logical connections and coordination mechanisms between these cameras often results in only acquiring partial and fragmented image information. It is difficult to effectively fuse the perspectives of different cameras based on the distribution map of the food establishment, which makes it impossible to construct a panoramic view covering the entire functional area. This affects the accuracy of abnormal operation events and reduces the accuracy of dynamic monitoring maps of abnormal situations in food establishments. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art. This invention provides a method and system for detecting abnormal behavior in food business premises based on visual detection.

[0004] This invention provides a method for detecting abnormal behavior in food service establishments based on visual detection, comprising: identifying multiple functional areas based on the distribution map of the food service establishment; triggering visual detection of corresponding camera groups based on the multiple functional areas to acquire multiple images of each functional area; determining a panoramic view of the functional area based on the synthesis of the multiple images; identifying multiple operation markers in each functional area based on the recognition of the panoramic view, the operation markers presenting the operation area-operation content; determining corresponding abnormal operation events based on each operation marker, the corresponding operation position, and adjacent environmental images; marking multiple abnormal operation events in the same functional area; identifying multiple abnormal behaviors based on the recognition of each abnormal operation event; determining the abnormality level of each abnormal operation event based on the multiple abnormal behaviors, the functional type of the functional area, and the corresponding operator; constructing an abnormal monitoring dynamic map of the food service establishment based on the abnormality level of each abnormal operation event, the corresponding operation area, and the distribution map of the food service establishment, the abnormal monitoring dynamic map outputting multiple abnormal monitoring areas and triggering corresponding abnormal warning signals to dynamically optimize the visual monitoring capabilities of the food service establishment.

[0005] This invention provides a visual detection-based abnormal behavior detection system for food service establishments, which is applied to the aforementioned visual detection-based abnormal behavior detection method for food service establishments. The system includes: a visual detection module, used to identify multiple functional areas based on the distribution map of the food service establishment; triggering visual detection of corresponding camera groups based on the multiple functional areas to acquire multiple images of each functional area; and determining a panoramic view of the functional area based on the synthesis of the multiple images; and an image recognition module, used to determine multiple operation markers in each functional area based on the recognition of the panoramic view, wherein the operation markers present... The operation area-operation content module determines corresponding abnormal operation events based on each operation marker, corresponding operation location, and adjacent environmental images. The abnormality level module is used to mark multiple abnormal operation events in the same functional area, identify multiple abnormal behaviors based on the identification of each abnormal operation event, and determine the abnormality level of the abnormal operation event based on multiple abnormal behaviors, the functional type of the functional area, and the corresponding operator. The visual monitoring capability module is used to construct an abnormal monitoring dynamic map of the food business premises based on the abnormality level of each abnormal operation event, the corresponding operation area, and the distribution map of the food business premises. This abnormal monitoring dynamic map outputs multiple abnormal monitoring areas and triggers corresponding abnormal early warning signals to dynamically optimize the visual monitoring capabilities of the food business premises.

[0006] Compared with the prior art, the beneficial effects of the present invention are: (1) Based on the identification of the distribution map of food business premises, multiple functional areas are identified. The visual detection of the corresponding camera group is triggered according to the multiple functional areas to obtain multiple images of each functional area. The panoramic view of the functional area is determined based on the synthesis of multiple images. In each functional area, multiple operation marks are determined based on the identification of the panoramic view. The operation marks present the operation area-operation content. The corresponding abnormal operation events are determined according to each operation mark, the corresponding operation position and the adjacent environmental image. The visual detection of multiple functional areas is realized, and the panoramic view of each functional area is further improved to improve the accuracy of abnormal operation events. (2) Determine the abnormal level of the abnormal operation event based on multiple abnormal behaviors, functional types of functional areas and corresponding operators; construct an abnormal monitoring dynamic map of food business premises based on the abnormal level of each abnormal operation event, the corresponding operation area and the distribution map of food business premises, and trigger the corresponding abnormal warning signal to dynamically optimize the visual monitoring capability of food business premises, further control the abnormal level of abnormal operation events, realize the overall consideration of the abnormal level of each abnormal operation event, the corresponding operation area and the distribution map of food business premises, improve the accuracy of the abnormal monitoring dynamic map of food business premises, and dynamically optimize the visual monitoring capability of food business premises. (3) Based on multiple early warning contents, the monitoring weight is determined and the monitoring weight is dynamically changed with the preset time period to iterate the abnormal monitoring area multiple times. This realizes that the monitoring resources can be adaptively adjusted according to the real-time evolution of the risk situation. Through the dynamic detection of real-time abnormal behavior and its location, the visual monitoring capability and computing resource utilization efficiency of food business premises are dynamically optimized from the execution dimension. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating the abnormal behavior detection method for food service establishments based on visual detection in an embodiment of the present invention. Figure 2 This is a flowchart illustrating step S11 of the visual detection-based abnormal behavior detection method for food service establishments in this embodiment of the invention. Figure 3 This is a flowchart illustrating step S12 of the visual detection-based abnormal behavior detection method for food service establishments in this embodiment of the invention. Figure 4 This is a flowchart illustrating step S13 of the visual detection-based abnormal behavior detection method for food service establishments in this embodiment of the invention. Figure 5 This is a flowchart illustrating step S14 of the visual detection-based abnormal behavior detection method for food service establishments in this embodiment of the invention. Figure 6 This is a schematic diagram of the structural composition of a visual detection-based abnormal behavior detection system for food service establishments according to an embodiment of the present invention. Detailed Implementation

[0008] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0009] Please see Figures 1 to 6 A visual detection-based method for detecting abnormal behavior in food service establishments is presented, applicable to visual detection scenarios. The method includes: Step S11: Based on the identification of the distribution map of food business premises, multiple functional areas are identified. Visual detection of the corresponding camera groups is triggered according to the multiple functional areas to obtain multiple images of each functional area. A panoramic view of the functional area is determined by synthesizing the multiple images. Step S12: In each functional area, multiple operation markers are determined based on the recognition of the panoramic image. The operation markers present the operation area and operation content. The corresponding abnormal operation events are determined according to each operation marker, the corresponding operation position and the adjacent environmental image. Step S13: Mark multiple abnormal operation events in the same functional area, determine multiple abnormal behaviors based on the identification of each abnormal operation event, and determine the abnormality level of the abnormal operation event based on the multiple abnormal behaviors, the functional type of the functional area and the corresponding operator. Step S14: Based on the abnormality level of each abnormal operation event, the corresponding operation area, and the distribution map of the food business premises, construct an abnormal monitoring dynamic map of the food business premises. This abnormal monitoring dynamic map outputs multiple abnormal monitoring areas and triggers corresponding abnormal early warning signals to dynamically optimize the visual monitoring capabilities of the food business premises.

[0010] refer to Figure 2 In step S11, the specific steps are as follows: S111: Collect the distribution map of food business premises, perform image segmentation on the distribution map of food business premises, and output multiple functional areas as the segmentation semantics change during the segmentation process. The multiple functional areas are the kitchen cooking area, the preliminary processing area, the washing and disinfection area, and the food preparation area. S112: In each functional area, mark the regional space of each functional area, and determine multiple cameras in the traversal of the regional space to construct a corresponding camera group. At the same time, the camera group performs multi-position visual detection on the functional area and acquires multiple images of each functional area. S113: In multiple images, multiple features are determined based on image recognition of each image, corresponding weight content is determined according to multiple images and corresponding semantics, and the synthesis of multiple images is triggered along each weight content, the corresponding image and the corresponding image contour to determine the panoramic view of the functional area.

[0011] In the embodiments of this application, a distribution map of food business premises is collected, and the distribution map of food business premises is segmented. During the segmentation process, multiple core functional areas are output as the segmentation semantics change. The multiple core functional areas are the kitchen cooking area, the preliminary processing area, the washing and disinfection area, and the food preparation area.

[0012] At this point, to obtain a distribution map of food business premises, in order to adapt to the visual recognition network, the vectorized distribution map needs to be rasterized and converted into a standard tensor format; geometric normalization is performed to eliminate spatial distortion caused by inconsistent shooting angles or drawing scales; at the same time, edge detection operators are used to enhance the edge response of structural features such as walls and passages, and a high signal-to-noise ratio input data stream is constructed.

[0013] The model extracts contextual features of images through multi-scale atrous spatial pyramid pooling to capture regions of different sizes. The model performs pixel-by-pixel classification on the input distribution map and clusters pixels with similar semantic attributes into the same instance mask based on the texture features, shape features and spatial adjacency of the pixels.

[0014] During the segmentation process, the system monitors the rate of change of semantic vectors in real time. When the semantic category of adjacent pixels changes (e.g., from "cooking area" feature to "channel" feature), the system determines that the position is the semantic boundary of the functional region. Morphological operations (such as closing operation) are used to smooth the edge of the region, remove small noise areas, and finally output a set of closed and connected polygonal functional regions.

[0015] Specifically, in the catering center, the system read the second-floor floor plan of the center; the image preprocessing module identified the thick lines representing solid walls and the blank areas representing passageways in the drawing, and converted them into an RGB image matrix with a resolution of 2048x2048 pixels.

[0016] In a preferred embodiment, the distribution map includes a pre-defined functional area labeling layer. This layer clearly defines core functional areas (such as "kitchen cooking area," "preliminary processing area," "washing and disinfection area," and "food preparation area") and non-core areas (such as "passageways" and "walls") in the form of closed polygons, each labeled with a corresponding semantic tag. By parsing this labeling layer, the system directly extracts the set of polygon vertex coordinates and their semantic tags for each functional area, thereby determining the spatial extent and attributes of multiple functional areas without loss of quality.

[0017] In another preferred embodiment of the present invention, the system automatically parses the original distribution map of food business premises to construct the logical boundaries of functional areas: First, it uses object detection and semantic segmentation models to identify basic elements in the distribution map, including structural features such as walls, passageways, and corridors, as well as legends or outlines of equipment and facilities such as stoves, sinks, workbenches, and disinfection cabinets, forming a structural semantic layer and an equipment semantic layer; then, the system loads a preset "functional zoning knowledge base," which defines the typical equipment composition, spatial association rules, and hygiene isolation requirements of each functional area. Based on this knowledge base, the system performs spatial density clustering of the identified equipment and combines it with its semantic categories to... Equipment clusters are mapped as candidate functional areas. Then, taking the spatial distribution of each equipment cluster as the core, and comprehensively considering the standard operating radius, personnel movement width, and adjacent structural boundaries, the initial logical polygon of the area is generated through convex hull calculation or region growing algorithm. Finally, the system automatically resolves conflicts and optimizes boundaries for overlaps or gaps between adjacent areas according to priority rules in the knowledge base. For example, it forces a physical or logical separation between the "raw food processing area" and the "cooked food preparation area," thereby outputting a set of spatially reasonable functional area division results that comply with food safety regulations and match the physical layout. This division result includes the spatial coordinates of pixels and their corresponding semantic attributes.

[0018] For example, when analyzing preprocessed images of a catering center, dense heat points of stoves were identified in the upper left corner of the image (assuming the drawing contains equipment annotations or textures based on known equipment layouts), and these pixels were clustered and labeled as the "back kitchen cooking area"; large areas of sink and drainage ditch textures were identified in the right side area, and these were segmented and labeled as the "washing and disinfection area"; the central area was labeled as the "rough processing area" based on the texture of the long strip-shaped worktable.

[0019] When processing the boundary between the food preparation area and the passageway in the catering center, the system detected a sudden change in semantic feature from "food preparation station" to "passageway" and immediately locked this coordinate as the boundary. After edge smoothing, the system finally output four distinct sets of region coordinates: Region A (x1, y1): kitchen cooking area; Region B (x2, y2): preliminary processing area; Region C (x3, y3): washing and disinfection area; Region D (x4, y4): food preparation area. These four regions are not only graphically divided, but each is also bound to a corresponding semantic label.

[0020] Furthermore, in each functional area, the regional space of each functional area is marked, and multiple cameras are determined during the traversal of the regional space to construct a corresponding camera group. At the same time, the camera group performs multi-position visual detection on the functional area and acquires multiple images of each functional area, thus introducing multiple images of each functional area.

[0021] At this point, based on the semantic segmentation mask output by S111, the system transforms unstructured pixel regions into structured spatial geometric descriptions. By calculating the minimum bounding rectangle or polygon convex hull of the pixel set, the two-dimensional spatial range (ROI, RegionofInterest) of each functional region in a unified world coordinate system is determined. Simultaneously, region attribute labels are established to associate and map physical spatial features (such as area, aspect ratio, and entrance / exit coordinates) with functional semantics (such as cooking area and washing / disinfection area), generating unique region spatial identifiers.

[0022] The system maintains the pose information (including intrinsic parameter matrices and extrinsic parameter rotation and translation vectors) of all registered cameras in 3D space, and constructs the view frustum of each camera through ray casting. The system traverses the spatial region such as Zone_ID_01 and calculates the spatial positional relationship between the vertices of the region polygon and the view frustum of each camera. The system uses the view frustum inclusion test to select the set of cameras whose field of view (FOV) intersects or covers the target functional area space.

[0023] Based on spatial intersection relationships, the selected candidate cameras are instantiated into a logical "camera group". The system sends a synchronization trigger signal or frame synchronization command to the group to control the cameras at different locations to acquire images according to a preset time resolution or event triggering mechanism. During this process, the system performs spatiotemporal alignment of the acquired data based on the overlapping fields of view of the cameras to ensure the consistency of the acquired multiple image streams in terms of timestamps and spatial reference systems, forming a comprehensive perception matrix of the target area.

[0024] Specifically, in the catering center's system, the "back kitchen cooking area" is marked as Zone_ID_01; the system calculates the world coordinate range of this area as (X: 10.5m-25.3m, Y: 5.2m-12.8m) and marks its attributes as "high temperature, high dynamic range, and high risk of obstruction". This spatial marking not only delineates the physical boundaries but also provides a benchmark reference system for subsequent calculations of camera coverage.

[0025] When the system traversed the spatial coordinates of Zone_ID_01 (kitchen cooking area), it found that the view frustum of Cam_A (installation height 3.5 meters, downward angle 45 degrees) deployed on the north wall covered 70% of the area; while the view frustum of Cam_B (installation height 2.8 meters, horizontal viewing angle) deployed on the column covered the southeast blind spot of the area; the system determined that the effective fields of view of these two cameras intersected with the cooking area space, and therefore locked them as candidate nodes.

[0026] The system constructs a monitoring group for Zone_ID_01, Group_Cooking_Zone={Cam_A,Cam_B}. When the "lunch preparation peak period" arrives, the system triggers this group to enter high-frequency detection mode. Cam_A is responsible for macroscopic monitoring of the chef's overall operation flow, while Cam_B is responsible for supplementing the monitoring of the condiment station operation in the corner. The two cameras simultaneously acquire image streams and transmit them back to the processing unit in real time, thereby obtaining a multi-view image dataset of the kitchen cooking area without blind spots, providing raw data support for subsequent panoramic image synthesis and behavior analysis.

[0027] Therefore, in multiple images, multiple features are determined based on image recognition of each image, and corresponding weight content is determined according to multiple images and corresponding semantics. The synthesis of multiple images is triggered along each weight content, the corresponding image, and the corresponding image contour to determine the panoramic view of the functional area. This approach is compatible with the overall consideration of image recognition of each image and ensures the accuracy of multiple features.

[0028] At this point, the system receives multiple image streams captured by a group of cameras and uses computer vision feature extraction operators (such as ORB, SIFT, or deep convolution features) to detect local feature points in each frame of the image. At the same time, an image quality assessment (IQA) module is introduced to calculate the sharpness, contrast, and signal-to-noise ratio (SNR) of each image. In addition, the system will also perform semantic saliency detection to identify whether there are key operational objects (such as personnel, equipment, and ingredients) in the image. These key features will serve as the basis for subsequent weight allocation.

[0029] Based on the extracted image features and corresponding semantic information, the system constructs a dynamic weighted matrix. The calculation logic of the weights follows the dual standards of "semantic importance" and "image quality": Semantic weight: If the image feature belongs to a high-priority semantic (such as "open flame operating table" or "knife operating area"), it is assigned a high weight; if it belongs to the background or non-core area (such as "ceiling" or "empty passageway"), it is assigned a low weight; Quality weight: Combining sharpness and lighting conditions, image patches containing high-confidence features are given higher fusion weights; Finally, a weight map corresponding to each image is generated, which reflects the contribution of the image to the panoramic synthesis.

[0030] The system performs multi-view geometric registration based on weight content and feature point matching results; it calculates the homography matrix or fundamental matrix between images using feature matching to establish spatial transformation relationships between images; in the fusion stage, the system performs Laplacian pyramid fusion or multi-band fusion along the contour edges of the images and performs weighted averaging of pixel values ​​according to pre-calculated weight content; in particular, for high-weight regions, the system will forcibly lock the image features of that region to suppress the blurring or ghosting effects of other low-weight images in that region; and at the contour boundaries, smoothing techniques such as gradient domain Poisson editing are used to eliminate stitching seams, ultimately generating a seamless panoramic image of functional areas with key information highlighted.

[0031] Specifically, for the input image of Zone_ID_01 (kitchen cooking area), the system extracted 500 feature points from Cam_A, including the edge of the stove and the corner of the stainless steel shelf; and extracted 300 feature points from Cam_B, including the outline of the condiment bottle and the lines of the floor tiles. At the same time, the quality assessment module detected that the image of Cam_A had a slightly lower local clarity score due to the influence of oil fumes, while the preparation area captured by Cam_B had clear texture and strong feature saliency.

[0032] When analyzing images of the kitchen in the catering center, the system determined that the "wok and stove" area covered by Cam_A was a high-risk core operation area and assigned it a semantic weight of 0.9. Although its image was slightly blurry, it still maintained a high weight due to its irreplaceability. The "corner preparation station" area covered by Cam_B had relatively low semantic importance and was assigned a semantic weight of 0.6. However, due to its high image clarity, its quality weight was adjusted to 0.8. The system ultimately determined that when synthesizing the panoramic image, the stove action details in Cam_A should be preserved first, while the high-definition texture of Cam_B should be used in the overlapping areas.

[0033] When synthesizing the panoramic view of the cooking area in the catering center, the system aligns the images of Cam_A and Cam_B according to the stove coordinates. In the overlapping area of ​​the two images (i.e., the central aisle of the cooking area), the system mainly adopts the high-definition texture data of Cam_B to fill the details of the floor tiles based on the weight map. In the non-overlapping edge areas, the system feathers the contour edges of Cam_A and Cam_B to eliminate the stitching lines caused by differences in lighting. A high-quality panoramic monitoring image of the kitchen cooking area is generated, which includes both the unique stove operation perspective of Cam_A and the clear details of Cam_B, providing a standardized panoramic input for the accurate identification of abnormal behavior in step S12.

[0034] refer to Figure 3 In step S12, the specific steps are as follows: S121: Real-time monitoring of each functional area, multi-target recognition of the panoramic image, and identification of the interaction between multiple key points of the operator's torso and object features during the recognition process, and marking them as corresponding operation markers. These operation markers cover the operation type, the corresponding operation area, and the operation range to present the operation area-operation content. S122: Based on the tracing of the operation mark, determine the corresponding multiple operation boundary points, and compare the multiple operation boundary points with the qualified area box marked by the panoramic image to determine multiple abnormal operation points, each of the multiple abnormal operation points being outside the qualified area; S123: Multiple abnormal operation points and the operation positions corresponding to the operation markers are subjected to multi-factor analysis in the time series dimension. During the analysis, the abnormal content of multiple abnormal operation points is determined. At the same time, the corresponding environmental image is determined based on the environmental traversal of the operation position corresponding to the operation marker. The abnormal content of the environment is determined based on the recognition of the environmental image. The corresponding abnormal operation event is determined based on the abnormal content of the environment and the abnormal content of multiple abnormal operation points.

[0035] In the embodiments of this application, each functional area is monitored in real time, multi-target recognition is performed on the panoramic image, and during the recognition process, the interaction operations of multiple key points of the operator's torso and object features are determined and marked as corresponding operation markers. The operation markers cover the operation type, the corresponding operation area and the operation range to present the operation area-operation content, thus introducing the operation area-operation content.

[0036] At this point, the system maintains real-time frame stream processing of the panoramic images of each functional area generated by S113. By deploying a multi-object detection network (such as YOLOv8 or Faster R-CNN), convolutional feature extraction is performed on each panoramic image frame. The detection head performs multi-scale prediction on the feature pyramid and outputs the class confidence and bounding boxes of all targets of interest in the scene. The system removes redundant boxes according to the non-maximum suppression (NMS) method and retains only high-confidence target instances. The detection categories include: personnel (Chef / Staff), food (Vegetable / Meat), tools (Knife / Cleaver), equipment (Stove / Pot), and critical facilities (CuttingBoard / Sink).

[0037] The system applies a top-down pose estimation network (such as HRNet or PoseResNet) to the detected human targets (such as Obj_01); it crops and scales the region of interest (ROI) of the human to a standard input size, and predicts the spatial location of human key points through heatmap regression; the system focuses on extracting key points of the torso such as shoulders, elbows, and wrists to accurately characterize the kinematic chain structure of the upper limbs; the system maps the coordinates of the key points back to the unified world coordinate system of the panoramic image to achieve spatiotemporal alignment of the key points with the detected objects (Obj_02, Obj_03).

[0038] The system calculates the spatial relationship metric between human keypoints and object detection boxes. It determines whether physical contact or effective interaction has occurred by judging whether the keypoint falls inside the object's bounding box (Point-in-Box test) and whether the Euclidean distance between the keypoint and the object's center is less than a preset interaction threshold (InteractionThreshold). For temporal data, the system uses optical flow or keypoint trajectory prediction to analyze whether the keypoint's motion vector relative to the object points towards the object's center (ApproachVector) or produces relative displacement. If the conditions of spatiotemporal proximity and motion consistency are met, it is determined to be a valid interaction operation. The system encapsulates interaction instances into structured "OperationMarkers," which include: operation type (such as "cutting," "stir-frying," "washing"), operation area (the spatial range where the interaction occurs, usually defined by the keypoint convex hull or the object-keypoint joint bounding box), and operation content (the specific object category involved).

[0039] Specifically, in the real-time panoramic stream of the preliminary processing area of ​​the catering center, the target detection module identified multiple targets in the frame at time T=10:05:32: instance Obj_01 (category: Chef, confidence: 0.98, bounding box: [x1=1200, y1=800, x2=1400, y2=1500]), instance Obj_02 (category: Cleaver, confidence: 0.95, bounding box: [x1=1250, y1=950, x2=1350, y2=1050]), and instance Obj_03 (category: Cutting_Board, confidence: self-check passed, bounding box: [x1=1200, y1=900, x2=1400, y2=1100]). These detection results constitute the basic data layer for subsequent interactive analysis.

[0040] For Obj_01 (the chef), the pose estimation network outputs the pixel coordinates of the right wrist key point P_R_Wrist in the panoramic image as [x = 1300, y = 1000]; at the same time, the coordinates of the left wrist key point P_L_Wrist are [x = 1220, y = 980]; the system finds that the coordinates of P_R_Wrist coincide exactly with the center point [x = 1300, y = 1000] of the object Obj_02 (Cleaver), and there is a spatial association between P_L_Wrist and the upper left corner area [x = 1200, y = 900] of the object Obj_03 (Cutting_you). It is initially determined that there may be an interaction relationship of holding a knife and pressing ingredients.

[0041] The system calculates and finds that P_R_Wrist continuously falls within the bounding box of Obj_02 (Cleaver), and its motion vector is perpendicular to the surface of Obj_03 (Cutting_Board), which conforms to the mechanical characteristics of the cutting action; at the same time, P_L_Wrist shows a periodic pressing pattern at the edge of Obj_03; based on this, the system generates an operation marker Marker_001 at time T.

[0042] Furthermore, based on the traceability of the operation marker, a corresponding number of operation boundary points are determined, and the multiple operation boundary points are compared with the qualified area box marked in the panoramic image to determine multiple abnormal operation points. Each of the multiple abnormal operation points is outside the qualified area, which takes into account the overall consideration of the traceability of the operation marker and ensures the accuracy of the corresponding multiple operation boundary points.

[0043] At this time, the system locks the specific "operation marker" generated by S121, and takes the timestamp of this marker as the center, and traces back a preset length of time series window (such as N frames before and after or a t-second time window) forward and backward; within this time window, the system extracts the spatial coordinates of all relevant instances constituting this operation; specifically, the system combines the detection frames of the person's torso key points (such as wrists, elbows) participating in the interaction and the interactive objects (such as knives, ingredients), and calculates the union area or convex hull of these elements; dynamically samples the spatial area within this time window, and extracts multiple discrete coordinate points representing the maximum influence range of this operation, that is, "operation boundary points". These boundary points not only include the instantaneous position of the operation, but also cover the sweeping trajectory during the operation process.

[0044] The system calls the spatially logical constraint data pre-configured in this functional area (such as the rough processing area). These data are usually stored in the form of masks or vector graphics, and define the "qualified area box" allowing specific types of operations; the qualified area box is generated based on the physical facility layout (such as the size of the operating table, the width of the aisle) and food safety specifications (such as raw and cooked food zoning, isolation of clean areas); the system maps these logical constraints to the panoramic Figure 1In the world coordinate system, load its geometric boundary parameters (such as polygon vertex coordinates, inner radius, or repulsion zone boundary).

[0045] The system performs point-in-polygon (PIP) testing or ray casting; it calculates the spatial topological relationship between each extracted "operation boundary point" and the "qualified region box"; the system establishes judgment logic: if the operation boundary point is located on or inside the boundary of the qualified region box, it is judged as a "compliant point"; if the operation boundary point is located in the geometric complement of the qualified region box (i.e., external space), it is immediately marked as an "abnormal operation point"; the system can also calculate the distance offset from the abnormal point to the qualified region to quantify the severity of the violation.

[0046] Specifically, in the preliminary processing area of ​​the catering center, the system traces the operation marker Marker_001 (cutting and preparation operation). Within the time window from T=10:05:30 to T=10:05:35, the system extracts the motion trajectory of the chef's right wrist key point (P_R_Wrist) and the "knife" object box (Bbox_Cleaver). The system analysis found that although the wrist mainly moves above the cutting board, when swinging the knife, the movement range of the knife tip exceeds the edge of the cutting board. Based on this, the system extracts four key discrete coordinate points as operation boundary points: Point_A (knife tip, x=1150, y=950), Point_B (wrist, x=1300, y=1000), Point_C (knife end, x=1350, y=1020), and the extreme point Point_D (x=1140, y=980) on the motion sweep trajectory. These points together define the actual physical boundary of the cutting and preparation operation in space.

[0047] The configuration regulations for the preliminary processing area of ​​the catering center stipulate that "meat thawing and cutting" operations are strictly limited to the red stainless steel special operating table and are strictly prohibited from extending into the adjacent blue washing pool area; the system loads the geometric definition of the "qualified area box": its coordinate range is [x_min=1200,y_min=900,x_max=1400,y_max=1100], which is a rectangular space constraint. Any legal meat cutting operation must have its operation boundary point strictly located within this rectangular enclosure.

[0048] The system compares the four extracted operation boundary points with the qualified area box [1200, 900, 1400, 1100]. Point_B (1300, 1000) and Point_C (1350, 1020) are both located inside the rectangle and are deemed compliant. However, the X coordinates (1150, 1140) of Point_A (1150, 950) and Point_D (1140, 980) are both smaller than the minimum X coordinate (1200) of the qualified area box. According to the ray casting method, these two points are located outside the qualified area box (operating table) and actually fall within the coordinate range of the adjacent "cleaning pool". Therefore, the system marks Point_A and Point_D as abnormal operation points, which indicates that the operator swung the knife too much during the cutting process, exceeding the compliant operating table surface and constituting a spatial boundary violation.

[0049] Therefore, multiple abnormal operation points and the operation positions corresponding to the operation markers are analyzed in a multi-factor time-series dimension. During the analysis, the abnormal content of multiple abnormal operation points is determined. At the same time, the corresponding environmental image is determined based on the environmental traversal of the operation position corresponding to the operation marker. The abnormal content of the environment is determined based on the recognition of the environmental image. The corresponding abnormal operation event is determined based on the abnormal content of the environment and the abnormal content of multiple abnormal operation points. This approach takes into account both the abnormal content of the environment and the abnormal content of multiple abnormal operation points, ensuring the accuracy of the corresponding abnormal operation event.

[0050] At this point, the system performs continuous verification of the abnormal operation points locked in S122 in the time dimension. By using Kalman filtering or optical flow tracing, the system calculates the motion trajectory and retention time of the abnormal operation points in the continuous frame sequence. The system analyzes the spectral characteristics of the "boundary crossing" behavior to distinguish between "instantaneous through-traffic interference" (such as rapid arm swing) and "continuous violation operation" (such as prolonged stay in non-compliant areas). In addition, the system analyzes the action semantics corresponding to the abnormal points (such as "hovering", "pressing", "cutting") by combining the operation type in the operation mark. Based on the time-accumulated frame number and action semantic features, the system transforms the original coordinate points into specific abnormal content descriptions, such as "continuous boundary crossing operation" or "staying in non-compliant areas".

[0051] The system constructs a dynamic region of interest (ROI) centered on the abnormal operation point. Based on the spatial distribution of the abnormal operation point, the system extracts an image patch (i.e., a circle or bounding box extension region with radius r) from the panoramic image, which is called the "environmental image". The system uses a semantic segmentation network to perform pixel-level classification of the environmental image to identify the functional facility category to which the area belongs (e.g., "washing pool", "trash can", "stove", "ground"). At the same time, the system also identifies whether there are specific pollutants or sensitive targets (e.g., "stagnant water", "stains", "unprocessed food") in the environmental area to extract environmental context features.

[0052] The system constructs a multimodal decision fusion model, logically associating the determined "abnormal operation point content" (behavioral dimension) with the determined "environmental abnormal content" (environmental dimension). The system internally maintains a food safety risk rule base, such as "raw food processing area contacting cooked food area", "cleaning tools contacting food", and "knives used across areas". The system determines the final abnormality type by calculating the vector similarity between behavioral semantics and environmental semantics or by matching preset violation rule templates. When behavioral and environmental features jointly trigger a high-risk rule, the system generates a structured "abnormal operation event", which includes event level, triggering subject, behavioral description and environmental risk factors.

[0053] Specifically, in the preliminary processing area of ​​the catering center, the system performed a time-series backtracking on the abnormal operation point Point_A (blade tip position, coordinate x=1150). The analysis showed that within 5 seconds from T=10:05:30 to T=10:05:35, Point_A remained outside the qualified area box (x=1200) and did not exhibit rapid crossing characteristics, but was accompanied by high-frequency up-and-down reciprocating motion (spectral characteristics consistent with "cutting" action). Based on this, the system determined that this was not an accidental mis-touch, but a continuous and intentional out-of-bounds operation, and identified the abnormal content as: "The operator continuously performed cutting-type actions in the non-compliant area (above the cleaning pool).

[0054] For the out-of-bounds location of Point_A, the system extracted a 200x200 pixel neighborhood image patch centered on Point_A in the panoramic image as the environmental image. Using a high-precision semantic segmentation model, the system identified this area and determined that the pixel region mainly belonged to the semantic mask of the "washing pool" category, with a confidence level as high as 0.96. Further object detection showed that the washing pool area also contained the features of "stagnant water" and "vegetable baskets to be washed". Therefore, the system extracted the abnormal environmental content as: "The violation occurred in a washing / draining area with a low level of hygiene control, and the environment was damp and mixed with debris."

[0055] The system inputs "behavioral characteristics: continuously performing cutting actions in non-compliant areas" and "environmental characteristics: washing pool area (accumulated water, debris)" into the fusion module; the system matches the preset food safety rules: "Knives are strictly prohibited from being used on non-dedicated workbenches (such as next to washing pools or garbage cans) to prevent cross-contamination and damage to knives"; the judgment logic indicates that the chef not only crossed the boundary, but also brought meat processing actions into the washing area, which could easily lead to meat being contaminated by sewage or knives being dropped.

[0056] refer to Figure 4 In step S13, the specific steps are as follows: S131: In the same functional area, mark multiple abnormal operation events and perform multi-channel analysis on the multiple abnormal operation events simultaneously to identify multiple abnormal operation events that are close in time and space in the same functional area. Aggregate the multiple abnormal operation events to form corresponding continuous abnormal content. Based on the identification of continuous abnormal content, determine multiple abnormal behaviors. S132: Collect preset risk mapping relationships, perform multi-factor matching of multiple abnormal behaviors, functional types of functional areas and risk mapping relationships, determine the corresponding risk content during the matching process, mark the impact event corresponding to the risk content, and determine the abnormality level of the abnormal operation event based on the impact event corresponding to the risk content, the corresponding operator and the job position of the operator.

[0057] In the embodiments of this application, multiple abnormal operation events are marked in the same functional area, and multi-channel analysis is performed on the multiple abnormal operation events simultaneously to determine multiple abnormal operation events that are close in time and space in the same functional area. The multiple abnormal operation events are aggregated to form corresponding continuous abnormal content. Multiple abnormal behaviors are determined based on the identification of continuous abnormal content, which is compatible with the overall consideration of continuous abnormal content identification and ensures the accuracy of multiple abnormal behaviors.

[0058] At this point, the system maintains the spatiotemporal index within the same functional area (such as the rough processing area and the cooking area), and collects all triggered discrete abnormal operation events within a preset time sliding window (SlidingTimeWindow, e.g., Δt=30 seconds). To address the heterogeneity of event descriptions, the system employs a multi-channel analysis strategy to construct feature vectors for each event. Specifically, these include: a spatial channel (extracting the 2D / 3D coordinates of the event's center point and the operation bounding box), a temporal channel (extracting the start frame, duration, and timestamp of the event), a semantic channel (extracting the operation type, object category, and reason for violation), and a subject channel (extracting the operator's ID and skeletal pose features). These feature vectors are mapped to a high-dimensional feature space, laying the foundation for subsequent similarity measurements.

[0059] The system calculates the similarity between any two anomalous operation events in a high-dimensional feature space. It primarily employs spatiotemporal clustering, combining Euclidean distance (for spatial coordinates) and temporal Hausdorff distance to calculate the "spatiotemporal proximity" between events. A threshold θST (i.e., the maximum spatial interval Dmax and the maximum time interval Tmax) is set. If the spatiotemporal distance between two events satisfies Dist(Ei,Ej)<θST, then the two events are determined to belong to the same spatiotemporal cluster. The system logically merges these highly spatiotemporally clustered discrete events, removes redundant information, and generates an aggregated composite event description. This process transforms isolated, fragmented violation points into a continuous scene description.

[0060] For the "continuous abnormal content" formed after aggregation, the system uses natural language processing (NLP) technology or a behavior recognition model based on temporal logic to perform deep semantic understanding; the system extracts key action units and state changes in the continuous content, constructs action causal chains, and identifies the intent represented by the continuous content by matching a predefined "abnormal behavior pattern library" or by using a long short-term memory network (LSTM); the system abstracts the specific scene description into standardized "abnormal behavior" category labels, thereby transforming the underlying visual perception into a high-level business logic understanding.

[0061] Specifically, in the "preliminary processing area" of the catering center, the system marked three discrete abnormal operation events within the time window from 10:05:32 to 10:05:42: Event_A (knife swings out of bounds above the washing pool), Event_B (ingredients accidentally fall into the washing pool area), and Event_C (hands are put into the washing pool area and come into contact with water). The system performed multi-channel analysis on these three events and extracted that they were all located within a narrow spatial range of coordinates [x=1140-1160, y=950-990] (spatial channel), the time difference between the occurrences was no more than 5 seconds (time channel), and the subject of all of them was Chef_01 (subject channel).

[0062] The system calculates the spatiotemporal distances of Event_A (cutting beyond the boundary), Event_B (food falling), and Event_C (contact with water). The results show that the spatial distance between Event_B and Event_C is only 0.2 meters, with a time interval of 1.5 seconds. The distance between Event_A and the other two is also less than 0.5 meters, and they occur in the same continuous action sequence. Since the spatial distance of these three events is less than the set threshold of 0.5 meters, and the time interval is less than the threshold of 10 seconds, the system determines that they belong to the same spatiotemporal cluster. The system aggregates these three events, removes the intermediate instantaneous states, and forms a continuous scene description: "After the operator used the knife to cut beyond the boundary, the food fell into the contaminated area and was touched by bare hands."

[0063] The system performs semantic analysis on the aggregated continuous scene description ("out-of-bounds operation caused food to fall and be touched by bare hands"); the system identifies a logical chain in the sequence of "indiscriminate handling of raw and cooked food / clean and dirty food" (knife enters the washing area) and "improper handling of contamination" (food not isolated after contact with water); after matching with the abnormal behavior pattern library, the system determines that the continuous content corresponds to a typical serious violation category; therefore, the system finally identifies the abnormal behavior as "violation of cross-contamination and secondary contamination of food". This identification result not only includes the action itself, but also qualitatively defines its food safety risk attribute, providing a core basis for subsequent level assessment.

[0064] Furthermore, a pre-defined risk mapping relationship is collected, and multiple abnormal behaviors, functional types of functional areas, and risk mapping relationships are matched using multiple factors. During the matching process, the corresponding risk content is determined, and the impact event corresponding to the risk content is marked. Based on the impact event corresponding to the risk content, the corresponding operator, and the operator's job position, the abnormality level of the abnormal operation event is determined. This approach takes into account the overall consideration of the impact event corresponding to the risk content, the corresponding operator, and the operator's job position, ensuring the accuracy of the abnormality level of the abnormal operation event.

[0065] At this point, the system retrieves a pre-built "risk mapping relationship" graph from the food safety knowledge base. This graph is a multi-dimensional tensor structure that defines the mapping logic of [abnormal behavior type] × [functional area attribute] × [risk category] → [risk impact parameter]. The system extracts the "abnormal behavior" vector identified by S131 and the "functional type" label of the current functional area (such as the preliminary processing area, cooking area, etc.), and inputs them into the multi-factor matching engine. The engine retrieves the best match in the mapping graph through semantic similarity calculation and rule reasoning, thereby determining the specific "risk content" triggered by the abnormal behavior in a specific area, including the risk type (such as biological, chemical, physical) and its potential hazard probability.

[0066] Based on the identified risk content, the system further retrieves the associated consequence knowledge base and marks the specific "impact event" corresponding to the risk if it gets out of control. This is usually a physical description at the business logic level, transforming the abstract risk into a specific food safety incident scenario. The system maps the risk content to the most likely actual consequences through causal relationship chains and calculates the severity index of the consequences. This step aims to transform "what might happen" into "what will actually happen," providing a consequence dimension basis for the rating assessment.

[0067] The system integrates the above factors to construct a final anomaly level assessment model, which uses a weighted summation method: Level = f(Risk_Severity, Impact_Magnitude, Role_Weight, History_Penalty). The system obtains the "operator" ID bound to the anomaly event and queries the HR database to obtain its "job position" attribute (e.g., junior chef, supervisor, intern, etc.). Different positions have different "responsibility coefficients"—the higher the position or the more critical its role at the critical control point (CCP), the higher the overall anomaly level caused by its violation. The system calculates the final "anomaly level" (e.g., Level I - general violation, Level II - serious violation, Level III - major accident hazard) based on the severity of the risk content, the scale of the impact event, and the operator's job weight.

[0068] Specifically, the system identified the abnormal behavior as "illegal operation involving cross-contamination and secondary contamination of food ingredients," and it occurred within the "rough processing area" (a high-risk area for raw food handling). The system matched these two input items with a risk mapping map. One rule in the map clearly states: {Behavior: Contact with unclean areas, Area: Raw food processing area} → {Risk type: Bacterial cross-contamination, Probability of harm: Extremely high}. Based on this, the system determined the risk content of the event as: "High risk of microorganisms (such as Salmonella and E. coli) transferring from the environment to food ingredients."

[0069] Regarding the risk of "microbial cross-contamination," the system traverses the consequences knowledge base and, combined with the business characteristics of the catering center as a centralized food supply unit, marks the corresponding "impact event" as "mass food poisoning incident." The system determines the severity index of this impact event to be the highest (Critical), because once it occurs, it will affect a large number of diners and involve significant public health and safety responsibilities.

[0070] The system identified the person who committed the violation as the "on-duty head chef" (position: key personnel responsible for key positions). In the job weight table of the catering center, the "head chef's" responsibility coefficient is 1.5 (1.0 for ordinary employees), because their behavior has a demonstrative effect on kitchen standards and they bear direct responsibility for food safety. The system comprehensively calculated: Risk severity: extremely high (microbial contamination); Impact event scale: extremely large (mass food poisoning); Job responsibility coefficient: 1.5 (head chef); The system ultimately determined the abnormality level of this abnormal operation event to be "Level 1 (Extremely Serious)", which means that this event is not only an operational error, but also a major management accident hazard, and the highest level of physical blockade (such as work stoppage and rectification) and reporting mechanism must be triggered immediately.

[0071] refer to Figure 5In step S14, the specific steps are as follows: S141: In the distribution map of food business premises, load each abnormal operation event along the corresponding loading path into the distribution map of food business premises, and note the corresponding abnormal level in each abnormal operation event. Based on the combination of the abnormal level of each abnormal operation event and the corresponding operation area, determine the corresponding abnormal heat layer, and combine visualization technology to construct an abnormal monitoring dynamic map of food business premises. S142: Real-time monitoring of anomaly monitoring dynamic graph, and output real-time anomaly location based on the anomaly monitoring dynamic graph. Determine the corresponding anomaly monitoring area based on multiple real-time anomaly locations and corresponding anomaly operation content. Determine the corresponding real-time anomaly behavior based on the dynamic detection of the anomaly monitoring area. Determine the corresponding early warning content based on the real-time anomaly behavior and the corresponding behavior location, and output the corresponding anomaly early warning signal. S143: Mark multiple warning contents of the abnormal monitoring dynamic map, determine the monitoring weight of food business premises based on multiple warning contents and the distribution map of food business premises, and dynamically change the monitoring weight after a preset time period and iterate the abnormal monitoring area multiple times to dynamically optimize the visual monitoring capability of food business premises.

[0072] In the embodiments of this application, in the distribution map of food business premises, each abnormal operation event is loaded onto the distribution map of food business premises along the corresponding loading path, and the corresponding abnormal level is noted for each abnormal operation event. Based on the combination of the abnormal level of each abnormal operation event and the corresponding operation area, the corresponding abnormal heat map is determined, and a dynamic map of abnormal monitoring of food business premises is constructed by combining visualization technology. This approach takes into account the overall consideration of the combination of the abnormal level of each abnormal operation event and the corresponding operation area, ensuring the accuracy of the corresponding abnormal heat map.

[0073] At this point, the system establishes a homography matrix transformation relationship between the visual perception coordinate system (pixel coordinate system) and the physical layout coordinate system of the food business premises (world coordinate system). For each abnormal operation event determined in S13, the system extracts its core spatial features (such as the pixel coordinates [u,v] of the operation center point), and maps the two-dimensional pixel coordinates to the precise geographical location coordinates [X,Y] on the site distribution map through inverse perspective transformation (IPM) or geometric correction based on depth estimation. The system instantiates the event object containing the event ID, spatiotemporal stamp, and abnormality level description according to the preset data transmission protocol, and dynamically mounts it to the corresponding layer node of the distribution map along the determined logical loading path.

[0074] Based on the anomaly level calculated by S132 (e.g., Level-1 to Level-3), the system assigns specific visual attribute metadata to each abnormal operation event loaded onto the distribution map. This includes color mapping (e.g., Level-1 is mapped to a high-frequency flashing dark red, Level-2 to orange, and Level-3 to yellow), shape encoding (e.g., an explosion icon represents high risk, and a circle represents normal risk), and annotation labels. The system uses a graphics rendering engine to draw these primitives in real time at the corresponding coordinate points on the distribution map, and visualizes the anomaly level value, operation type text, and other metadata as floating labels, thus achieving an intuitive and symbolic expression of risk points.

[0075] Based on the geographic coordinates and anomaly level values ​​of abnormal events, the system generates a continuous "anomaly heatmap" using kernel density estimation (KDE) or Gaussian mixture model (GMM). In this process, the anomaly level is used as the bandwidth weight of the kernel function—the higher the level, the steeper the peak of the kernel function and the wider the radius of influence. At the same time, the system considers the physical boundaries of the operating area (such as walls and equipment obstructions) and masks and limits the heat diffusion to ensure that the heatmap fits the spatial topology of the actual operating area. Using the real-time rendering pipeline, the generated high-resolution heatmap is alpha-blended with the basic site distribution map in a semi-transparent overlay mode to construct an "anomaly monitoring dynamic map" that evolves in real time over time and with event updates.

[0076] Specifically, in the monitoring of the preliminary processing area of ​​the catering center, the system captured a first-level abnormal event of "the head chef illegally cutting food above the washing sink". The pixel coordinates of the operation center point in the image are (u=1150, v=980). The system uses a pre-calibrated homography matrix to convert it into physical coordinates (X=15.2m, Y=8.5m) on the distribution map of the catering center. The system loads the event object into the digital base map of the back kitchen management system and accurately locates it at the boundary between the "washing sink" and the "cooking table".

[0077] For the aforementioned unauthorized cutting and mixing incident located at (15.2m, 8.5m), since it was judged as a "Level 1 (Extremely Serious)" anomaly, the system rendered a deep red pulsed dot at that location on the distribution map; next to the dot was a semi-transparent floating text box with a clear note: "Anomaly Level: Level 1; Behavior: Cross-contamination". This highly visible red pulsed marker is extremely alarming on the calm blue plane map, intuitively reflecting the urgency of the location.

[0078] The system uses the primary anomaly point (15.2m, 8.5m) as the core heat source. Due to its extremely high level weight, it generates a high-intensity red radiation field. At the same time, the system considers that the adjacent "material preparation area" has also recently experienced a small number of primary (yellow) anomalies. These low-level events act as secondary heat sources, causing the heat map to show a gradient effect from the core red to orange and then to yellow. The system overlays this heat layer on the floor plan of the catering center with 40% transparency. At this time, in the dynamic diagram seen by the managers, the rough processing area shows a distinct "burning" red hot spot, intuitively revealing that the area is currently in a state of extremely high risk activity.

[0079] Furthermore, the system monitors anomaly monitoring dynamics in real time and outputs real-time anomaly locations based on these dynamics. It then determines corresponding anomaly monitoring areas based on multiple real-time anomaly locations and their corresponding abnormal operations. Based on the dynamic detection of these anomaly monitoring areas, it identifies corresponding real-time abnormal behaviors. Finally, it determines corresponding warning content based on these real-time abnormal behaviors and their corresponding locations, and outputs corresponding anomaly warning signals. This comprehensive approach considers both real-time abnormal behaviors and their corresponding locations, ensuring the accuracy of the warning content.

[0080] At this point, the system performs a high-frequency frame rate scan on the "anomaly monitoring dynamic map" generated by S141; it processes the heat map layer using a threshold segmentation method (such as the Otsu method or adaptive threshold) to extract pixel connected regions whose heat map value (RiskDensity) exceeds the preset warning line (such as Trisk>0.75); the system calculates the centroid coordinates of these connected regions and transforms them back to the coordinates of the actual physical space through inverse coordinate transformation, i.e., the "real-time anomaly location". This process is equivalent to finding the extreme point in the macroscopic heat map to lock the microscopic geographical location where the current risk outbreak is most intense.

[0081] The system centers on the "real-time abnormal location" and combines the "abnormal operation content" corresponding to that location (such as the specific operation type, tool size, and personnel activity range) to calculate a dynamic bounding box as the "abnormal monitoring area" (ROI). The size of this area is not fixed but is adaptively adjusted according to the complexity of the abnormal content: if it involves large movements such as cutting tools, the horizontal scanning range is expanded; if it involves delicate operations (such as seasoning), the range is reduced to improve accuracy. At the same time, the system eliminates the obstruction areas of physical obstacles (such as pillars) to ensure effective coverage of the monitoring area.

[0082] The system centrally allocates computing resources to the "anomaly monitoring area" and activates a high-precision, lightweight behavior recognition network (such as a simplified version of 3D-CNN or SlowFast network). It extracts spatiotemporal features from the video stream in this area, focusing on analyzing the motion trajectories of key points of personnel, the interaction state between hands and objects, and the changes of objects in the scene. By calculating the confidence of action features, the system filters out false alarms and finally determines the specific "real-time abnormal behavior" occurring in this area. This step is the practical verification from "thermal anomaly" to "specific behavior".

[0083] The system matches the identified "real-time abnormal behavior" with the risk logic library in S132, and combines it with the environmental attributes of the current "behavior location" (such as whether it is a clean area or whether there is an open flame) to generate multi-dimensional "early warning content". This content includes: risk level review, specific description of the violation, possible consequences and suggested intervention measures. The system triggers the corresponding signal output channel according to the early warning level, such as sending instructions to on-site IoT devices through network protocols (MQTT / TCP) or triggering a UI pop-up window on the monitoring terminal.

[0084] Specifically, in the abnormal monitoring dynamic map of the rough processing area of ​​the catering center, the system scan found that the thermal pixel value around the washing pool rose sharply in the most recent frame and exceeded the red warning threshold; through connected component analysis, the centroid coordinates of the high-temperature area were located as (X=15.25m, Y=8.55m); the system confirmed it as the current "real-time abnormal location", that is, the boundary between the washing pool and the operating table in the rough processing area.

[0085] For the abnormal location (15.25m, 8.55m), the system identified the associated operation as "the chef is cutting meat and handling possible waste". Given that the cutting action involves the swinging of the knife (with a large amplitude) and possible body rotation, the system used this coordinate as the center and extended outward by a radius of 1.5 meters to define a 3m x 3m rectangular area as the current "abnormal monitoring area". This area completely covers the washing sink, cutting board and the chef's full range of motion, excluding irrelevant areas such as the shelves behind.

[0086] Within the designated abnormal monitoring area, the system initiated high-frame-rate analysis, capturing the key points of the chef's right hand holding the "knife" frequently crossing the boundary of the "cutting board" and entering the water-filled area above the "washing pool." Simultaneously, the system detected "meat chunks" falling from the cutting board into the water-filled area, and the chef's hand entering the water-filled area to retrieve them. Based on these spatiotemporal characteristics, the system determined the current real-time abnormal behavior to be: "performing unauthorized cutting and preparation above the washing pool, and engaging in secondary contact with contaminated food that has fallen into the water."

[0087] The system combines the location of "above the washing pool" (a high-risk source of contamination) with the behavior of "contact with contaminated food" to generate an early warning message: "Warning: Level 1 cross-contamination risk detected. The head chef is handling food above the washing pool, which may cause excessive bacteria levels." Immediately afterward, the system outputs an abnormal warning signal: the on-site broadcast in the catering center's kitchen automatically broadcasts a voice prompt: "Please pay attention to operating procedures in the preliminary processing area." At the same time, the image of this area on the monitoring screen is automatically pinned to the top and flashes a red alarm box, and the violation segment is pushed to the food safety administrator's terminal.

[0088] Therefore, multiple early warning items are marked on the dynamic map of abnormal monitoring. The monitoring weight of the food business premises is determined based on the multiple early warning items and the distribution map of the food business premises. The monitoring weight changes dynamically after a preset time period and iterates the abnormal monitoring area multiple times to dynamically optimize the visual monitoring capability of the food business premises. This approach takes into account the overall consideration of multiple early warning items and the distribution map of the food business premises, ensuring the accuracy of the monitoring weight of the food business premises. At the same time, the abnormality level of abnormal operation events is further controlled. This approach realizes the overall consideration of the abnormality level of each abnormal operation event, the corresponding operation area, and the distribution map of the food business premises, thereby improving the accuracy of the dynamic map of abnormal monitoring of the food business premises and dynamically optimizing the visual monitoring capability of the food business premises.

[0089] At this point, the system constructs a sliding window in the time dimension, accumulating all "warning content" triggered within the window; it uses Natural Language Processing (NLP) technology to perform semantic analysis on the warning text, extracting core risk elements (such as "cross-contamination" and "unauthorized entry"); simultaneously, based on the grid structure of the "food business premises distribution map," the system maps each warning content to a specific geographic grid unit; the system calculates the warning frequency, average risk level, and decay factor of the most recent warning time within each grid unit, generating a multi-dimensional "risk feature vector," and marking it on the corresponding topological node of the distribution map, forming a risk situation awareness layer covering the entire domain.

[0090] The system calculates the "monitoring weight" of each grid area based on risk feature vectors using a weighted scoring method (such as the risk matrix method or exponentially weighted moving average, EWMA). The monitoring weight determines the allocation of visual computing resources. The weight is a scalar that dynamically evolves over time: when a preset time period (such as T=15 minutes) ends, the system introduces a time decay function λ(t) to reduce the weight coefficient of historical warning data. If no new warnings are generated during this time period, the weight decreases exponentially; if warnings continue to be generated, the weight remains high or continues to rise. This mechanism ensures that the monitoring weight can reflect the real-time trend of risk changes.

[0091] Based on the updated monitoring weight distribution map, the system performs multiple rounds of iterative optimization. In each iteration, the system dynamically filters out grids with weights higher than the baseline according to the current weight threshold, and redefines the set of "abnormal monitoring areas". The system calls the resource scheduler to dynamically adjust the PTZ (pan-tilt-zoom) parameters of the cameras to focus on high-weight areas. At the same time, at the back-end inference level, the system increases the video stream frame rate (FPS) and detection accuracy of high-weight areas (enabling a high-precision model) and reduces the detection frequency of low-weight areas (enabling a low-power model). Through this closed-loop iteration of "monitoring-feedback-adjustment", the system achieves adaptive optimization of the visual monitoring capabilities of the entire venue.

[0092] Specifically, the system constructs a sliding window in the time dimension, accumulating all "warning content" triggered within the window; it uses natural language processing (NLP) technology to perform semantic analysis on the warning text, extracting core risk elements (such as "cross-contamination" and "unauthorized entry"); simultaneously, based on the grid structure of the "food business premises distribution map," the system maps each warning content to a specific geographic grid unit; the system calculates the warning frequency, average risk level, and decay factor of the most recent warning time within each grid unit, generating a multi-dimensional "risk feature vector," and marking it on the corresponding topological node of the distribution map, forming a risk situation awareness layer covering the entire domain.

[0093] For the Grid_ID:C-12 grid in the coarse processing area, due to the continuous occurrence of high-risk warnings, the system calculates its initial monitoring weight as 0.95 (out of 1.0), meaning that this area occupies the highest priority. In the adjacent "food cold storage room" (Grid_ID:C-08), since there have been no abnormalities in the past hour, its monitoring weight has decayed to 0.2 over time. At this point, if the coarse processing area does not trigger any new warnings in the next 10 minutes, its weight will begin to slowly decrease according to the preset curve, releasing computing resources.

[0094] The system detected that Grid_ID:C-12 (cleaning pool) had a weight as high as 0.95, while other areas had lower weights. In the iterative optimization, the system decided to reallocate visual resources: control the camera Cam_B covering the roughing area to switch from panoramic cruise mode to close-up monitoring mode, lock the lens center on the coordinates of the cleaning pool, and increase the analysis frame rate of this video stream from 15fps to 30fps, while loading a high-sensitivity mode for "water contact". Conversely, for Cam_C (cold storage room) with a weight of only 0.2, the system reduced its frame rate to 5fps to save computing power. Through this dynamic adjustment, the system achieved accurate deployment and maximized efficiency of visual monitoring capabilities.

[0095] Please see Figure 6 , Figure 6This is a schematic diagram of the structural composition of a visual detection-based abnormal behavior detection system for food service establishments according to an embodiment of the present invention. The visual detection-based abnormal behavior detection system for food service establishments is applied to the aforementioned visual detection-based abnormal behavior detection method for food service establishments. The visual detection-based abnormal behavior detection system for food service establishments includes: a visual detection module 21, used to determine multiple functional areas based on the recognition of a distribution map of the food service establishment; triggering visual detection of corresponding camera groups based on the multiple functional areas to acquire multiple images of each functional area; and determining a panoramic view of the functional area based on the synthesis of the multiple images; and an image recognition module 22, used to determine multiple operation markers in each functional area based on the recognition of the panoramic view, wherein the operation markers... The system presents the operation area and operation content, and determines the corresponding abnormal operation events based on each operation mark, the corresponding operation location, and the adjacent environmental image. The abnormality level module 23 is used to mark multiple abnormal operation events in the same functional area, and determines multiple abnormal behaviors based on the identification of each abnormal operation event. The abnormality level of the abnormal operation event is determined based on the multiple abnormal behaviors, the functional type of the functional area, and the corresponding operator. The visual monitoring capability module 24 is used to construct an abnormal monitoring dynamic map of the food business premises based on the abnormality level of each abnormal operation event, the corresponding operation area, and the distribution map of the food business premises. This abnormal monitoring dynamic map outputs multiple abnormal monitoring areas and triggers corresponding abnormal early warning signals to dynamically optimize the visual monitoring capability of the food business premises.

[0096] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for detecting abnormal behavior in food service establishments based on visual inspection, characterized in that, include: Based on the identification of the distribution map of food business premises, multiple functional areas are determined. Based on the multiple functional areas, visual detection of the corresponding camera groups is triggered to obtain multiple images of each functional area. Based on the synthesis of multiple images, a panoramic view of the functional area is determined. In each functional area, multiple operation markers are determined based on the recognition of the panoramic image. These operation markers represent the operation area and operation content. Based on each operation marker, the corresponding operation position, and the adjacent environmental image, the corresponding abnormal operation event is determined. Mark multiple abnormal operation events in the same functional area, determine multiple abnormal behaviors based on the identification of each abnormal operation event, and determine the abnormality level of the abnormal operation event based on the multiple abnormal behaviors, the functional type of the functional area, and the corresponding operator. Based on the anomaly level of each abnormal operation event, the corresponding operation area, and the distribution map of the food business premises, an abnormal monitoring dynamic map of the food business premises is constructed. This abnormal monitoring dynamic map outputs multiple abnormal monitoring areas and triggers corresponding abnormal early warning signals to dynamically optimize the visual monitoring capabilities of the food business premises.

2. The method for detecting abnormal behavior in food service establishments based on visual detection according to claim 1, characterized in that, The identification of multiple functional areas based on the distribution map of food business premises, and the visual detection of the corresponding camera group based on the multiple functional areas, so as to obtain multiple images of each functional area; The panoramic view of the functional area is determined by synthesizing multiple images, including: acquiring a distribution map of the food business premises, performing image segmentation on the distribution map of the food business premises, and outputting multiple functional areas as the segmentation semantics change during the segmentation process. The multiple functional areas are the kitchen cooking area, the preliminary processing area, the washing and disinfection area, and the food preparation area. In each functional area, the regional space of each functional area is marked, and multiple cameras are identified during the traversal of the regional space to construct a corresponding camera group. At the same time, the camera group performs multi-position visual detection on the functional area and acquires multiple images of each functional area.

3. The method for detecting abnormal behavior in food service establishments based on visual detection according to claim 2, characterized in that, The identification of multiple functional areas based on the distribution map of food business premises, and the visual detection of the corresponding camera group based on the multiple functional areas, so as to obtain multiple images of each functional area; Determining a panoramic view of a functional area based on the synthesis of multiple images also includes: determining multiple features based on image recognition of each image, determining corresponding weight content based on the multiple images and their corresponding semantics, and triggering the synthesis of multiple images along the weight content, the corresponding image, and the corresponding image contour to determine a panoramic view of the functional area.

4. The method for detecting abnormal behavior in food service establishments based on visual detection according to claim 1, characterized in that, In each functional area, multiple operation markers are determined based on the recognition of the panoramic image. These operation markers present the operation area and operation content. Based on each operation marker, the corresponding operation position, and the adjacent environmental image, the corresponding abnormal operation event is determined, including: real-time monitoring of each functional area, multi-target recognition of the panoramic image, and during the recognition process, determining the interaction between multiple key points of the operator's torso and object features, and marking them as corresponding operation markers. These operation markers cover the operation type, the corresponding operation area, and the operation range to present the operation area and operation content.

5. The method for detecting abnormal behavior in food service establishments based on visual detection according to claim 4, characterized in that, In each functional area, multiple operation markers are determined based on the recognition of the panoramic image. These operation markers represent the operation area and operation content. Based on each operation marker, its corresponding operation location, and adjacent environmental images, corresponding abnormal operation events are determined. This also includes: determining multiple corresponding operation boundary points based on the tracing of the operation markers, comparing these boundary points with the qualified area boxes marked on the panoramic image to identify multiple abnormal operation points, each of which is outside the qualified area; performing multi-factor analysis on the multiple abnormal operation points and the operation locations corresponding to the operation markers in a time-series dimension, determining the abnormal content of the multiple abnormal operation points during the analysis; simultaneously, determining the corresponding environmental image based on the environmental traversal of the operation location corresponding to the operation marker, identifying the abnormal environmental content based on the recognition of the environmental image, and determining the corresponding abnormal operation events based on the abnormal environmental content and the abnormal content of the multiple abnormal operation points.

6. The method for detecting abnormal behavior in food service establishments based on visual detection according to claim 1, characterized in that, The process of marking multiple abnormal operation events in the same functional area, identifying multiple abnormal behaviors based on the identification of each abnormal operation event, and determining the abnormality level of the abnormal operation event based on the multiple abnormal behaviors, the functional type of the functional area, and the corresponding operator includes: marking multiple abnormal operation events in the same functional area, simultaneously performing multi-channel analysis on the multiple abnormal operation events to identify multiple abnormal operation events that are close in time and space in the same functional area, aggregating the multiple abnormal operation events to form corresponding continuous abnormal content, and identifying multiple abnormal behaviors based on the identification of continuous abnormal content.

7. The method for detecting abnormal behavior in food service establishments based on visual detection according to claim 6, characterized in that, The process of marking multiple abnormal operation events in the same functional area, identifying multiple abnormal behaviors based on the identification of each abnormal operation event, and determining the abnormality level of the abnormal operation event based on the multiple abnormal behaviors, the functional type of the functional area, and the corresponding operator, further includes: collecting a preset risk mapping relationship, performing multi-factor matching on the multiple abnormal behaviors, the functional type of the functional area, and the risk mapping relationship, determining the corresponding risk content during the matching process, marking the impact event corresponding to the risk content, and determining the abnormality level of the abnormal operation event based on the impact event corresponding to the risk content, the corresponding operator, and the operator's job position.

8. The method for detecting abnormal behavior in food service establishments based on visual detection according to claim 1, characterized in that, The method involves constructing a dynamic monitoring map of food business premises based on the anomaly level of each abnormal operation event, the corresponding operation area, and the distribution map of the food business premises. This dynamic monitoring map outputs multiple abnormal monitoring areas and triggers corresponding abnormal early warning signals to dynamically optimize the visual monitoring capabilities of the food business premises. This includes: loading each abnormal operation event along the corresponding loading path into the distribution map of the food business premises, noting the corresponding anomaly level for each abnormal operation event, determining the corresponding abnormal heat map based on the combination of the anomaly level of each abnormal operation event and the corresponding operation area, and constructing the dynamic monitoring map of food business premises using visualization technology.

9. The method for detecting abnormal behavior in food service establishments based on visual detection according to claim 8, characterized in that, The method involves constructing an abnormal monitoring dynamic map of food business premises based on the abnormality level of each abnormal operation event, the corresponding operation area, and the distribution map of the food business premises. This abnormal monitoring dynamic map outputs multiple abnormal monitoring areas and triggers corresponding abnormal early warning signals to dynamically optimize the visual monitoring capabilities of food business premises. It also includes: real-time monitoring of the abnormal monitoring dynamic map and outputting real-time abnormal locations based on the abnormal monitoring dynamic map; determining corresponding abnormal monitoring areas based on multiple real-time abnormal locations and corresponding abnormal operation content; determining corresponding real-time abnormal behavior based on the dynamic detection of the abnormal monitoring area; determining corresponding early warning content based on the real-time abnormal behavior and the corresponding behavior location; and outputting corresponding abnormal early warning signals. Multiple warning items are marked on the dynamic map of abnormal monitoring. The monitoring weight of food business premises is determined based on the multiple warning items and the distribution map of food business premises. The monitoring weight changes dynamically after a preset time period and iterates the abnormal monitoring area multiple times to dynamically optimize the visual monitoring capability of food business premises.

10. A visual detection-based abnormal behavior detection system for food service establishments, characterized in that, The vision-based abnormal behavior detection system for food service establishments is applied to the vision-based abnormal behavior detection method for food service establishments as described in any one of claims 1-9. The vision-based abnormal behavior detection system for food service establishments includes: a vision detection module, used to determine multiple functional areas based on the recognition of a distribution map of the food service establishment; triggering vision detection of corresponding camera groups based on the multiple functional areas to acquire multiple images of each functional area; and determining a panoramic view of the functional area based on the synthesis of the multiple images; and an image recognition module, used to determine multiple operation markers in each functional area based on the recognition of the panoramic view, the operation markers representing the operation area and operation content, and so on. The system identifies corresponding abnormal operation events based on various operation markers, their corresponding operation locations, and adjacent environmental images. An anomaly level module is used to mark multiple abnormal operation events within the same functional area, identify multiple abnormal behaviors based on the recognition of each abnormal operation event, and determine the anomaly level of each abnormal operation event based on the multiple abnormal behaviors, the functional type of the functional area, and the corresponding operator. A visual monitoring capability module is used to construct a dynamic anomaly monitoring map of the food business premises based on the anomaly level of each abnormal operation event, the corresponding operation area, and the distribution map of the food business premises. This dynamic anomaly monitoring map outputs multiple anomaly monitoring areas and triggers corresponding anomaly warning signals to dynamically optimize the visual monitoring capabilities of the food business premises.

Images