Image recognition-based canteen catering compliance real-time monitoring system and method

Abstract

The invention provides a canteen catering compliance real-time monitoring system and method based on image recognition. The canteen catering compliance real-time monitoring system comprises a heterogeneous sensor array. Edge computing nodes; the environment self-adaption module is used for solving the recognition problems in complex scenes such as background color interference, day and night illumination fluctuation and steam shielding through HSV histogram similarity analysis and dynamic weight adjustment; a behavior compliance analysis engine; and a real-time feedback execution module. A multi-source sensor clock is synchronized, a unified space-time coordinate system is established, an illumination-infrared weight dynamic mapping and environment feature compensation algorithm is adopted, a visible light image flow, thermodynamic data and depth information are processed in parallel, multi-modal features are fused for composite operation behavior recognition, and a dynamic response instruction is generated according to a predefined compliance strategy library. According to the method, the recognition accuracy in a background color interference scene can be improved, and the night false alarm rate is reduced. According to the invention, the problem of poor adaptability to complex environments in traditional supervision can be effectively solved, and the food safety management level is significantly improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent monitoring and food processing safety technology, and in particular to a system and method for real-time monitoring of canteen meal distribution compliance based on image recognition. Background Art

[0002] In the food processing industry, especially in collective canteens, compliance supervision of operational procedures has always been a key link in food safety management. According to statistics from the "Food Safety Operation Specifications for Catering Services", back-kitchen operations account for as much as 63.5% of foodborne illness incidents caused by non-standard operations in catering units in my country. Traditional supervision models mainly rely on the following two methods: (1) Manual inspection mechanism: Quality inspectors conduct on-site inspections 2-3 times a day, but due to limited human resources, the actual monitoring coverage rate is less than 15%, and there are blind spots during inspections; (2) Video recording tracing: Ordinary surveillance cameras are used to record the operation process, but manual playback and retrieval are required. The average event response time exceeds 48 hours, and real-time intervention cannot be achieved.

[0003] In recent years, although some companies have tried to introduce intelligent monitoring technologies, such as the deep learning-based kitchen behavior recognition system disclosed in CN112950715A, its technical solutions have obvious limitations: first, single-modal visual analysis is difficult to cope with the complex environment of the canteen (such as steam interference causing image blur, low illumination at night, etc.); second, existing algorithms mostly focus on single-dimensional detection (such as only identifying the wearing status of the chef's hat) and lack the ability to make coordinated compliance judgments on multiple factors (such as simultaneously verifying dress codes and operation sequences); third, the timing analysis module generally adopts a simple LSTM structure, and the recognition accuracy of continuous operation processes is less than 82%, especially when processing parallel processes, the false alarm rate is as high as 37%.

[0004] Deeper technical bottlenecks lie in the following: ① Existing systems often utilize cloud-based processing architectures, which, due to network latency, result in an average delay of over 5 seconds between image acquisition and violation alerts; ② They lack dynamic adaptive mechanisms, making it impossible to adjust testing standards based on specific cuisine characteristics (e.g., differences in cooking processes between Chinese and Western cuisine); and ③ They lack deep integration with HACCP critical control points, resulting in a disconnect between monitoring data and the food safety management system. These issues severely limit the practical application value of intelligent monitoring systems, necessitating urgent breakthroughs through technological innovation. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a real-time monitoring system and method for canteen meal distribution compliance based on image recognition to solve the technical defects of the above-mentioned canteen monitoring.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: A real-time monitoring system for canteen meal distribution compliance based on image recognition. The monitoring system consists of a hardware layer, a data layer, an algorithm layer, and an application layer connected in sequence. The hardware layer consists of heterogeneous sensor arrays and edge computing nodes: A heterogeneous sensor array, including visible light cameras, infrared thermal imagers, and depth sensors, is deployed in the food preparation operation area. The depth sensor is used to collect three-dimensional depth information of the operation area, construct the spatial position and motion trajectory of the operator, and assist in three-dimensional motion recognition in scenarios such as steam occlusion. Edge computing nodes communicate with heterogeneous sensor arrays and have built-in multi-task detection models and timing analysis models; The data layer receives data output by edge computing nodes in the hardware layer. The data layer is equipped with a video database, feature warehouse, and compliance policy library. The algorithm layer includes a behavioral compliance analysis engine that fuses multimodal perception data and verifies the spatiotemporal compliance of operational processes; The application layer includes a real-time feedback execution module, which calls the predefined rules in the compliance policy library to generate dynamic response instructions based on the analysis results of the algorithm layer's fusion of multimodal features.

[0007] The specific configuration of the above-mentioned heterogeneous sensor array also includes: A macro camera deployed above the operating table, with an optical magnification of ≥5×, is used for hand cleanliness detection; A pressure sensor integrated into the handle of the kitchenware, with a sampling frequency of ≥100Hz, is used for grip posture recognition; The weighing module installed at the end of the canteen conveyor belt has a measurement accuracy of ±2g and is used to verify the weight of finished products.

[0008] The above multi-task detection model adopts the improved YOLOv8 architecture, including: Shared feature extraction layer, using CSPDarknet53 backbone network; Parallel detection branch, including: Clothing compliance detection head outputs the wearing status of masks, hats, and work clothes; Tool usage detection head to identify the type of kitchenware and how it is held; Food classification detection head to determine the risk of cross-contamination between raw and cooked food.

[0009] The objective function of the improved YOLOv8 architecture mentioned above is:

[0010] Among them, λattire is the weight coefficient of dress compliance detection; λtool is the weight coefficient of tool compliance detection; λfood is the weight coefficient of dress compliance; L attireClassification loss for dress compliance testing; L tool The tool uses the bounding box regression loss for compliance detection, L food Classification loss for food classification compliance testing; L pose is the pose constraint loss based on key point coordinates; The constraints are:

[0011] Among them, P mask is the probability threshold of wearing a mask; P hat is the confidence threshold for chef hat detection; P tool is the probability threshold of the correct tool holding method; the detection head weight is dynamically optimized through the Lagrange multiplier method.

[0012] The above timing analysis model includes: A spatiotemporal graph convolutional network constructs a spatiotemporal topological graph of the operator's skeletal key points; Finite state automaton, defining the legal state transition matrix of the standard operating procedure; Dynamic time warping module for flexible time matching of process steps.

[0013] The multi-task detection model mentioned above also includes an environment adaptation module to solve the recognition problems in the following scenarios: Background color interference suppression: Separate the chef's uniform from the workbench background using the HSV color space, dynamically adjust the detection threshold, and enable infrared thermal imager-assisted positioning when the similarity between the background color and the uniform color is greater than 80%; Day and night illumination compensation: Based on ambient light sensor data, the visible light camera's white balance mode is dynamically switched, and infrared thermal imager data is integrated to improve nighttime recognition accuracy. Steam interference suppression: When the thermal imaging module detects a steam area, it suspends visible light image analysis and prioritizes using depth sensor data to construct a 3D motion trajectory.

[0014] The background color interference suppression in the above-mentioned environment adaptation module specifically includes:

[0015] Among them, Hattire is the HSV histogram of the clothing area, and Hbg is the HSV histogram of the background area; When Similarity>0.8, infrared-assisted positioning is triggered and thermodynamic features are integrated to enhance target detection; Day and night light compensation specifically includes: Create a light intensity-detection sensitivity mapping table:

[0016] Dynamically adjust multimodal fusion weights according to real-time lighting data.

[0017] The above-mentioned real-time feedback execution module includes: Augmented reality projection device to mark illegal areas on the operating table; Mechanical interlock controller, electrically connected to the kitchen equipment, with response levels including: Level 1 response: trigger LED indicator to flash; Level 2 response: Activate voice warning and record video footage of the violation; Level 3 response: Cut off the power supply to the designated equipment and send an emergency notification.

[0018] The monitoring method using the above-mentioned canteen meal distribution compliance real-time monitoring system based on image recognition includes the following steps: S1. Establish a unified spatiotemporal coordinate system by synchronizing the clocks of multiple sensors. S2. Parallel execution by edge computing nodes: Visible light image stream processing: extracting clothing, tools, and food features; Thermodynamic data analysis: calculation of temperature gradient and rate of change; Depth information processing: constructing the three-dimensional motion trajectory of the operation area; S3. The behavioral compliance analysis engine at the algorithm layer integrates multimodal features and identifies complex operational behaviors through a spatiotemporal graph convolutional network. S4. Generate dynamic response instructions based on the predefined compliance policy library.

[0019] The above-mentioned S3 specifically includes: S3a. Divide the continuous operation into time windows; S3b. Extract spatiotemporal feature vectors for each window; S3c. Use the improved DTW algorithm to match the standard process template; S3d. When the matching degree is lower than the threshold θ, a process abnormality event is triggered.

[0020] The dynamic response instruction generation method in S4 above includes: Calculate the environment complexity factor: ; Where α = 0.15, β = 0.08, N_people is the number of operators, and N_dishes is the number of dishes processed simultaneously; Dynamically adjust the detection sensitivity according to the γ value:

[0021] The above monitoring method further includes a knowledge base updating step S5: When collecting new violation samples, start the incremental learning process: Freeze backbone network parameters; Only fine-tune the fully connected layers of the detection head; Update the state transition rules of the compliance policy library.

[0022] The present invention provides a real-time monitoring system and method for canteen meal distribution compliance based on image recognition. Through multimodal perception and edge intelligent analysis, it can achieve real-time supervision of the entire canteen meal distribution process, accurately identify violations such as dress, operation, and hygiene (with an accuracy rate of 96.5%), with a response delay of less than 0.8 seconds. The dynamic adjustment of the detection threshold reduces the false alarm rate to 2.3%, reduces deployment costs by 60%, and pilot projects show a 76% decrease in the violation rate. The simultaneous generation of visual traceability reports improves management efficiency by 83%, achieving the dual value of food safety assurance and operating cost optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention will be further described below with reference to the accompanying drawings and examples: Figure 1 This is a block diagram of the overall system structure of the canteen meal distribution compliance real-time monitoring system and method based on image recognition in the present invention; Figure 2 This is a flow chart of detection image feature extraction of the real-time monitoring system and method for canteen meal distribution compliance based on image recognition in the present invention; Figure 3 It is a state transition verification diagram of the meal preparation operation process of the real-time monitoring system and method for canteen meal preparation compliance based on image recognition in the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0025] A real-time monitoring system for canteen meal distribution compliance based on image recognition. The monitoring system consists of a hardware layer, a data layer, an algorithm layer, and an application layer connected in sequence. The hardware layer includes a heterogeneous sensor array 101 and edge computing nodes 102: Heterogeneous sensor array 101, comprising a visible light camera, an infrared thermal imager, and a depth sensor, is deployed in the food preparation operation area. The depth sensor is used to collect three-dimensional depth information of the operation area, construct the spatial position and motion trajectory of the operator, and assist in three-dimensional motion recognition in scenarios such as steam occlusion. The edge computing node 102 is in communication with the heterogeneous sensor array 101 and has a built-in multi-task detection model and a time series analysis model; The data layer receives data output by edge computing nodes 102 within the hardware layer. The data layer includes a video database, a feature warehouse, and a compliance policy library. The compliance policy library is built based on the HACCP system and stored using the Neo4j knowledge graph. It contains operating specifications, process state transition rules, and violation thresholds (storing >200 entity nodes and >500 relationship edges), providing a standard basis for compliance verification for the algorithm layer. The algorithm layer includes a behavioral compliance analysis engine 103 for fusing multimodal perception data and verifying the spatiotemporal compliance of operational processes; The application layer includes a real-time feedback execution module 104, which generates dynamic response instructions by calling predefined rules in the compliance policy library based on the analysis results of the algorithm layer integrating multimodal features.

[0026] The specific configuration of the above-mentioned heterogeneous sensor array 101 also includes: A macro camera deployed above the operating table, with an optical magnification of ≥5×, is used for hand cleanliness detection; A pressure sensor integrated into the handle of the kitchenware, with a sampling frequency of ≥100Hz, is used for grip posture recognition; The weighing module installed at the end of the canteen conveyor belt has a measurement accuracy of ±2g and is used to verify the weight of finished products.

[0027] The above multi-task detection model adopts the improved YOLOv8 architecture, including: Shared feature extraction layer 201, using CSPDarknet53 backbone network; The parallel detection branch 202 includes: Clothing compliance detection head 202a outputs the wearing status of masks, hats, and work clothes; Tool use detection head 202b, identifying the type of kitchenware and the way it is held; The food classification detection head 202c determines the risk of cross contamination between raw and cooked food.

[0028] The objective function of the improved YOLOv8 architecture mentioned above is:

[0029] Among them, λattire=1.2 is the weight coefficient of dress compliance detection; among them, λtool=0.9 is the weight coefficient of tool compliance detection; among them, λfood=1.2 is the weight coefficient of dress compliance; L attire is the classification loss for clothing compliance detection (using Focal Loss); L tool The tool uses the bounding box regression loss of compliance detection (using CIoU Loss), L food The classification loss for food classification compliance testing (using Focal Loss); Lpose is the posture constraint loss based on the key point coordinates (using MSE Loss); The constraints are:

[0030] Among them, P mask is the probability threshold of wearing a mask; P hat is the confidence threshold for chef hat detection; P tool is the probability threshold of the correct tool holding method; the detection head weight is dynamically optimized through the Lagrange multiplier method.

[0031] The above timing analysis model includes: A spatiotemporal graph convolutional network constructs a spatiotemporal topological graph of the operator's skeletal key points; Finite state automaton, defining the legal state transition matrix of the standard operating procedure; Dynamic time warping module for flexible time matching of process steps.

[0032] The multi-task detection model mentioned above also includes an environment adaptation module to solve the recognition problems in the following scenarios: Background color interference suppression: Separate the chef's uniform from the workbench background using the HSV color space, dynamically adjust the detection threshold, and enable infrared thermal imager-assisted positioning when the similarity between the background color and the uniform color is greater than 80%; Day and night illumination compensation: Based on ambient light sensor data, the visible light camera's white balance mode is dynamically switched (daytime: 5500K color temperature, nighttime: 3000K color temperature), and infrared thermal imager data is integrated to improve nighttime recognition accuracy. Steam interference suppression: When the thermal imaging module detects a steam area (temperature gradient > 5°C / cm²), visible light image analysis is suspended and depth sensor data is used to construct a 3D motion trajectory.

[0033] The background color interference suppression in the above-mentioned environment adaptation module specifically includes:

[0034] Among them, Hattire is the HSV histogram of the clothing area, and Hbg is the HSV histogram of the background area; When Similarity>0.8, infrared-assisted positioning is triggered, and thermodynamic features (temperature distribution variance>2°C) are integrated to enhance target detection; Day and night light compensation specifically includes: Create a light intensity-detection sensitivity mapping table:

[0035] Dynamically adjust multimodal fusion weights according to real-time lighting data.

[0036] The above-mentioned real-time feedback execution module 104 includes: Augmented reality projection device to mark illegal areas on the operating table; Mechanical interlock controller, electrically connected to the kitchen equipment, with response levels including: Level 1 response: trigger LED indicator to flash; Level 2 response: Activate voice warning and record video footage of the violation; Level 3 response: Cut off the power supply to the designated equipment and send an emergency notification.

[0037] The monitoring method using the above-mentioned canteen meal distribution compliance real-time monitoring system based on image recognition includes the following steps: S1. Establish a unified spatiotemporal coordinate system by synchronizing the clocks of multiple sensors. S2. Executed in parallel by edge computing node 102: Visible light image stream processing S2a: extracting clothing, tools, and food features; Thermodynamic data analysis S2b: calculation of temperature gradient and rate of change; Depth information processing S2c: constructing the three-dimensional motion trajectory of the operation area; S3. The behavioral compliance analysis engine 103 at the algorithm layer fuses multimodal features and identifies complex operational behaviors through a spatiotemporal graph convolutional network. S4. Generate dynamic response instructions based on the predefined compliance policy library.

[0038] The above-mentioned S3 specifically includes: S3a. Divide the continuous operation into time windows (window length T = 30s, overlap rate 50%); S3b. Extract spatiotemporal feature vectors (dimension 256×1) for each window; S3c. Use the improved DTW algorithm to match the standard process template; S3d. When the matching degree is lower than the threshold θ, a process abnormality event is triggered.

[0039] The dynamic response instruction generation method in S4 above includes: Calculate the environment complexity factor: ; Where α = 0.15, β = 0.08, N_people is the number of operators, and N_dishes is the number of dishes processed simultaneously; Dynamically adjust the detection sensitivity according to the γ value:

[0040] The above monitoring method further includes a knowledge base updating step S5: When collecting new violation samples, start the incremental learning process: Freeze backbone network parameters; Only fine-tune the fully connected layers of the detection head (learning rate η=5e-5); Update the state transition rules of the compliance policy library.

[0041] Example 2: Real-time monitoring system for canteen meal distribution compliance based on image recognition: 1. Hardware system deployment 1.1 Visual Acquisition Network Configuration Camera layout plan: Panoramic monitoring unit: Four 8K ultra-wide-angle cameras (model: AXIS Q6155-E) are deployed at the four corners of the catering area. The horizontal field of view is ≥180°, the vertical field of view covers a height of 0.5-2.5 meters, the frame rate is 30fps, and it supports H.265 encoding. Macro detection unit: A focus-adjustable industrial camera (Basler ace 2acA2440-75um) with a resolution of 2448 × 2048 and a ring light (wavelength adjustable from 450 to 650 nm) is installed 20 cm above the operating table. Thermal imaging unit: FLIR A315 infrared thermal imager is deployed at the food outlet, with a temperature detection range of -20~150℃ and an accuracy of ±2℃ Depth perception unit: uses Intel RealSense D455 depth camera, with a maximum detection distance of 6m and depth accuracy of <2%.

[0042] 1.2 Edge Computing Nodes Each operating area is equipped with an NVIDIA Jetson AGX Orin computing unit. Technical specifications: GPU computing power: 200 TOPS (INT8) Memory: 32GB LPDDR5 Storage: 1TB NVMe SSD Supports parallel processing of multiple video streams (up to 8 channels of 4K@30fps).

[0043] 1.3 Feedback actuator Sound and light alarm: installed in front of the operating console, supports speech synthesis (violation type broadcast) and RGB LED warning (red light - serious violation, yellow light - minor violation); Mechanical interlocking device: An electromagnetic brake (response time < 200ms) is integrated into the conveyor system to automatically pause the production line when a major violation is detected.

[0044] 2. Software system implementation 2.1 Data Preprocessing Process Multi-source data synchronization: Use PTP (Precision Time Protocol) to achieve cross-camera clock synchronization, with time deviation less than 1ms; Establish spatial coordinate system mapping relationship and complete multi-view image registration through checkerboard calibration method; Keyframe extraction algorithm: Calculate inter-frame motion vectors based on the optical flow method (Farneback algorithm), and trigger keyframe capture when the cumulative displacement is greater than 15 pixels; Adaptive sampling interval adjustment: 10fps sampling is used during busy periods (operation frequency > 2 times / second), and reduced to 2fps during idle periods.

[0045] 2.2 Feature Recognition Technical Details 2.2.1 Multimodal Feature Extraction Method This model collects multi-source data through a heterogeneous sensor array (including visible light cameras, infrared thermal imagers, and depth sensors) and uses the following methods to extract and fuse multimodal features: (1) Visual feature extraction: The input image is passed through the CSPDarknet53 backbone network to extract multi-scale feature maps, and the ECA attention module is used to enhance the key area response. The attention weight is calculated as: ; Among them, X is the input feature map, GAP is the global average pooling, MLP is the multi-layer perceptron, and σ is the Sigmoid function.

[0046] (2) Thermodynamic feature fusion: The infrared thermal imager data and the visible light image are weightedly fused to generate a thermodynamic characteristic map. The fusion formula is:

[0047] Among them, α=0.6 is the thermal image weight, β=0.4 is the visible light weight, T(x,y) is the temperature gradient, and I(x,y) is the RGB pixel value.

[0048] (3) Spatiotemporal trajectory modeling: The 3D trajectory of the operator's skeleton key points is constructed based on the depth sensor data, and the Kalman filter is used to optimize the motion equation:

[0049] Among them, x t is the state vector at time t, F is the state transfer matrix, B is the control matrix, w t is the process noise, ut is the control input vector at time t (such as hand movement speed), which is used to describe the impact of external control on state transition.

[0050] Network architecture improvements: Backbone: The CSPDarknet53 backbone network integrates the ECA attention module (compression ratio = 16) to improve the detection ability of small objects (such as the edges of masks and kitchenware handles).

[0051] Neck: BiFPN (Bidirectional Feature Pyramid Network) is used to replace the original PANet to achieve efficient fusion of multi-scale features.

[0052] Head: Separate detection head design, responsible for: Clothing compliance testing (mask, hat, work clothes status); Tool use identification (kitchen utensil types and holding posture); Food classification and cross-contamination risk assessment.

[0053] 2.2.2 Optimization of the Time Series Analysis Model Based on the Spatiotemporal Graph Convolutional Network (ST-GCN) To improve the recognition accuracy of continuous operation processes, this model optimizes the spatiotemporal graph convolutional network (ST-GCN) as follows: (1) Define the rules for updating the spatiotemporal graph node features: ; Among them, A is the adjacency matrix, D is the degree matrix, W is the learnable weight, X is the input feature, and ⊕ represents the time dimension convolution.

[0054] (2) The state transition constraints of the finite state machine (FSM) are: Define process state transition constraints to ensure that the operation process complies with predefined standards: ; Among them, S i →S j Indicates process state transition, t max is the maximum allowed time difference, and ΔT is the actual time interval.

[0055] Feature extraction and behavior recognition Feature extraction layer: A 3D ResNet-50 network is used to extract spatiotemporal features (input size: 16 frames × 224 × 224). This layer is combined with an improved OpenPose algorithm to detect 21 key points of the hands, enhancing fine-grained motion capture capabilities.

[0056] Behavior recognition layer: The ST-GCN network is configured with 9 layers of graph convolution and a hidden layer dimension of 256, supporting multi-process parallel analysis.

[0057] The Bi-LSTM module models long temporal dependencies, has a time window length of 64 frames, and a hidden layer of 128 units, effectively identifying process delays or sequence errors.

[0058] 3. System integration and debugging 3.1 Multimodal Data Fusion Develop space-time alignment middleware to achieve: Pixel-level fusion of infrared temperature data and visible light images (weighted average method, α = 0.7); Depth information is mapped to 2D detection boxes (3D violation area delineation).

[0059] 3.2 Dynamic Update Mechanism The compliance standards library is stored in the Neo4j knowledge graph and includes: Entity: process steps / tools / dress requirements, etc. (>200 nodes); Relationships: precedence / dependencies / prohibited behaviors, etc. (>500 edges); Model online update interface: Supports incremental learning (newly added violation samples automatically trigger fine-tuning training); Dynamic adjustment of thresholds (e.g., adjusting the frequency of hand hygiene testing based on seasonal changes).

[0060] 3.3 System calibration and verification Calibration process: Perform multi-camera geometric calibration once a month (using AprilTag marker board); Temperature sensor calibration (referenced to a blackbody radiation source) is performed weekly; Performance test indicators:

[0061] 4. Typical implementation cases Scenario: Monitoring the Chinese hot dish preparation process (1) Preparation phase testing: When the infrared camera detects that the temperature of the chef's uniform cuff is greater than 60°C, it determines that there may be contact with a heat source and triggers a wrist guard wearing check. The macro camera detects whether a hand is wearing a Band-Aid (using HSV color space + morphological detection).

[0062] (2) Cooking process monitoring: Detect the cooking action frequency (normal range is 2-3 times / second), and abnormal jitter will trigger the pot burning warning; The oil temperature is monitored through thermal imaging, and an alarm for excessive oil smoke is issued when it is greater than 230℃.

[0063] (3) Food delivery compliance inspection: The weighing sensor (integrated at the bottom of the plate) is linked with visual inspection to ensure that the weight error of each dish is less than 5%; Use 3D reconstruction technology to check whether the stacking height of lunch boxes meets safety standards (error < 2cm).

[0064] (4) Technical effect verification data

[0065] 5. Problems and solutions for canteen monitoring 5.1 To address multi-scene interference such as background color confusion, illumination differences, and vapor blur, this paper constructs a multimodal dynamic compensation and spatiotemporal attention system. It uses environmental feature detectors to identify interference types, dynamically adjusts the weights of visible light, infrared, and depth sensors, and combines spatiotemporal Transformers to enhance feature extraction. Specifically, it includes: (1) Multimodal interference perception module to solve the problem of environmental discrimination Deploy environmental feature detectors at the hardware layer to collect real-time: Color similarity: Calculate the HSV distance between the background and the chef's uniform. Formula: ;when It was determined to be color confusion; Light intensity: The sensor obtains the Lux value and distinguishes between daytime (>1000 Lux), dusk (100 - 1000 Lux), and nighttime (<100 Lux); Steam density: Based on the thermal imager temperature gradient ( Determined to be a steam interference area).

[0066] (2) Dynamic modal scheduling network to solve multi-scenario adaptation problems Design a modal attention allocation algorithm to dynamically adjust sensor weights based on interference type: Color confusion scene: reduce the weight of visible light ( ), improve infrared thermal imaging ( ) and depth data ( ) ratio, using the body temperature characteristics (36-37℃) and the three-dimensional structure of the bones to distinguish targets; Lighting Difference Scenarios: HDR + polarization filtering (to suppress stainless steel reflections) is enabled during the day, while "ring fill light (450nm blue light) + infrared enhancement" is switched to at night, using the attention mechanism to enhance edge features in low light conditions. Steam Blur Scene: Pause visible light analysis and activate the "Deep Skeleton Tracking (ST-GCN) + Thermal Imager Motion Contour" dual modality to use temperature gradient to compensate for motion trajectory.

[0067] (3) Spatiotemporal feature enhancement model to solve feature extraction problems Constructing a multi-scale spatiotemporal Transformer: Spatial dimension: Using hierarchical feature alignment, we dynamically register visible light (high detail), infrared (temperature profile), and depth (3D structure) features through deformable convolution. Time dimension: Long short-term temporal memory (LSTM - Transformer) is introduced to learn action sequences within 30 seconds (such as the spatiotemporal logic of the meal preparation process) and suppress transient noise caused by sudden changes in steam and light. Innovation: Adding interference perception loss function ( ), reversely optimize the modal scheduling strategy and adaptively improve the robustness of complex scenarios.

[0068] 5.2 Technical Test Scenario: Western steak process monitoring Special issues: High-temperature oil mist and steam interfere with visible light cameras; System response: When the thermal imager detects that the oil temperature is greater than 200°C, the visible light camera exposure time is automatically reduced; The chef’s hand movements (such as flipping frequency) are tracked through a depth sensor, and the process timing is verified using a finite state automaton (frying time per side: 60±5 seconds).

[0069] Test data results: (1) Improved adaptability to complex scenarios: The recognition accuracy under background color interference increased from 82% to 96%, and the false alarm rate at night was reduced by 40%; (2) Enhanced system robustness: In steam interference scenarios, the behavior recognition accuracy can still be maintained at >90% through multimodal fusion.

Claims

1. A real-time monitoring system for canteen meal distribution compliance based on image recognition, characterized by: The monitoring system includes a hardware layer, a data layer, an algorithm layer, and an application layer connected in sequence; The hardware layer includes heterogeneous sensor arrays (101) and edge computing nodes (102): A heterogeneous sensor array (101) includes a visible light camera, an infrared thermal imager, and a depth sensor, and is configured in a meal preparation operation area; the depth sensor is used to collect three-dimensional depth information of the operation area, construct the spatial position and motion trajectory of the operator, and assist in three-dimensional motion recognition in scenarios such as steam occlusion; An edge computing node (102), communicating with the heterogeneous sensor array (101), and having a built-in multi-task detection model and a timing analysis model; The data layer receives data output by the edge computing node (102) in the hardware layer, and the data layer is provided with a video database, a feature warehouse and a compliance policy library; The algorithm layer includes a behavioral compliance analysis engine (103) for fusing multimodal perception data and verifying the spatiotemporal compliance of operational procedures; The application layer includes a real-time feedback execution module (104), which generates dynamic response instructions by calling predefined rules in the compliance policy library based on the analysis results of the algorithm layer integrating multimodal features.

2. The real-time monitoring system for canteen meal distribution compliance based on image recognition according to claim 1 is characterized in that: The specific configuration of the heterogeneous sensor array (101) further includes: A macro camera deployed above the operating table, with an optical magnification of ≥5×, is used for hand cleanliness detection; A pressure sensor integrated into the handle of the kitchenware, with a sampling frequency of ≥100Hz, is used for grip posture recognition; The weighing module installed at the end of the canteen conveyor belt has a measurement accuracy of ±2g and is used to verify the weight of finished products.

3. The real-time monitoring system for canteen meal distribution compliance based on image recognition according to claim 1 is characterized in that: The multi-task detection model adopts an improved YOLOv8 architecture, including: Shared feature extraction layer (201), using CSPDarknet53 backbone network; The parallel detection branch (202) includes: A clothing compliance detection head (202a) outputs the wearing status of masks, hats, and work clothes; Tool use detection head (202b) to identify the type of kitchenware and the way it is held; The food classification detection head (202c) determines the risk of cross contamination between raw and cooked food.

4. The real-time monitoring system for canteen meal distribution compliance based on image recognition according to claim 3 is characterized in that: The objective function of the improved YOLOv8 architecture is: Among them, λattire is the weight coefficient of dress compliance detection; λtool is the weight coefficient of tool compliance detection; λfood is the weight coefficient of dress compliance; L attire Classification loss for dress compliance testing; L tool The tool uses the bounding box regression loss for compliance detection, L food Classification loss for food classification compliance testing; L pose is the pose constraint loss based on key point coordinates; The constraints are: Among them, P mask is the probability threshold of wearing a mask; P hat is the confidence threshold for chef hat detection; P tool is the probability threshold of the correct tool holding method; the detection head weight is dynamically optimized through the Lagrange multiplier method.

5. The real-time monitoring system for canteen meal distribution compliance based on image recognition according to claim 4 is characterized in that: The timing analysis model includes: A spatiotemporal graph convolutional network constructs a spatiotemporal topological graph of the operator's skeletal key points; Finite state automaton, defining the legal state transition matrix of the standard operating procedure; Dynamic time warping module for flexible time matching of process steps.

6. The real-time monitoring system for canteen meal distribution compliance based on image recognition according to claim 5 is characterized in that: The multi-task detection model also includes an environment adaptation module to solve the recognition problems in the following scenarios: Background color interference suppression: Separate the chef's uniform from the workbench background using the HSV color space, dynamically adjust the detection threshold, and enable infrared thermal imager-assisted positioning when the similarity between the background color and the uniform color is greater than 80%; Day and night illumination compensation: Based on ambient light sensor data, the visible light camera's white balance mode is dynamically switched, and infrared thermal imager data is integrated to improve nighttime recognition accuracy. Steam interference suppression: When the thermal imaging module detects a steam area, it suspends visible light image analysis and prioritizes using depth sensor data to construct a 3D motion trajectory.

7. The real-time monitoring system for canteen meal distribution compliance based on image recognition according to claim 6 is characterized in that: The background color interference suppression in the environment adaptation module specifically includes: Among them, Hattire is the HSV histogram of the clothing area, and Hbg is the HSV histogram of the background area; When Similarity>0.8, infrared-assisted positioning is triggered and thermodynamic features are integrated to enhance target detection; Day and night light compensation specifically includes: Create a light intensity-detection sensitivity mapping table: Dynamically adjust multimodal fusion weights according to real-time lighting data.

8. The real-time monitoring system for canteen meal distribution compliance based on image recognition according to claim 1 is characterized in that: The real-time feedback execution module (104) includes: Augmented reality projection device to mark illegal areas on the operating table; Mechanical interlock controller, electrically connected to the kitchen equipment, with response levels including: Level 1 response: trigger LED indicator to flash; Level 2 response: Activate voice warning and record video footage of the violation; Level 3 response: Cut off the power supply to the designated equipment and send an emergency notification.

9. A monitoring method using the canteen meal distribution compliance real-time monitoring system based on image recognition according to any one of claims 1 to 8, characterized in that: The following steps are involved: S1. Establish a unified spatiotemporal coordinate system by synchronizing the clocks of multiple sensors. S2. Executed in parallel by the edge computing node (102): Visible light image stream processing (S2a): extracting clothing, tools, and food features; Thermodynamic data analysis (S2b): calculation of temperature gradient and rate of change; Depth information processing (S2c): constructing the three-dimensional motion trajectory of the operation area; S3. The behavioral compliance analysis engine (103) at the algorithm layer fuses multimodal features and identifies composite operational behaviors through a spatiotemporal graph convolutional network. S4. Generate dynamic response instructions based on the predefined compliance policy library.

10. The method for real-time monitoring of canteen meal distribution compliance based on image recognition according to claim 9 is characterized in that: The S3 specifically includes: S3a. Divide the continuous operation into time windows; S3b. Extract spatiotemporal feature vectors for each window; S3c. Use the improved DTW algorithm to match the standard process template; S3d. When the matching degree is lower than the threshold θ, a process abnormality event is triggered.

11. The method for real-time monitoring of canteen meal distribution compliance based on image recognition according to claim 9 is characterized in that: The dynamic response instruction generation method in S4 includes: Calculate the environment complexity factor: ； Where α = 0.15, β = 0.08, N_people is the number of operators, and N_dishes is the number of dishes processed simultaneously; Dynamically adjust the detection sensitivity according to the γ value: 。 12. The method for real-time monitoring of canteen meal distribution compliance based on image recognition according to claim 9 is characterized in that: The monitoring method further includes a knowledge base updating step S5: When collecting new violation samples, start the incremental learning process: Freeze backbone network parameters; Only fine-tune the fully connected layers of the detection head; Update the state transition rules of the compliance policy library.

Images