Intelligent evaluation method and system for large hot pot catering service based on deep learning

Abstract

The application provides a large hot pot catering service intelligent evaluation method and system based on deep learning, which realizes real-time target detection and event recognition on a video stream by using a YOLO series deep learning model, accurately calculates service time consumption in combination with a POS system timestamp, and automatically compares a timeout threshold value, simultaneously analyzes the color type and use order of a cleaning towel based on continuous frame timing to determine operation standardization, thereby effectively solving the technical problem that there is a lack of real-time, automatic and quantitative monitoring of the whole process (such as pot cooking time, dish serving time, table collecting efficiency and cleaning standardization) of large hot pot catering service in the prior art, realizing two-dimensional intelligent evaluation of service efficiency and operation quality, and significantly improving the real-time and accuracy of supervision.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring technology, specifically to a method and system for intelligent evaluation of large-scale hot pot catering services based on deep learning. Background Technology

[0002] With the rapid development of the catering industry, large-scale hot pot restaurants, characterized by high customer traffic, numerous service stages, and fast-paced operations, face higher demands for service quality management. Currently, service quality assessment in large hot pot restaurants primarily relies on manual inspections, customer satisfaction surveys, and simple POS system data statistics. Manual inspections suffer from limitations such as narrow coverage, high randomness, difficulty in identifying service problems in real time, and susceptibility to subjective influences. Customer satisfaction surveys are reactive and cannot intervene or correct issues immediately after service occurs. Existing POS systems only record order placement times and lack objective, quantitative monitoring methods for the actual service process (such as speed of serving hot pot, timeliness of food delivery, and efficiency of table clearing), making it difficult for management to accurately pinpoint service bottlenecks. Furthermore, for details such as the proper use of cleaning towels (e.g., color differentiation, usage order), existing technology completely lacks effective automated monitoring solutions, often relying on employee self-discipline or post-event spot checks, making it difficult to guarantee the effective implementation of standardized procedures.

[0003] Therefore, a major technical problem in the existing technology is the lack of a method that can quantitatively evaluate the entire process of large hot pot catering services (including the timeliness of serving the pot, the timeliness of serving the dishes, the standardization of clearing the table, and the cleaning operation standards) in real time and automatically based on video streams. This results in low service efficiency, high supervision costs, and difficulty in continuously optimizing service quality through data-driven approaches. Summary of the Invention

[0004] This invention aims to provide a method and system for intelligent evaluation of large-scale hot pot catering services based on deep learning, which realizes intelligent evaluation of service efficiency and operational quality in two dimensions, and significantly improves the real-time and accuracy of supervision.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: a deep learning-based intelligent evaluation method for large-scale hot pot catering services, comprising: Video streams are captured by cameras deployed in the lobby of the hot pot restaurant, and the video streams are preprocessed and then input into the YOLO series deep learning models. The YOLO series deep learning models are used to perform target detection on video frames, identifying at least one target among hot pot pots, food containers, waiters, customers, and cleaning towels. Based on the spatiotemporal relationship of the targets in consecutive video frames, the event of serving food to the pot, the event of serving the first tray of food, or the event of clearing the table are identified. Obtain the order placement timestamp recorded by the POS system, and combine it with the timestamp of the identified cooking event or the first tray of food being served to calculate the actual service time. Compare the actual service time with a preset timeout threshold to determine whether a timeout has occurred. During the identified table clearing event, the YOLO series deep learning model is used to identify the color of the cleaning towels, record the time of first appearance of each color towel and the order of use, and compare the order of use with the preset standard cleaning process to determine the compliance of the operation. Based on the timeout result and the operational compliance result, a two-dimensional service evaluation report is generated. When a timeout or non-compliance is detected, video footage is captured as evidence and an alarm notification is pushed.

[0006] Preferably, the camera is installed at a height between 2.5m and 3.5m, and is tilted at an angle of 30° to 60° for side and top coverage. The video stream resolution is not less than 1080P, the frame rate is between 15 and 25 FPS, and the H.265 encoding format is used.

[0007] Preferably, the training dataset of the YOLO series deep learning model consists of approximately 30,000 to 50,000 labeled images, covering different store environments, lighting conditions, and customer flow densities, and performs fine boundary annotations on key targets; the model outputs the category label, bounding box position information, and confidence score of each target, with a long-tail compensation strategy used for small target detection.

[0008] Preferably, the rule for determining the hot pot serving event is as follows: when the hot pot is detected to have moved from the waiter's bounding box area to the table's bounding box area, and the pot remains stably in the table area for more than 3 seconds for a preset time, the hot pot serving event is determined to be completed. The rule for determining the first tray serving event is as follows: when it is detected that the food container has moved from the waiter's bounding box area to the table's bounding box area, and the food container is the first food container to appear in the current order, the first tray serving event is determined to be completed.

[0009] Preferably, the preset timeout threshold is set with different standards depending on the service stage: The timeout threshold for serving hot pot base after ordering is 8 minutes, and the timeout threshold for serving the first tray of dishes after ordering is 10 minutes. At the same time, it checks whether the order of serving dishes and hot pot base is consistent with the standard service process. If the order is abnormal, the order abnormal field is set to true.

[0010] Preferably, the table clearing reminder time adopts a time-segmented dynamic threshold strategy: the threshold is 8 minutes during peak dining hours, 15 minutes during off-peak hours, and no assessment is conducted during nighttime hours.

[0011] Preferably, the cleaning towels include blue towels, yellow towels, and white towels; the process for determining the compliance of the operation includes: The system checks whether the number and types of towels held by the server meet the requirements, and whether the order in which the towels are used conforms to the preset standard procedure of blue, yellow, and white. If towels of the same color are used consecutively or in a skipped or reversed order, it is marked as an abnormal sequence event.

[0012] Preferably, to compensate for missed detections and false detections in towel recognition, a multi-frame temporal smoothing strategy and a short-time trajectory completion method are used: When a towel is not detected in a certain frame but the towels in the preceding and following frames are of the same color, interpolation compensation is performed within a set frame range; at the same time, confidence filtering and minimum continuous frame constraints are combined to eliminate instantaneous false detection results, and the recognition stability is improved by constraining the waiter's hand area.

[0013] Preferably, the start of the table clearing event is determined when: after a customer is detected leaving the table area, the waiter enters the table area and the target number of tableware / pots in the table area continues to decrease; The termination of the table clearing event is determined when all tableware / pots in the table area have disappeared and the waiter has completed the cleaning action with a towel, and the model outputs the empty table category for three consecutive frames.

[0014] On the other hand, this invention proposes a large-scale intelligent evaluation system for hot pot catering services based on deep learning, comprising: The data acquisition module is used to acquire video streams through cameras deployed in the lobby of the hot pot restaurant. The video streams are preprocessed and then input into the YOLO series deep learning models. The event recognition module is used to perform target detection on video frames using the YOLO series deep learning models, identify at least one target among hot pot pot, food container, waiter, customer and cleaning towel, and identify the hot pot serving event, the first tray serving event or the table clearing event based on the spatiotemporal relationship of the targets in the continuous video frames. The efficiency evaluation module is used to obtain the order placement timestamp recorded by the POS system, and combine it with the timestamp of the cooking event or the first tray of food event identified by the event recognition module to calculate the actual service time. The actual service time is then compared with a preset timeout threshold to determine whether a timeout has occurred. The standardization assessment module is used to identify the color of the cleaning towels through the YOLO series deep learning model during the table clearing event identified by the event recognition module, record the time node of the first appearance of each color towel and the order of use, and compare the order of use with the preset standard cleaning process to determine the standardization of operation. The alarm management module generates a two-dimensional service evaluation report based on the timeout results determined by the efficiency evaluation module and the operational compliance results determined by the compliance evaluation module. When a timeout or operational non-compliance is detected, video footage is captured as evidence and an alarm notification is pushed.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention utilizes YOLO series deep learning models for real-time target detection and event recognition of video streams, combines POS system timestamps to accurately calculate service time consumption and automatically compares it with timeout thresholds, and analyzes the color types and usage order of cleaning towels based on continuous frame time sequence to determine operational compliance. This effectively solves the technical problem of lacking real-time, automated, and quantitative monitoring of the entire process of large hot pot catering services (such as pot serving time, food serving time, table clearing efficiency, and cleaning standards) in existing technologies. It achieves dual-dimensional intelligent evaluation of service efficiency and operational quality, significantly improving the real-time nature and accuracy of supervision. Attached Figure Description

[0016] Figure 1 The flowchart of the intelligent evaluation method for large-scale hot pot catering services based on deep learning is shown in this invention. Figure 2 This is a block diagram of the intelligent evaluation system for large-scale hot pot catering services based on deep learning, as described in this invention. Detailed Implementation

[0017] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0018] like Figure 1 As shown, this invention proposes a deep learning-based intelligent evaluation method for large-scale hot pot restaurant services. It collects video streams using cameras deployed in the restaurant's lobby, utilizes YOLO deep learning models to perform object detection and event recognition on the video frames, and combines this with order timestamps recorded by the POS system to calculate the actual service time and determine timeouts. Furthermore, it compensates for missed and false detections in the cleaning towel identification process using a multi-frame temporal smoothing strategy and a short-term trajectory completion method. Finally, it generates a two-dimensional service evaluation report, specifically including: Video streams are captured by cameras deployed in the lobby of the hot pot restaurant. After preprocessing, the video streams are input into the YOLO series deep learning models. The camera installation height is controlled between 2.5m and 3.5m, and it covers the upper side at an angle of 30° to 60°. The video stream resolution is no less than 1080P, the frame rate is between 15 and 25 FPS, and the H.265 encoding format is used.

[0019] Furthermore, the training dataset for the YOLO series of deep learning models consists of approximately 30,000 to 50,000 labeled images, covering different store environments, lighting conditions, and customer flow densities, and providing fine-grained boundary annotations for key targets. The model outputs the category label, bounding box location information, and confidence score for each target, with a long-tail compensation strategy employed for small target detection.

[0020] By optimizing the camera installation angle (2.5m-3.5m height, 30°-60° tilt) and high-resolution encoding, the obstruction caused by hot pot smoke and customer flow was effectively avoided. Combined with multi-scenario large-sample training and small target long-tail compensation strategy, the detection accuracy and robustness of key targets such as towels and dishes in complex dynamic environments were significantly improved. At the same time, relying on the real-time inference capability of the YOLO model, millisecond-level accurate determination of service time and operation specifications was achieved.

[0021] In this embodiment, the YOLO series deep learning model (YouOnlyLookOnce) is the core perception engine. It adopts a one-stage object detection architecture, which can directly perform end-to-end regression prediction on images without the need for candidate region generation (such as RPN), thereby achieving extremely high real-time performance and meeting the real-time processing requirements of the hot pot restaurant's lobby video stream (15-25FPS).

[0022] 1. Specific model construction process: The YOLO model building process used in this embodiment includes four key steps: data preparation, network architecture construction, loss function design, and training strategy.

[0023] 1.1 Dataset Construction and Augmentation: Data scale: Construct a training set containing 30,000-50,000 labeled images, covering different store environments (lighting, decoration), lighting conditions (daytime / nighttime / dim lighting) and customer flow density scenarios.

[0024] Category definition: Label the set of categories These correspond to hot pot utensils, food containers, waiters, customers, and cleaning towels, respectively.

[0025] Data augmentation: To improve model robustness, Mosaic augmentation, random scaling, color jitter (HSV transformation), and background blending techniques are introduced during training.

[0026] Mosaic enhancement formula: stitches four random images into one to simulate multi-scale target distribution. ;in These are four randomly selected training images.

[0027] 1.2 Network Architecture Design (Backbone + Neck + Head): The model adopts a typical "trunk-neck-head" structure, and is constructed using YOLOv8 or YOLOv7 as examples: 1. Backbone (Feature Extraction Network): Employs an improved CSPDarknet or EfficientNet as the backbone network, responsible for extracting multi-level feature maps from the input image. It progressively reduces spatial resolution and increases the number of channels through downsampling operations (stride=2), capturing features ranging from edge texture to semantic information.

[0028] Output feature layers at three scales: They are used to detect small, medium, and large targets, respectively.

[0029] 2. Neck (Feature Fusion Network): Employs PANet (Path Aggregation Network) or BiFPN (Bidirectional Feature Pyramid Network) structures. Through top-down and bottom-up bidirectional path aggregation, it fuses deep semantic features with shallow spatial detail features, addressing the scale variation problem of food items (small targets) and people (multi-scale) in the hot pot scene.

[0030] 3. Head (Detection Head): Detection heads are deployed in parallel on the feature maps at each scale. Each detection head outputs a tensor with dimensions (...). ),in The number of anchor boxes is 5, where 5 represents (x, y, w, h, conf), and C is the number of categories.

[0031] 1.3 Loss Function Design: To simultaneously optimize bounding box location, confidence score, and classification results, a combined loss function is employed. : ; Bounding box loss Using CIoULoss (CompleteIoU) considers not only the overlap area, but also the center point distance and aspect ratio consistency. ; in: These are the ground truth bounding boxes and the predicted bounding boxes, respectively. is the Euclidean distance function; c is the diagonal length; v measures the consistency of the aspect ratio; For balancing parameters.

[0032] Confidence loss Use Binary Cross Entropy (BCE) or Focal Loss to distinguish foreground from background and suppress the weights of simple samples. ; in To predict confidence levels; is the adjustment factor (taken as 2); N is the total number of grid cells or anchor frames; i is the index of each grid cell or anchor frame traversed; This is an indicator function that takes the value 1 when the target actually exists at the i-th position, and 0 otherwise; This is an indicator function that takes the value 1 when there is no target (background) at the i-th position, and 0 otherwise.

[0033] Classification loss Calculate the category prediction error using BCELoss: ; In the formula, C is the total number of categories in the dataset; j is the index of each category. The true label for the j-th category; This represents the probability that the j-th category predicted by the model belongs to this category, with a value in the range [0,1].

[0034] 1.4 Training Strategies Optimizer: SGD or AdamW is used, along with a cosine annealing learning rate scheduling strategy.

[0035] Pre-trained weights: Load weights pre-trained on the COCO dataset for transfer learning to accelerate convergence.

[0036] Long-tail compensation: To address the issue of low frequency occurrence of certain targets (such as towels of a specific color) in the hot pot scenario, class weights are introduced into the loss function. : ; in This represents the number of samples of this category in the training set.

[0037] 2. Model Working Principle The core idea of ​​the YOLO model is to transform object detection into a regression problem. For an input image, the model only needs to look at it once to directly predict the category and location of all objects.

[0038] 2.1 Forward reasoning process; 1. Input preprocessing: processing the raw video frames... Resize to a fixed size (e.g., 640×640) and normalize to obtain the result. .

[0039] 2. Feature extraction: Multi-scale feature maps are generated by sequentially passing through the Backbone and Neck. .

[0040] 3. Grid Prediction: The model divides the image into S×S grid cells. Each grid cell is responsible for predicting the properties if the center of an object falls within that grid cell.

[0041] For each anchor box, the model predicts: bounding box offset ; Object confidence (conf) (indicates the probability that the bounding box contains an object); Category Probability .

[0042] Example of a prediction formula (using the coordinates of the center point as an example): ; ; in The coordinates of the top left corner of the grid; Preliminary dimensions for the anchor frame; This represents the regression bias value of the network output relative to the grid center. The Sigmoid activation function restricts the output to the range (0,1), ensuring that the center point falls within the current grid. The predicted bounding box width and height; This represents the regression bias value of the aspect ratio output by the network.

[0043] 2.2 Post-processing and Non-maximum Suppression (NMS): The model outputs a large number of raw predicted bounding boxes, many of which are duplicated, requiring post-processing. 1. Confidence filtering: Remove items with a confidence level below the threshold. (e.g., a prediction box of 0.5).

[0044] 2. Non-maximum suppression (NMS): Sort the remaining boxes by confidence level.

[0045] Select the box with the highest confidence level and calculate its intersection-over-union (IoU) with the remaining boxes.

[0046] If the IoU between a certain bounding box and the highest bounding box is greater than the threshold If the value is 0.45, the box is removed (considered as a duplicate detection of the same target).

[0047] Repeat the above steps until all boxes have been processed.

[0048] ; In the formula, The intersection-union ratio (IUGR) measures the degree of overlap between two bounding boxes, A (predicted box) and B (ground truth box). This represents the predicted bounding box region. This represents the actual bounding box region.

[0049] 2.3 Spatiotemporal event recognition logic: After detecting the target, the system combines information from consecutive frames to determine the event: Trajectory association: Kalman filtering or the Hungarian algorithm is used to associate target IDs in adjacent frames to form motion trajectories. . The complete set of spatiotemporal trajectories of the k-th food tray (or specific target); A unique identifier (ID) for the target instance, used to distinguish different instances; This is a timestamp, representing the time point of a video frame; Let be the position state of the k-th tray on the image plane at time t.

[0050] State machine judgment: The incident involving the cooking pot: When the pot's trajectory... from Enter and the length of stay Triggered at any time.

[0051] Table clearing incident: When the number of tableware in the dining area It continues to decrease and eventually reaches zero, triggered when the waiter holds a towel.

[0052] Through the above construction process and working principle, the YOLO model can efficiently and accurately identify complex dynamic scenarios in hot pot restaurants, providing a reliable data foundation for subsequent service time calculation and standardization assessment.

[0053] Using YOLO series deep learning models to perform target detection on video frames, the system identifies at least one target among hot pot pots, food containers, waiters, customers, and cleaning towels. Based on the spatiotemporal relationship of targets in consecutive video frames, the system identifies events such as the pot being served, the first tray being served, or the table being cleared. The rule for determining the hot pot event is as follows: when the hot pot is detected to have moved from the waiter's bounding box area to the table's bounding box area, and the pot remains stably in the table area for more than 3 seconds for a preset time, the hot pot event is determined to be completed. The rule for determining the first tray serving event is as follows: when it is detected that the food container has moved from the waiter's bounding box area to the table's bounding box area, and the food container is the first food container to appear in the current order, the first tray serving event is determined to be completed.

[0054] By constructing a spatiotemporal transfer logic of "pot / food container - waiter - table," abstract service actions are transformed into quantifiable and precise events. Utilizing area transfer judgment and dwell time thresholds (>3s), false alarms caused by waiters carrying items or briefly placing them are effectively filtered out, ensuring the accuracy of pot-serving event judgment. Simultaneously, an order-level first-appearance mechanism is introduced to accurately pinpoint the first tray of food served as a key service node, solving the problem of traditional monitoring failing to distinguish between single servings and the first dish of an entire order. This multi-objective association-based temporal reasoning method not only achieves comprehensive capture of key service process nodes but also provides a highly reliable data source for the automation and standardization of subsequent service time calculations.

[0055] Obtain the order placement timestamp recorded by the POS system, and combine it with the timestamp of the identified cooking event or the first tray of food being served to calculate the actual service time. Compare the actual service time with the preset timeout threshold to determine whether a timeout has occurred. Specifically, the preset timeout thresholds are set with different standards depending on the service stage: the timeout threshold for the hot pot base after ordering is 8 minutes, and the timeout threshold for the first tray of dishes after ordering is 10 minutes; at the same time, it checks whether the order of dishes and hot pot base served is consistent with the standard service process, and if the order is abnormal, the order abnormal field is set to true.

[0056] The table clearing reminder time adopts a time-based dynamic threshold strategy: the threshold is 8 minutes during peak dining hours, 15 minutes during off-peak hours, and no assessment is conducted during nighttime hours.

[0057] The start of a table clearing event is determined when a customer leaves the table area, a waiter enters the table area, and the number of tableware / pots in the table area continues to decrease; the end of a table clearing event is determined when all tableware / pots in the table area disappear and the waiter completes the cleaning action with a towel, and the model outputs the empty table category for three consecutive frames.

[0058] By integrating multi-source data (POS orders and visual events), the system achieves precise quantification of service time. Utilizing differentiated threshold strategies (8 minutes for hot pot base / 10 minutes for dishes) and dynamic table clearing assessments (8 minutes during peak hours / 15 minutes during off-peak hours / no assessment at night), it effectively adapts to the varying levels of activity at different times and stages, significantly improving the fairness and rationality of the assessment. Simultaneously, a process sequence verification mechanism is introduced to automatically identify and mark abnormal operations such as serving dishes before the hot pot. This not only constructs a closed-loop monitoring system covering the entire chain from order placement to table service and clearing, but also addresses pain points in manual supervision such as inconsistent standards, delayed recording, and blind spots in nighttime management through objective data-driven approaches, achieving refined control over service quality.

[0059] During the identified cleaning events, the YOLO series deep learning model identifies the color of the cleaning towels, records the first appearance time and usage order of each color towel, and compares the usage order with the preset standard cleaning process to determine the compliance of the operation. Cleaning towels include blue towels, yellow towels, and white towels; the process of determining operational compliance includes: checking whether the number and type of towels held by the server meet the requirements, and checking whether the order in which the towels are used conforms to the preset standard procedure of blue, yellow, and white; if there are cases of continuous use of towels of the same color or skipping steps or reversing the order, it is marked as an abnormal sequence event.

[0060] To address missed and false detections in towel recognition, a multi-frame temporal smoothing strategy and a short-term trajectory completion method are employed for compensation: when a towel is not detected in a certain frame but the towels in the preceding and following frames are of the same color, interpolation compensation is performed within a set frame range; at the same time, confidence filtering and minimum continuous frame constraints are combined to eliminate instantaneous false detection results, and the recognition stability is improved by constraining the waiter's hand area.

[0061] By comparing color classification with temporal logic, abstract cleaning specifications are transformed into quantifiable blue-yellow-white sequence verification, accurately identifying violations such as mixing of the same color, skipping steps, or reversing the order, effectively eliminating the risk of cross-contamination during cleaning. Simultaneously, multi-frame temporal smoothing and trajectory completion strategies are introduced, combined with confidence filtering and hand area constraints, significantly resolving the issues of missed and false detections caused by occlusion and jitter in video streams, greatly improving the robustness of small targets (towels) in dynamic scenes. This mechanism achieves refined control from result acceptance to process compliance, providing highly reliable data support for service standardization.

[0062] Based on the timeout result and the operational compliance result, a two-dimensional service evaluation report is generated. When a timeout or non-compliance is detected, video footage is captured as evidence and an alarm notification is pushed.

[0063] The dual-dimensional evaluation model creates a comprehensive profile of service quality, upgrading single-dimensional monitoring to a comprehensive performance evaluation. By utilizing automatic evidence screenshotting and real-time alarm mechanisms, a closed-loop management process of discovery, evidence collection, and feedback has been constructed. This not only solves the problems of delayed manual review and difficulty in tracing responsibility, but also reduces communication costs through visualized evidence, prompting immediate rectification of problems, thereby significantly improving the response speed and execution of service management.

[0064] On the other hand, this invention proposes a large-scale intelligent evaluation system for hot pot catering services based on deep learning, such as... Figure 2 As shown, it includes: The data acquisition module is used to acquire video streams through cameras deployed in the lobby of the hot pot restaurant. The video streams are preprocessed and then input into the YOLO series deep learning models. The event recognition module is used to perform target detection on video frames using the YOLO series deep learning models, identify at least one target among hot pot pot, food container, waiter, customer and cleaning towel, and identify the hot pot serving event, the first tray serving event or the table clearing event based on the spatiotemporal relationship of the targets in the continuous video frames. The efficiency evaluation module is used to obtain the order placement timestamp recorded by the POS system, and combine it with the timestamp of the cooking event or the first tray of food event identified by the event recognition module to calculate the actual service time. The actual service time is then compared with a preset timeout threshold to determine whether a timeout has occurred. The standardization assessment module is used to identify the color of the cleaning towels through the YOLO series deep learning model during the table clearing event identified by the event recognition module, record the time node of the first appearance of each color towel and the order of use, and compare the order of use with the preset standard cleaning process to determine the standardization of operation. The alarm management module generates a two-dimensional service evaluation report based on the timeout results determined by the efficiency evaluation module and the operational compliance results determined by the compliance evaluation module. When a timeout or operational non-compliance is detected, video footage is captured as evidence and an alarm notification is pushed.

[0065] Furthermore, the aforementioned modules, during execution, are also used to implement other steps of the aforementioned deep learning-based intelligent evaluation method for large-scale hot pot catering services, including: 1. Data acquisition and preprocessing; The system deploys high-definition cameras in key areas of the hot pot restaurant's main hall (such as the food delivery area and dining area), capturing images from the side and above at a height of 2.5-3.5m and an angle of 30°-60°. The captured video stream V(t) has a resolution of 1080P and a frame rate of [missing information]. It is 15-25-FPS and the encoding format is H.265.

[0066] The raw video stream undergoes preprocessing, including denoising, brightness equalization, and normalization, before being converted into a model input tensor. For the t-th frame image, its normalization formula is: Where: x, y are pixel coordinates (dimensionless); For color channels; These are the mean and standard deviation of the channel, respectively (unit: gray level). The normalized pixel value (range mapped to [0,1]).

[0067] 2. Object detection and feature extraction using YOLO series deep learning models; The preprocessed video stream is input into the YOLO series deep learning model (denoted as YOLO). The model's training dataset consists of 30,000 to 50,000 labeled images, covering scenes with varying lighting and customer density. The model performs real-time inference on video frames, identifying targets such as hot pot utensils, food containers, waiters, customers, and cleaning towels.

[0068] For the t-th frame image, the model outputs a set of detection results. , is represented as: ; in: The total number of targets detected in frame t; For the target category label, ; The coordinates of the target bounding box; The target confidence score.

[0069] For small targets (such as distant dishes or small towels), the model introduces a long-tail compensation strategy, which corrects the predicted probability. The calculation formula is: ; in: This represents the original classification output by the model; K is the total number of categories. This is the compensation coefficient for small targets (preset value, such as 0.5). As an indicator function, when the target area The value is 1 if the condition is met, and 0 otherwise. This is the threshold for the small target area.

[0070] 3. Spatiotemporal event identification and service time calculation; The system identifies "cooking event", "first tray serving event" or "table clearing event" based on the spatiotemporal relationship of targets in continuous video frames, and calculates the actual service time by combining the order timestamp recorded by the POS system.

[0071] 3.1 Extraction of event timestamps; Let the order ID recorded by the POS system be... Its order timestamp is .

[0072] The system uses an association algorithm to match events detected in the video with orders. If the "cooking event" is detected as complete, a timestamp is recorded. If the "first tray serving event" is detected as complete, record the timestamp. .

[0073] The logic for extracting event timestamps is as follows: ; in: The video frame number that triggered the event; This refers to the video frame interval time. This is the calibration offset of the video stream relative to the POS system time.

[0074] 3.2 Calculation of actual service time; Calculate the actual service time based on the identified event type. : ; in The unit is seconds.

[0075] 3.3 Timeout Decision Logic; The system sets a preset timeout threshold. (Unit: s): Pot bottom loading threshold: (8 minutes); First tray serving threshold: (10 minutes)

[0076] The logical expression for determining whether a timeout has occurred is: ; Furthermore, define the timeout deviation ratio. Used to quantify the degree of delay: ; like If the timeout occurs, the order is deemed to have timed out at that stage.

[0077] 4. Compensation for missed and false detections in the identification of cleaning towels during the table clearing incident; During the identified collection event, the system needs to identify the color type (blue, yellow, white) and usage order of the cleaning towels. Due to factors such as occlusion and changes in lighting, missed detections (not detected) or false detections (incorrect color identification) are prone to occur. To address this, a multi-frame temporal smoothing strategy and a short-time trajectory completion method are employed for compensation.

[0078] 4.1 Multi-frame timing smoothing strategy; Let the detected towel color sequence in frame t be... The values ​​are: 0 indicates not detected, 1 is blue, 2 is yellow, and 3 is white.

[0079] Define the smoothing window size as W (e.g., W = 5 frames). For the current frame t, if... Then, interpolation compensation is performed using information from the preceding and following frames:

[0080] in The mode function selects the non-zero color that appears most frequently within the window as the compensation result. This operation ensures the continuity of color recognition under brief occlusion.

[0081] 4.2 Short-time trajectory completion method; To address trajectory breaks caused by missed detections in a single frame, a short-term trajectory completion model is constructed. Let the waiter's hand area be... The detected trajectory of the center point of the towel is ,in These are spatial coordinates.

[0082] If within a certain time interval A series of L frames were missed (i.e., And it meets the following conditions: 1. Complete the previous time step and the completed time The same color was detected in all of them. ; 2. During the replenishment process, the waiter's hand area It always exists and its movement trajectory is continuous; 3. Complete the duration Less than the maximum allowed interruption time (e.g., 0.5s).

[0083] Then perform trajectory completion and mark the missing frames with color. : ; Simultaneously, combined with confidence filtering, only when the confidence of adjacent frames... Completion is only performed when the value is 0.7 (e.g., to eliminate momentary false detection interference).

[0084] 4.3 Standard process comparison; The compensated color sequence is denoted as .

[0085] Standard cleaning process is defined as (Blue → Yellow → White).

[0086] Determine the number of violations of operational standards : ; The penalty function Defined as:

[0087] like If so, the operation is deemed to be non-standard.

[0088] 5. Comprehensive assessment and alarm generation; Based on the timeout result IsTimeout calculated above and the operational compliance result... Generate a two-dimensional service evaluation report.

[0089] Efficiency rating (like Then take 0); Standardized scoring .

[0090] Final overall score ,in The weighting coefficients and .

[0091] When IsTimeout=True} or At that time, the system captures the current frame. and before and after The frames are used to create evidence images / video clips, and alarm notifications are pushed to the management terminal via the network.

[0092] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A deep learning-based intelligent evaluation method for large-scale hot pot restaurant services, characterized in that, include: Video streams are captured by cameras deployed in the lobby of the hot pot restaurant, and the video streams are preprocessed and then input into the YOLO series deep learning models. The YOLO series deep learning models are used to perform target detection on video frames, identifying at least one target among hot pot pots, food containers, waiters, customers, and cleaning towels. Based on the spatiotemporal relationship of the targets in consecutive video frames, the event of serving food to the pot, the event of serving the first tray of food, or the event of clearing the table are identified. Obtain the order placement timestamp recorded by the POS system, and combine it with the timestamp of the identified cooking event or the first tray of food being served to calculate the actual service time. Compare the actual service time with a preset timeout threshold to determine whether a timeout has occurred. During the identified table clearing event, the YOLO series deep learning model is used to identify the color of the cleaning towels, record the time of first appearance of each color towel and the order of use, and compare the order of use with the preset standard cleaning process to determine the compliance of the operation. Based on the timeout result and the operational compliance result, a two-dimensional service evaluation report is generated. When a timeout or non-compliance is detected, video footage is captured as evidence and an alarm notification is pushed.

2. The intelligent evaluation method for large-scale hot pot catering services based on deep learning according to claim 1, characterized in that, The camera is installed at a height between 2.5m and 3.5m and is tilted at an angle of 30° to 60° for side and top coverage. The video stream resolution is no less than 1080P, the frame rate is between 15 and 25 FPS, and the H.265 encoding format is used.

3. The intelligent evaluation method for large-scale hot pot catering services based on deep learning according to claim 1, characterized in that, The training dataset for the YOLO series deep learning models consists of 30,000 to 50,000 labeled images, covering different store environments, lighting conditions, and customer flow densities, and provides fine-grained boundary annotations for key targets. The model outputs the category label, bounding box location information, and confidence score for each target, with a long-tail compensation strategy used for small target detection.

4. The intelligent evaluation method for large-scale hot pot catering services based on deep learning according to claim 1, characterized in that, The rule for determining the hot pot serving event is as follows: when the hot pot is detected to have moved from the waiter's bounding box area to the table's bounding box area, and the pot remains stably in the table area for more than 3 seconds for a preset time, the hot pot serving event is determined to be completed. The rule for determining the first tray serving event is as follows: when it is detected that the food container has moved from the waiter's bounding box area to the table's bounding box area, and the food container is the first food container to appear in the current order, the first tray serving event is determined to be completed.

5. The intelligent evaluation method for large-scale hot pot catering services based on deep learning according to claim 1, characterized in that, The preset timeout threshold is set with different standards depending on the service stage: The timeout threshold for serving hot pot base after ordering is 8 minutes, and the timeout threshold for serving the first tray of dishes after ordering is 10 minutes. At the same time, it checks whether the order of serving dishes and hot pot base is consistent with the standard service process. If the order is abnormal, the order abnormal field is set to true.

6. The intelligent evaluation method for large-scale hot pot catering services based on deep learning according to claim 1, characterized in that, The reminder timing for the closing of tables adopts a time-segmented dynamic threshold strategy: the threshold is 8 minutes during peak dining hours, 15 minutes during off-peak hours, and no assessment is conducted during nighttime hours.

7. The intelligent evaluation method for large-scale hot pot catering services based on deep learning according to claim 1, characterized in that, The cleaning towels include blue towels, yellow towels, and white towels; The process for determining the compliance of operations includes: The inspection checks whether the types and quantities of towels held by the waiters meet the requirements, and whether the order in which the towels are used conforms to the preset standard procedure of blue, yellow, and white. If towels of the same color are used consecutively, or if there are skipped steps or reversed order, it is marked as an abnormal sequence event.

8. The intelligent evaluation method for large-scale hot pot catering services based on deep learning according to claim 1, characterized in that, To address the issues of missed and false detections in towel recognition, a multi-frame temporal smoothing strategy and a short-time trajectory completion method are employed for compensation. When a towel is not detected in a certain frame but the towels in the preceding and following frames are of the same color, interpolation compensation is performed within a set frame range; at the same time, confidence filtering and minimum continuous frame constraints are combined to eliminate instantaneous false detection results, and the recognition stability is improved by constraining the waiter's hand area.

9. The intelligent evaluation method for large-scale hot pot catering services based on deep learning according to claim 1, characterized in that, The start of the table clearing event is determined when: after a customer leaves the table area, the waiter enters the table area and the target number of tableware / pots in the table area continues to decrease; The termination of the table clearing event is determined when all tableware / pots in the table area have disappeared and the waiter has completed the cleaning action with a towel, and the model outputs the empty table category for three consecutive frames.

10. A deep learning-based intelligent evaluation system for large-scale hot pot catering services, used to implement the method as described in any one of claims 1-9, characterized in that, include: The data acquisition module is used to acquire video streams through cameras deployed in the lobby of the hot pot restaurant. The video streams are preprocessed and then input into the YOLO series deep learning models. The event recognition module is used to perform target detection on video frames using the YOLO series deep learning models, identify at least one target among hot pot pot, food container, waiter, customer and cleaning towel, and identify the hot pot serving event, the first tray serving event or the table clearing event based on the spatiotemporal relationship of the targets in the continuous video frames. The efficiency evaluation module is used to obtain the order placement timestamp recorded by the POS system, and combine it with the timestamp of the cooking event or the first tray of food event identified by the event recognition module to calculate the actual service time. The actual service time is then compared with a preset timeout threshold to determine whether a timeout has occurred. The standardization assessment module is used to identify the color of the cleaning towels through the YOLO series deep learning model during the table clearing event identified by the event recognition module, record the time node of the first appearance of each color towel and the order of use, and compare the order of use with the preset standard cleaning process to determine the standardization of operation. The alarm management module generates a two-dimensional service evaluation report based on the timeout results determined by the efficiency evaluation module and the operational compliance results determined by the compliance evaluation module. When a timeout or operational non-compliance is detected, video footage is captured as evidence and an alarm notification is pushed.

Images