Dining table state recognition method and system based on image recognition

Abstract

The invention relates to a dining table state recognition method and system based on image recognition. The method comprises the following steps: acquiring dining table image data; the dining table image data carries human body information and food information; performing food state analysis on the dining table image data based on deep learning image classification to obtain food state information; according to a human body posture estimation technology, dining state analysis is carried out on the dining table image data to obtain dining state information, and the dining state information comprises whether dining is carried out, the number of diners and whether leaving is carried out; and performing dining table state evaluation by combining the food state information and the dining state information to obtain a current dining table state. According to the method, the dining table image data containing the human body and food information is acquired, the food state and the dining state are intelligently analyzed by means of deep learning and human body posture estimation technologies, then the current dining table state is comprehensively evaluated, efficient and reliable technical guarantee is provided for intelligent management and optimization service of a restaurant, and the operation efficiency of the restaurant and the customer experience are improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer and communication technology, and in particular to a table status recognition method and system based on image recognition. Background Art

[0002] In the catering industry, table status assessment is crucial for table turnover in the catering industry. Table turnover refers to the restaurant quickly cleaning and re-preparing the table after the customers finish their meal so that the next batch of customers can sit down. However, in traditional table status assessment methods, most of them rely on manual inspections. Restaurant staff need to regularly shuttle between various areas of the restaurant, observe the actual situation of each table, and then manually record the current table status to infer whether to turn the table over. Although this method is intuitive and easy to operate, manual inspections consume a lot of manpower and increase the restaurant's operating costs. In addition, the manual observation and judgment method has certain subjectivity and inaccuracy, which is prone to misjudgment. In addition, manual inspections cannot achieve real-time monitoring, and there is a delay in information feedback, which may cause customers to wait too long or service resources are not allocated in time, seriously affecting the restaurant's operating efficiency and customer satisfaction. Summary of the invention

[0003] Based on this, it is necessary to provide a table status recognition method and system based on image recognition to address the above technical problems, so as to scientifically and accurately evaluate the current status of the table and improve catering management efficiency and user experience.

[0004] In a first aspect, the present application provides a method for identifying a table state based on image recognition, the method comprising: Acquire table image data; the table image data carries human body information and food information; Perform food status analysis on table image data based on deep learning image classification to obtain food status information; The dining state of the table image data is analyzed according to the human posture estimation technology to obtain the dining state information, which includes whether the person is dining, the number of people dining and whether the person has left the table; The table status is evaluated by combining the food status information and the dining status information to obtain the current table status, which includes not starting dining, dining in progress, and dining finished.

[0005] In one embodiment, dining state analysis is performed on table image data according to human posture estimation technology to obtain dining state information, including: Based on human posture estimation technology, the image data of each dining table at consecutive moments are input into the human posture estimation model for analysis to obtain the motion trajectory of the key points of each diner. Extract the temporal and spatial features of the motion trajectory of each key point of the human body. The temporal features include velocity and acceleration, and the spatial features include the relative displacement of the key points and the angle of the key points. Combined with the action recognition algorithm, the temporal features and spatial features are input into the action classification model for action recognition to obtain the action category information of each diner; A comprehensive judgment is made based on the action category information of each diner to obtain dining status information, which includes whether the diner is dining, the number of diners and whether the diner has left the table.

[0006] In one embodiment, the food status information obtained by performing food status analysis on the table image data based on deep learning image classification includes: Based on deep learning, food is identified from table image data to obtain food category information; Based on the food category information, food usage is judged according to the table image data at different times to obtain a judgment result, which includes whether the food has been eaten and the degree of consumption; Extracting food shape features and food texture features from table image data, and comparing the food shape features and food texture features with preset normal foods based on food category information to obtain food abnormality information; The food status information is obtained by combining the judgment result and the food abnormality information.

[0007] In one embodiment, the calculation formula of the relative displacement of the key point and the angle of the key point is: ; Among them, d represents the relative displacement of any two different key points. and Represent the 3D coordinates of any two different key points i and j, respectively. Representation vector and vector The angle of the key point, the vector and the vector It is obtained by the 3D coordinates of any three different key points i, j and k.

[0008] In one embodiment, the table status is evaluated by combining the food status information and the dining status information to obtain the current table status, including: Quantify the food state information and the dining state information to obtain food state characteristics and dining state characteristics; Based on the graph convolutional network, the food state features and the dining state features are fused to obtain the dining state feature vector; The dining state feature vector is input into the dining state evaluation model to obtain the current table state.

[0009] In one embodiment, the method further comprises: When the current table status is that the meal has not started, the tableware status is recognized by combining the target detection algorithm with the semantic segmentation technology to obtain the tableware status information, which includes the tableware cleanliness and missing information; The cleanliness of the tabletop is evaluated by the texture and color features of the table image data to obtain the cleanliness information of the tabletop; When the tableware status information and / or the tabletop cleanliness information meets the preset conditions, a prompt message is generated, and the prompt message is used to instruct the service personnel to handle it.

[0010] In one embodiment, the tableware status information is obtained by using a target detection algorithm combined with a semantic segmentation technology to identify the tableware status of the table image data, including: The table image data is input into the YOLOv5 model for detection to obtain the bounding box information, which includes the tableware location and tableware category; Divide the table image data into regions according to the bounding box information to obtain a tableware region image; The tableware area image is input into the encoder for feature extraction to obtain high-level semantic features, which include texture features and shape features; The tableware area image is input into the decoder for upsampling, and high-level semantic features are fused through skip connections to obtain the tableware image. The feature similarity between the tableware image and the preset clean tableware image is calculated to obtain the tableware cleanliness; According to the expected number of each category of tableware in the preset tableware configuration, missing information is obtained by performing missing detection based on bounding box information; Combine the tableware cleanliness and missing information to get the tableware status information.

[0011] In one embodiment, the method further comprises: When the current table status is that the meal is over, a cleaning instruction is generated. The cleaning instruction includes the specific location of the table, the current table status, and the cleaning priority. The cleaning instruction is used to send and instruct the cleaning staff to clean the table; Get the response information from the cleaning staff and update the current table status to being cleaned; Get the cleaning completion information from the cleaning staff, update the current table status to an empty table, and send an empty table instruction. The empty table instruction is used to send it to the ranking system for ranking.

[0012] In a second aspect, the present application also provides a table status recognition system based on image recognition, the system comprising: An image acquisition module is used to acquire table image data, which contains human body information and food information; A food analysis module is used to analyze the food status of table image data based on deep learning image classification to obtain food status information; A dining analysis module is used to analyze the dining status of the table image data according to the human posture estimation technology to obtain dining status information, which includes whether the dining is in progress, the number of diners and whether the dining is complete. The status evaluation module is used to evaluate the table status by combining the food status information and the dining status information to obtain the current table status, which includes the status of not starting to eat, eating, and finishing the meal.

[0013] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the steps of the method of the first aspect are performed when the computer program is executed by a processor.

[0014] The present application provides a table status recognition method and system based on image recognition. First, by acquiring table image data, the image contains human body information and food information. These image data provide necessary visual input for subsequent table status evaluation. The table image data is processed based on deep learning image classification technology to perform food status analysis to obtain food status information, such as whether the food has been placed, whether it has been eaten, etc. This process can effectively identify the specific state of the food and provide data support for judging the dining situation at the table. Secondly, the table image data is analyzed using human posture estimation technology to evaluate the dining status. By analyzing the posture and position of the human body, it is possible to determine whether a meal is currently being eaten, the number of people eating, and whether anyone has left the table. This dining status information provides key behavioral data for table status evaluation.

[0015] By combining food status information and dining status information, a comprehensive evaluation of the table status can be performed to obtain the overall status of the current table. This evaluation result includes three possible states: not yet started, in progress, and finished. This process can accurately reflect the actual dining situation at the table, thereby providing efficient support for table management, smart dining and other scenarios. Compared with the traditional sensor-based table monitoring method, this method has the following beneficial effects through the organic combination of image recognition and deep learning technology: 1. The efficiency of table status recognition has been greatly improved. In traditional table status judgment methods, it may rely on manual observation or simple sensor data, which is inefficient and prone to errors. This method is based on image recognition and deep learning technology, which can quickly process a large amount of table image data, significantly improve the recognition frequency, and obtain the table status more timely in scenarios such as restaurants, which helps to improve the operating efficiency of restaurants.

[0016] 2. The accuracy of table status recognition is significantly improved. Due to the complex environment of the restaurant, traditional methods may be interfered by environmental factors, such as people moving around, light changes, etc., resulting in inaccurate judgments. In this method, based on image recognition technology, table information is obtained from multiple aspects, and deep learning can be used to accurately analyze various complex situations. In principle, it avoids erroneous judgments caused by environmental interference, further improves the accuracy of table status recognition, and effectively reduces problems such as waste of resources caused by inaccurate table status judgments, providing a strong guarantee for the intelligent management of restaurants. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flowchart of a table status recognition method based on image recognition is provided as an exemplary embodiment of the present invention; Figure 2 A schematic diagram of the structure of a table status recognition system based on image recognition is provided as an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0020] In one embodiment, Figure 1 As shown, a table status recognition method based on image recognition is provided. This embodiment takes the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps: S101: Acquire table image data; the table image data carries human body information and food information.

[0021] In an illustrative manner, the dining table area can be photographed by a camera device installed in the restaurant to obtain dining table image data. These cameras can be installed on the ceiling of the restaurant to photograph the dining table area from a bird's-eye view, or they can be installed on the wall to obtain dining table image data from the side. This dining table image data contains various features and details of the human body and food related to the dining table. For example, human body information includes the position, posture, and movements of diners around the dining table, and food information can include the type, quantity, placement, and remaining status of the food on the table.

[0022] S102: Perform food status analysis on the table image data based on deep learning image classification to obtain food status information.

[0023] Deep learning is a machine learning method based on artificial neural networks, which can learn the features and patterns of images through a large amount of training data. Common deep learning architectures, such as convolutional neural networks, can automatically extract local features of images, and gradually convert raw image data into high-level abstract features through structures such as convolutional layers, pooling layers, and fully connected layers, and use these features for classification and recognition. The table image data can be analyzed by a trained deep learning model to obtain food status information. This food status information can be in the form of classification, for example, the food status can be divided into categories such as "untouched", "partially eaten", and "almost eaten"; it can also be more specific quantitative information, such as the remaining proportion of food.

[0024] S103: Perform dining status analysis on the table image data according to the human posture estimation technology to obtain dining status information, which includes whether dining is in progress, the number of diners, and whether the dining is complete.

[0025] Human posture estimation is a computer vision technology that can infer the posture and movement of the human body by analyzing the human bone structure and joint position in the image. After obtaining the table image data, it is input into the trained human posture estimation model to obtain the dining status information, which includes whether the person is dining, the number of people dining, and whether the person has left the table. Specifically, the model can determine whether someone is dining at the current moment based on the position and posture of the detected human joints, such as the relative position relationship between the hand and the food and mouth, and the movement trajectory of the arm. And it can combine human body detection and counting algorithms to count the number of people who are identified as dining in the dining table area. This number can intuitively reflect the current use of the dining table and provide a basis for restaurant resource allocation such as tableware preparation and service staff arrangement. In addition, by continuously monitoring the changes in human posture and position, when it is detected that the person who was originally at the dining table disappears from the dining table area in the image, or the posture has changed significantly, such as from sitting to standing and away from the dining table, it can be determined as leaving the table.

[0026] S104: Evaluate the table status by combining the food status information and the dining status information to obtain the current table status, which includes the status of not starting dining, dining in progress, and dining finished.

[0027] Specifically, the food status information and dining status information describe the situation of the dining table from different perspectives, and can more comprehensively and accurately understand the overall situation of the dining table. Indicatively, the current table status can be judged and output through preset rules or machine learning models based on the combination of food status and dining status. Indicatively, when the system detects that the food on the table has not been taken, and no person is in a dining posture, such as the diner is not sitting or interacting with the table, it can be judged as "not starting to eat", or when it is detected that the food on the table is partially eaten or the remaining part is taken, and there are diners sitting and interacting with the table, it can be judged as "dining". If the food is eaten or there is not much left, that is, the food status is that the food has been eaten or there is a small amount left, and the diner has left, the current table status can be judged as "dining is over". Through this process, the dining status on the table can be monitored in real time, the different stages of the dining process can be automatically judged, and real-time feedback can be provided for restaurant management, making the restaurant operation more efficient, orderly and intelligent.

[0028] In the above-mentioned table state recognition method based on image recognition, by obtaining table image data carrying human information and food information, the limitations of relying solely on manual observation or simple sensor data are avoided, which helps to more comprehensively and accurately identify the table state. And based on the image classification technology of deep learning, the food state analysis of the table image data is performed to obtain food state information, which effectively reduces the error caused by subjective judgment of food state and improves the accuracy and reliability of food state analysis. By analyzing the dining state of the table image data according to the human posture estimation technology, the dining state information is refined into specific contents such as whether the table is dining, the number of diners and whether the table is leaving, which further improves the scientificity and reliability of the dining state analysis. Finally, the table state is evaluated by combining the food state information and the dining state information to obtain the current table state including not starting to eat, dining and finishing the meal, which not only improves the accuracy of the table state evaluation, but also provides key data support for restaurant management, service optimization, etc.

[0029] Compared with traditional table status judgment methods, this method not only greatly improves the efficiency and accuracy of table status recognition, but also can provide more stable and real-time table status data support in complex restaurant environments, providing important guarantees for the restaurant's intelligent operation and service quality improvement.

[0030] In an exemplary embodiment, dining state analysis is performed on table image data according to human posture estimation technology to obtain dining state information, including: Based on human posture estimation technology, the image data of each dining table at consecutive moments are input into the human posture estimation model for analysis to obtain the motion trajectory of the key points of each diner. Extract the temporal and spatial features of the motion trajectory of each key point of the human body. The temporal features include velocity and acceleration, and the spatial features include the relative displacement of the key points and the angle of the key points. Combined with the action recognition algorithm, the temporal features and spatial features are input into the action classification model for action recognition to obtain the action category information of each diner; A comprehensive judgment is made based on the action category information of each diner to obtain dining status information, which includes whether the diner is dining, the number of diners and whether the diner has left the table.

[0031] Schematically, the human posture estimation model can be a model based on OpenPose combined with deep learning, which identifies the positions of various key points of the human body in the image by analyzing the input table image data. Specifically, OpenPose is an open source human posture estimation framework that can simultaneously detect the skeletons of multiple people in the image and accurately identify the key points of the human body. These key points usually include the head, shoulders, elbows, knees, wrists, knees and other parts. And by processing the images at consecutive moments, it is possible to track the position changes of these key points at different time points, thereby forming the motion trajectory of the key points of the human body. For example, for a person who is having a meal, the model can track the process of his hand from picking up the tableware to putting the food into his mouth, forming the motion trajectory of the key points of the hand, including the path of the position change of the hand in space, reflecting the continuity and change of his movements. From the motion trajectory of the key points of the human body, the speed of the key point movement can be obtained by calculating the displacement between adjacent key points and the time difference between each frame, and the rate of change of the speed can be calculated to obtain the acceleration. This time feature helps to identify whether the diner makes certain movements quickly or slowly. In addition, by calculating the spatial displacement between the same key points in adjacent frames, the relative displacement of the key points can be obtained to further understand the range of motion and changes of the key points. By calculating the angles of different key points, the body posture of the diners can be understood.

[0032] Combined with the action recognition algorithm, these time features and spatial features are input into the action classification model to determine the action category of each diner, such as whether they are picking up tableware, eating with tableware, talking to people, getting up and leaving the table, or sitting in the seat to rest, so as to provide more accurate information for the subsequent dining status judgment. And by analyzing the action category information of each diner, if the action category is detected as taking tableware, using tableware to eat and other actions directly related to dining, then it can be determined that the diner is dining. Or when the action category information shows that a certain diner's action is to get up and leave the dining table area, and the diner's human body key points no longer appear in the subsequent images, then it can be determined that the diner has left the table. In addition, in the processing of each table image data, through the identified human body key points and subsequent action classification, some misjudgments such as waiters or passers-by can be eliminated, and the number of people who are dining at the table can be accurately counted.

[0033] In an exemplary embodiment, the calculation formula of the relative displacement of the key point and the angle of the key point is: ; Among them, d represents the relative displacement of any two different key points. and Represent the 3D coordinates of any two different key points i and j, respectively. Representation vector and vector The angle of the key point, and vector It is obtained by the 3D coordinates of any three different key points i, j and k.

[0034] In the above formula, the relative displacement between the two key points can be obtained by calculating the square of the coordinate difference on the x, y, and z axes respectively, then summing them up and taking the square root. . Schematically, if key point i is the position of the wrist and key point j is the position of the elbow, then the calculated displacement of the wrist relative to the elbow can reflect the extension or contraction of the arm. In the calculation formula of the key point angle, if key points i, j, and k are the elbow joint, wrist joint, and shoulder joint respectively, then the angle at the elbow joint can be calculated. , which can reflect the degree of arm bending, helping to identify more subtle movements. For example, changes in the angle of arm bending can indicate actions such as putting food into the mouth or putting the arm down.

[0035] In an exemplary embodiment, the food status information obtained by performing food status analysis on the table image data based on deep learning image classification includes: Based on deep learning, food is identified from table image data to obtain food category information; Based on the food category information, food usage is judged according to the table image data at different times to obtain a judgment result, which includes whether the food has been eaten and the degree of consumption; Extracting food shape features and food texture features from table image data, and comparing the food shape features and food texture features with preset normal foods based on food category information to obtain food abnormality information; The food status information is obtained by combining the judgment result and the food abnormality information.

[0036] Using deep learning algorithms to process table image data can automatically extract image features, such as the underlying features of food such as color, texture, and shape, as well as the overall features of different food combinations, and finally output food category information through multi-layer neural network processing, such as identifying steak, salad, and bread. For each identified food category, compare images at different times to observe changes in their appearance. If the comparison of the before and after images shows that the appearance of the food has changed significantly, such as the integrity of the food is destroyed or part of the food disappears, it can be determined that the food has been eaten, and the area or volume change of the food in the before and after images is calculated to obtain the degree of consumption of the food.

[0037] In addition, the food shape features and food texture features of the table image data are extracted. Food shape features may include the outline, size, aspect ratio, etc. of the food, and food texture features may be the texture of the food surface, such as the texture of steak, the graininess of bread, the skin texture of fruit, etc. For each food category, the extracted food shape features and food texture features are compared with the features of preset normal food. If the two are significantly different, food abnormality information is generated. For example, for some liquid foods, such as soups, beverages, etc., if the shape appears to overflow or spill outside the plate, it also belongs to food abnormality information. In this way, the deep learning algorithm can not only identify the category of food, but also conduct a detailed analysis of the use of food and whether there is any abnormality, further improving service efficiency and customer satisfaction, and realizing automated restaurant operation management and resource optimization.

[0038] In an exemplary embodiment, the table status is evaluated by combining the food status information and the dining status information to obtain the current table status, including: Quantify the food state information and the dining state information to obtain food state characteristics and dining state characteristics; Based on the graph convolutional network, the food state features and the dining state features are fused to obtain the dining state feature vector; The dining state feature vector is input into the dining state evaluation model to obtain the current table state.

[0039] In the process of quantifying food status information and dining status information, different quantification methods are used for different information. For example, whether the food is eaten can be represented by a Boolean value (0 for uneaten, 1 for eaten); the degree of consumption can be represented by a value between 0 and 1. Graph convolutional network is a deep learning model that can be used to process graph structure data. The quantified food status features and dining status features are input into the graph convolutional network for feature fusion, which can handle the complex relationship between features and capture the mutual influence between food status and diner status, and generate a global and more representative feature vector, namely, the dining status feature vector. The dining status feature vector is input into the dining status evaluation model. The model can process and analyze the input feature vector according to its internal parameters and learned patterns, and finally output the prediction result of the current table status, that is, whether the meal has not started, is in progress, or has ended, etc., which provides important basis and support for applications such as restaurant management and intelligent services.

[0040] In an exemplary embodiment, the method further comprises: When the current table status is that the meal has not started, the tableware status is recognized by combining the target detection algorithm with the semantic segmentation technology to obtain the tableware status information, which includes the tableware cleanliness and missing information; The cleanliness of the tabletop is evaluated by the texture and color features of the table image data to obtain the cleanliness information of the tabletop; When the tableware status information and / or the tabletop cleanliness information meets the preset conditions, a prompt message is generated, and the prompt message is used to instruct the service personnel to handle it.

[0041] Specifically, target detection algorithms such as YOLO and Faster R-CNN can be combined with semantic segmentation technology to identify the location and category of tableware in table image data, and to evaluate the cleanliness and quantity of tableware, so as to obtain tableware status information. In addition, the cleanliness of the tabletop is evaluated by the texture and color features of the table image data. For example, whether the color of the tabletop surface is uniform, whether there are obvious stains, dust or other debris, etc., to obtain the tabletop cleanliness information. When the tableware status information and / or the tabletop cleanliness information meets the preset conditions, such as poor tableware cleanliness, missing tableware, stains on the tabletop, etc., a prompt message is generated to prompt the service staff to clean the tableware, replenish the missing tableware, or clean the tabletop, etc.

[0042] Exemplarily, the tableware status information is obtained by performing tableware status recognition on the table image data by combining the target detection algorithm with the semantic segmentation technology, including: The table image data is input into the YOLOv5 model for detection to obtain the bounding box information, which includes the tableware location and tableware category; Divide the table image data into regions according to the bounding box information to obtain a tableware region image; The tableware area image is input into the encoder for feature extraction to obtain high-level semantic features, which include texture features and shape features; The tableware area image is input into the decoder for upsampling, and high-level semantic features are fused through skip connections to obtain the tableware image. The feature similarity between the tableware image and the preset clean tableware image is calculated to obtain the tableware cleanliness; According to the expected number of each category of tableware in the preset tableware configuration, missing information is obtained by performing missing detection based on bounding box information; Combine the tableware cleanliness and missing information to get the tableware status information.

[0043] YOLOv5 is a target detection algorithm that can quickly and accurately detect target objects in images. The table image data is input into the YOLOv5 model for detection to obtain bounding box information. Among them, the bounding box is a rectangular box used to calibrate the position of the tableware, usually including the coordinates of the upper left corner and the lower right corner, and each tableware is also marked with a category, such as knife, fork, spoon, plate, etc. According to the bounding box detected by YOLOv5, the table image data is divided into multiple small areas, each area corresponds to a tableware, and the tableware area image is obtained. The encoder is a part of the neural network. The tableware area image is input into the encoder for convolution and other operations, and high-level semantic features including texture features and shape features can be obtained. Subsequently, the decoder will upsample the tableware area image, restore the image to a size close to the original size, and fuse the high-level semantic features extracted by the encoder with the features in the decoder upsampling process through jump connections. Through this process, the final tableware image contains both rich detail information and important semantic features, thereby providing a more accurate basis for subsequent cleanliness judgment.

[0044] The cleanliness of the tableware is determined by calculating the feature similarity between the tableware image and the preset clean tableware image. If the similarity is high, it means that the tableware is relatively clean; if the similarity is low, it means that there may be stains or other dirty places on the tableware, and the cleanliness is low. And according to the bounding box information, the number of each type of tableware actually detected can be counted, and it can be compared with the expected number of each type of tableware that should be equipped on each table in the restaurant's preset tableware configuration to determine whether there is any tableware missing and obtain missing information. Finally, the tableware status information combines the calculated tableware cleanliness and the detected missing information, which helps restaurant service staff to manage the table, clean and replenish the tableware before dining.

[0045] In an exemplary embodiment, the method further comprises: When the current table status is that the meal is over, a cleaning instruction is generated. The cleaning instruction includes the specific location of the table, the current table status, and the cleaning priority. The cleaning instruction is used to send and instruct the cleaning staff to clean the table; Get the response information from the cleaning staff and update the current table status to being cleaned; Get the cleaning completion information from the cleaning staff, update the current table status to an empty table, and send an empty table instruction. The empty table instruction is used to send it to the ranking system for ranking.

[0046] Indicatively, the cleaning priority of the current table in the cleaning instruction can be set according to the restaurant's operating strategy, such as tables in certain areas need to be cleaned first due to large customer flow or other reasons. When the cleaning staff receives the cleaning instruction, they can send a response message by replying with a handheld device. After obtaining the response information of the cleaning staff, the current table status will be updated to cleaning, so as to track the status changes of the table in real time and avoid repeated arrangements for cleaning work or other confusion. When the cleaning staff completes the cleaning work of the table, the cleaning staff's cleaning end information is obtained, and the current table status is updated to an empty table to indicate that the table has been cleaned and can be used by new customers. And send an empty table instruction to the ranking system for ranking. Among them, the ranking system is a system used by the restaurant to manage customers waiting in line for dining, and can arrange customers to sit according to the situation of empty tables in the restaurant. Through the above steps, the restaurant can realize the automatic management of the table status and the real-time update of information, ensure that the cleaning work of the restaurant is carried out in an orderly manner, and reasonably coordinate the dining arrangements of customers, and improve the overall service quality and operation effect.

[0047] Based on the same inventive concept, Figure 2 As shown, the embodiment of the present application also provides a table status recognition system 200 based on image recognition, the system comprising: The image acquisition module 201 is used to acquire table image data, which contains human body information and food information; A food analysis module 202, for performing food status analysis on table image data based on deep learning image classification to obtain food status information; The dining analysis module 203 is used to perform dining status analysis on the table image data according to the human posture estimation technology to obtain dining status information, which includes whether the dining is in progress, the number of diners and whether the dining is complete; The status evaluation module 204 is used to evaluate the table status by combining the food status information and the dining status information to obtain the current table status, which includes the status of not starting dining, dining in progress and dining finished.

[0048] The system can obtain table image data carrying human body information and food information through the image acquisition module 201, ensuring that the system can fully capture the dining environment on the table and provide a comprehensive and accurate data basis for subsequent analysis. In the food analysis module 202, based on deep learning image classification, the table image data obtained by the image acquisition module 201 is analyzed for food status to obtain food status information. This process converts image data into information related to food status, namely food status information, by using deep learning technology and image classification algorithms. This result reflects the current status of the food on the table and provides key information for evaluating the status of the table.

[0049] In the dining analysis module 203, dining status analysis is performed based on human posture estimation technology and combined with table image data, and the dining status information is refined into specific contents such as whether dining is in progress, the number of diners, and whether the diners have left the table. This analysis can reflect the dynamic status of people at the table in real time, help the system to more accurately judge the behavior patterns during the dining process, and can help relevant personnel better understand the actual situation of dining and ensure accurate judgment of the table status. In the status evaluation module 204, the food status information and dining status information are combined to evaluate the table status, and the current table status including the status of not starting to eat, dining, and finishing the meal can be obtained. This data can provide key support for restaurant management, service optimization, etc., and further improve the efficiency and service quality of restaurant operations.

[0050] Furthermore, the dining analysis module 203 includes: Action category recognition subunit, used to: Based on human posture estimation technology, the image data of each dining table at consecutive moments are input into the human posture estimation model for analysis to obtain the motion trajectory of the key points of each diner. Extract the temporal and spatial features of the motion trajectory of each key point of the human body. The temporal features include velocity and acceleration, and the spatial features include the relative displacement of the key points and the angle of the key points. Combined with the action recognition algorithm, the temporal features and spatial features are input into the action classification model for action recognition to obtain the action category information of each diner; The dining judgment subunit is used to: A comprehensive judgment is made based on the action category information of each diner to obtain dining status information, which includes whether the diner is dining, the number of diners and whether the diner has left the table.

[0051] Furthermore, the food analysis module 202 includes: A food category recognition subunit is used to perform food recognition on table image data based on deep learning to obtain food category information; A food usage judgment subunit is used to judge the food usage based on the food category information and the table image data at different times, and obtain a judgment result, which includes whether the food has been eaten and the degree of consumption; The food abnormality recognition subunit is used to extract food shape features and food texture features from the table image data, and compare the food shape features and food texture features with preset normal foods according to the food category information to obtain food abnormality information; The food comprehensive assessment subunit is used to combine the judgment results and food abnormality information to obtain food status information.

[0052] Furthermore, the calculation formulas for the relative displacement of key points and the angle of key points are: ; Among them, d represents the relative displacement of any two different key points. and Represent the 3D coordinates of any two different key points i and j, respectively. Representation vector and vector The angle of the key point, and vector It is obtained by the 3D coordinates of any three different key points i, j and k.

[0053] Furthermore, the status assessment module 204 includes: An information quantification subunit, used to quantify the food state information and the dining state information to obtain food state characteristics and dining state characteristics; The information processing subunit is used to: Based on the graph convolutional network, the food state features and the dining state features are fused to obtain the dining state feature vector; The dining state feature vector is input into the dining state evaluation model to obtain the current table state.

[0054] Furthermore, the system also includes a pre-meal detection module for: When the current table status is that the meal has not started, the tableware status is recognized by combining the target detection algorithm with the semantic segmentation technology to obtain the tableware status information, which includes the tableware cleanliness and missing information; The cleanliness of the tabletop is evaluated by the texture and color features of the table image data to obtain the cleanliness information of the tabletop; When the tableware status information and / or the tabletop cleanliness information meets the preset conditions, a prompt message is generated, and the prompt message is used to instruct the service personnel to handle it.

[0055] Furthermore, the pre-meal detection module includes a tableware detection subunit, which is used to: The table image data is input into the YOLOv5 model for detection to obtain the bounding box information, which includes the tableware location and tableware category; Divide the table image data into regions according to the bounding box information to obtain a tableware region image; The tableware area image is input into the encoder for feature extraction to obtain high-level semantic features, which include texture features and shape features; The tableware area image is input into the decoder for upsampling, and high-level semantic features are fused through skip connections to obtain the tableware image. The feature similarity between the tableware image and the preset clean tableware image is calculated to obtain the tableware cleanliness; According to the expected number of each category of tableware in the preset tableware configuration, missing information is obtained by performing missing detection based on bounding box information; Combine the tableware cleanliness and missing information to get the tableware status information.

[0056] Furthermore, the system also includes a table cleaning module, which is used to: When the current table status is that the meal is over, a cleaning instruction is generated. The cleaning instruction includes the specific location of the table, the current table status, and the cleaning priority. The cleaning instruction is used to send and instruct the cleaning staff to clean the table; Get the response information from the cleaning staff and update the current table status to being cleaned; Get the cleaning completion information from the cleaning staff, update the current table status to an empty table, and send an empty table instruction. The empty table instruction is used to send it to the ranking system for ranking.

[0057] In an exemplary embodiment, the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of a table status recognition method based on image recognition of the present application are implemented. The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk, etc. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM).

[0058] The above-mentioned embodiments only express several implementation methods of the embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the embodiments of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the embodiments of the present application, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. A table status recognition method based on image recognition, characterized in that: The method comprises: Acquire table image data; the table image data carries human body information and food information; Perform food status analysis on table image data based on deep learning image classification to obtain food status information; Perform dining status analysis on the table image data according to the human posture estimation technology to obtain dining status information, wherein the dining status information includes whether the person is dining, the number of people dining, and whether the person has left the table; Combining the food status information and the dining status information to evaluate the dining table status, and obtaining the current dining table status, wherein the current dining table status includes not starting dining, dining in progress, and dining finished; When the current table status is that dining has not started, tableware status recognition is performed on the table image data by using a target detection algorithm combined with a semantic segmentation technology to obtain tableware status information, wherein the tableware status information includes tableware cleanliness and missing information; The cleanliness of the tabletop is evaluated by the texture and color features of the table image data to obtain the cleanliness information of the tabletop; When the tableware status information and / or the tabletop cleanliness information meets a preset condition, a prompt message is generated, and the prompt message is used to instruct the service personnel to handle it.

2. The method according to claim 1, characterized in that The dining state analysis of the dining table image data according to the human body posture estimation technology to obtain dining state information includes: Based on human posture estimation technology, the image data of each dining table at consecutive moments are input into a human posture estimation model for analysis to obtain the motion trajectory of key points of each diner's body; Extracting time features and space features from the motion trajectory of each key point of the human body, wherein the time features include speed and acceleration, and the space features include relative displacement of the key points and angle of the key points; In combination with the action recognition algorithm, the time feature and the spatial feature are input into the action classification model for action recognition to obtain action category information of each diner; A comprehensive judgment is made based on the action category information of each of the diners to obtain the dining status information, which includes whether the diners are dining, the number of diners and whether they have left the table.

3. The method according to claim 1, characterized in that: The image classification based on deep learning performs food status analysis on the table image data to obtain food status information including: Performing food recognition on the table image data based on deep learning to obtain food category information; Based on the food category information, food usage judgment is performed according to the table image data at different times to obtain a judgment result, wherein the judgment result includes whether the food has been eaten and the degree of consumption; Extracting food shape features and food texture features from the table image data, and comparing the food shape features and food texture features with preset normal foods according to the food category information to obtain food abnormality information; The food status information is obtained by combining the judgment result and the food abnormality information.

4. The method according to claim 2, characterized in that: The calculation formula of the relative displacement of the key point and the angle of the key point is: ； Where d represents the relative displacement of any two different key points. and Represent the 3D coordinates of any two different key points i and j, respectively. Representation vector and vector The angle of the key point, the vector and the vector It is obtained by the 3D coordinates of any three different key points i, j and k.

5. The method according to claim 1, characterized in that The table status evaluation is performed by combining the food status information and the dining status information to obtain the current table status, including: quantifying the food state information and the dining state information to obtain food state characteristics and dining state characteristics; Based on a graph convolutional network, the food state feature and the dining state feature are subjected to feature fusion to obtain a dining state feature vector; The dining state feature vector is input into a dining state evaluation model to obtain the current dining table state.

6. The method according to claim 1, wherein the tableware status information is obtained by performing tableware status recognition on table image data by combining a target detection algorithm with a semantic segmentation technique, comprising: Input the table image data into the YOLOv5 model for detection to obtain bounding box information, where the bounding box information includes tableware position and tableware category; Divide the table image data into regions according to the boundary box information to obtain a tableware region image; Inputting the tableware area image into an encoder for feature extraction to obtain high-level semantic features, wherein the high-level semantic features include texture features and shape features; Inputting the tableware region image into a decoder for upsampling, and fusing high-level semantic features through skip connections to obtain a tableware image; Calculating feature similarity between the tableware image and a preset clean tableware image to obtain the tableware cleanliness; According to the expected number of each category of tableware in the preset tableware configuration, performing missing detection based on the bounding box information to obtain the missing information; The tableware status information is obtained by combining the tableware cleanliness and the missing information.

7. The method according to claim 1, characterized in that The method further comprises: When the current table status is that the meal is over, a cleaning instruction is generated, wherein the cleaning instruction includes the specific location of the table, the current table status, and the cleaning priority, and the cleaning instruction is used to send and instruct the cleaning staff to clean the table; Obtaining the response information of the cleaning staff, and updating the current table status to being cleaned; The cleaning completion information of the cleaning staff is obtained, the current dining table status is updated to an empty table, and an empty table instruction is sent, and the empty table instruction is used to send the ranking system for ranking.

8. A table status recognition system based on image recognition, characterized in that: The system comprises: An image acquisition module, used to acquire table image data, wherein the table image data carries human body information and food information; A food analysis module is used to analyze the food status of table image data based on deep learning image classification to obtain food status information; A dining analysis module is used to analyze the dining status of the table image data according to the human posture estimation technology to obtain dining status information, which includes whether the dining is in progress, the number of diners and whether the dining is complete. A status evaluation module, used to evaluate the table status by combining the food status information and the dining status information to obtain the current table status, wherein the current table status includes the status of not starting dining, dining in progress, and dining finished; Pre-meal detection module, used for: When the current table status is that dining has not started, tableware status recognition is performed on the table image data by using a target detection algorithm combined with a semantic segmentation technology to obtain tableware status information, wherein the tableware status information includes tableware cleanliness and missing information; The cleanliness of the tabletop is evaluated by the texture and color features of the table image data to obtain the cleanliness information of the tabletop; When the tableware status information and / or the tabletop cleanliness information meets a preset condition, a prompt message is generated, and the prompt message is used to instruct the service personnel to handle it.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images