State recognition method and device, restaurant management method and device and computer storage medium

Abstract

The invention provides a state recognition method, a restaurant management method, a restaurant management device and a computer storage medium. The state identification method is applied to a video acquisition device, and comprises the following steps: acquiring continuous multi-frame dining table images; detecting human body key points of people in a preset dining table boundary range in each dining table image; determining the personnel type of the personnel based on the human body key points of each dining table image; and in response to the fact that the person is a non-working person, determining the dining state of the person in combination with the movement change rule of the human body key points in the continuous multi-frame dining table image. Through the above state identification method, the table-going state of the non-staff can be accurately identified.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a state recognition method, a restaurant management method, a restaurant management device, and a computer storage medium. Background Art

[0002] Currently, restaurants use restaurant management systems to synchronize lobby information with the restaurant front desk and restaurant kitchens, realizing the informatization of ordering and queuing. However, due to technical limitations, traditional restaurant management still relies on lobby managers to manually discover the status of customers' table arrival, and use verbal or paging methods to arrange staff to provide ordering services, table cleaning, and checkout tracking. The table turnover efficiency is not high, and customers occasionally run away with the bill, which affects the restaurant's revenue and the dining experience. Summary of the invention

[0003] In order to solve the above technical problems, the present application proposes a state identification method, a restaurant management method, a restaurant management device and a computer storage medium.

[0004] In order to solve the above technical problems, the present application proposes a state recognition method, which is applied to a video acquisition device, including:

[0005] Collect multiple consecutive frames of dining table images;

[0006] Detect key points of the human body of the person within the preset table boundary range in each table image;

[0007] Determining the personnel type of the personnel based on the human body key points of each dining table image;

[0008] In response to the fact that the person is a non-staff member, the table status of the person is determined by combining the movement change rules of key points of the human body in the multiple consecutive frames of table images.

[0009] Wherein, the human body key points include head key points and shoulder key points;

[0010] The determining the type of the person based on the key points of the human body in each dining table image includes:

[0011] Based on the head key points and the shoulder key points, a personnel type differentiation area is expanded;

[0012] In response to searching the personnel type distinguishing area for a staff member identifier, it is determined that the personnel type of the person is a staff member.

[0013] Wherein, after detecting the key points of the human body of a person within the preset table boundary range in each table image, the state recognition method further comprises:

[0014] Based on the head key points, determining the left and right order of the shoulder key points;

[0015] Taking the shoulder key point as a base point, expanding the search for the shoulder clothes of the person;

[0016] According to the left-right order of the shoulder key points and the shoulder clothes, the arm key points of the person are determined.

[0017] Wherein, after determining that the personnel type of the personnel is a staff member, the status identification method further includes:

[0018] Comparing the staff identification of the personnel with a preset staff standard identification to obtain the image inspection result of the personnel;

[0019] Wherein, the standard identification of the staff includes a work hat and / or a work shoulder label.

[0020] The table boundary includes a food area and a personnel area, wherein the food area and the personnel area are separated by a table boundary;

[0021] The method of detecting key points of a person's body within a preset table boundary range in each table image includes:

[0022] The human body key points of the people are detected in the preset people area in each dining table image.

[0023] Wherein, the state identification method further includes:

[0024] Detecting dish images in a preset dish area in each table image;

[0025] The service status of the dish area is determined by combining the comparison results of the dish images in the continuous multiple frames of table images with the preset idle images of the dish area.

[0026] The video acquisition device is installed above the dining table, and the video acquisition device is associated with the identification of the dining table.

[0027] Among them, in order to solve the above technical problems, the present application also proposes a restaurant management method, which is applied to a restaurant management system, wherein the restaurant management system is connected to several video acquisition devices;

[0028] Get the status of people at the table from any video acquisition device;

[0029] Acquiring the business status of the dish area in the dining table from the video acquisition device, or identifying the business status of the dish area from the dining table image uploaded by the video acquisition device;

[0030] Based on the table status of the personnel and the business status of the food area, determine the business status of the table associated with the video acquisition device, where the business status of the table is any one of ordering status, serving status, dining completion status, cleaning status, and idle status;

[0031] The video acquisition device identifies the table status of the person through the above-mentioned status recognition method.

[0032] In order to solve the above-mentioned technical problems, the present application also proposes a restaurant management device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned status identification method, and / or restaurant management method.

[0033] In order to solve the above technical problems, the present application also proposes a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the above-mentioned state identification method and / or restaurant management method.

[0034] Compared with the prior art, the beneficial effects of the present application are as follows: the restaurant management device collects multiple consecutive table images; detects the key points of the human body of the person within the preset table boundary range in each table image; determines the type of the person based on the key points of the human body in each table image; in response to the person being a non-staff member, determines the table status of the person in combination with the movement change law of the key points of the human body in the multiple consecutive table images. Through the above-mentioned status recognition method, the table status of non-staff members can be accurately identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] in:

[0037] Figure 1 It is a flow chart of an embodiment of a state identification method provided by the present application;

[0038] Figure 2 It is a functional framework diagram of the restaurant management system provided by this application;

[0039] Figure 3 This is a schematic diagram of marking an embodiment of a key point of a human body provided by the present application;

[0040] Figure 4 is a flow chart of another embodiment of the state identification method provided by the present application;

[0041] Figure 5 It is a schematic diagram of preset boundaries and areas of the video acquisition device provided by the present application;

[0042] Figure 6 It is a flowchart of another embodiment of the state identification method provided by the present application;

[0043] Figure 7 It is a flowchart of an embodiment of a restaurant management method provided by the present application;

[0044] Figure 8 It is a structural schematic diagram of an embodiment of a restaurant management device provided by the present application;

[0045] Fig. 9 It is a structural diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0048] The existing technology uses embedded sensors installed around the table or on the plate, or uses side-mounted video capture for human recognition and table recognition, and synchronizes it to the restaurant management system to achieve automatic table status recognition, but the recognition system is complex to install, the recognition rate is not high, or the use and maintenance costs are high, so it has not been promoted. Restaurants are public places with high-temperature dangerous goods. The existing technology does not have an automatic alarm function collection technology for abnormal events in restaurants, or the collected information is often inaccurate, relying on naked eye recognition or viewing collection and recognition, verbal or pager notification of relevant staff for processing, and lack of timeliness and accuracy, which expands the impact of abnormal events in restaurants and causes losses to diners and restaurants.

[0049] The following is an introduction to the technical terms of this application in the catering application field:

[0050] Table turnover: It is a term used in catering services. Every table in the restaurant is cleared and new tableware is placed on it after the guests finish their meal.

[0051] Lobby manager: is the person in charge of inspecting and supervising the restaurant staff to complete pre-meal preparation, meal service and post-meal finishing work during restaurant operations.

[0052] Restaurant front desk: responsible for receiving and registering diners in the restaurant and arranging diners to dine at vacant tables.

[0053] Restaurant management system: It is a comprehensive restaurant information management system connecting the kitchen, lobby and front office.

[0054] The restaurant table status recognition method of the present application is to install the video acquisition device supporting the scheme of the present application on the top roof or above the corresponding table. By setting the table area and the desktop area in the video acquisition screen, the present application uses the recognition module of the video acquisition device to identify the personnel, personnel behavior, and table usage in the recognition area, and the comprehensive output includes but is not limited to the following table status: idle state, ordering state, serving state, meal completion, waiting for cleaning state, etc. The video acquisition inputs the recognized state into the restaurant management system, and the restaurant management system can notify the staff to provide timely service to the diners based on the recognition results. At the same time, the video acquisition provides abnormal event alarms by collecting abnormal sounds and analyzing the diners' separation from the recognition. The restaurant management system notifies the staff to handle it in time, which can improve the restaurant service and reduce the loss of the restaurant and diners. The restaurant management system can perform operation-related statistical analysis based on the information and videos reported by the video acquisition in this case, including but not limited to the timeliness of serving and cleaning; the popularity of the dishes; and the attendance of the staff.

[0055] The solution of the present application is used for restaurant management to realize the informatization and automation of restaurant management, which will improve the accuracy of service, reduce the workload of restaurant staff, increase the utilization rate of tables, and reduce losses in case of sudden abnormal events in a timely manner, thereby helping to increase restaurant revenue; and the solution of the present application does not rely on installing sensors on tables and chairs, using wearable devices on staff, and the recognition algorithm only uses the algorithm module of video acquisition itself, does not rely on other expensive secondary recognition systems, and has a low cost of use.

[0056] Please refer to Figure 1 and Figure 2 , Figure 1 is a flow chart of an embodiment of a state identification method provided by the present application, Figure 2 It is a functional framework diagram of the restaurant management system provided by this application.

[0057] The state recognition method of the present application is applied to a video acquisition device. Specifically, the video acquisition device of the present application can be a processing unit or processor in a camera, or can be a processing platform mounted on a camera or remotely controlling a camera.

[0058] like Figure 1 As shown, the specific steps are as follows:

[0059] Step S11: Capture multiple consecutive frames of dining table images.

[0060] In the embodiment of the present application, the video acquisition device acquires multiple frames of continuous dining table images for the dining table area.

[0061] Specifically, the present application installs the video acquisition device on the roof or above the dining table, and makes the dining table range to be captured appear in the screen, including the dining table and dining chairs, which has the following advantages over the prior art:

[0062] The video capture equipment is installed above the dining table, which has less impact on the overall environment of the restaurant than the wall-mounted capture, especially after proper decoration, while meeting the restaurant management needs.

[0063] Compared with the solution of installing embedded sensors in tables and chairs, it avoids the problems of high sensor cost, easy damage due to frequent use, and difficult repair and maintenance.

[0064] Compared with other solutions that use one video acquisition to identify multiple dining tables, the solution of using one video acquisition for one dining table has less loss, higher fault tolerance and easier recovery when encountering equipment failure.

[0065] The present application uses a video capture device to capture a dining table, ensuring that each video capture device can capture real-time images of the dining table area in a vertical downward manner, and each video capture device is associated with the identification of the dining table. For example, the video capture device is associated with the table number of the dining table, so that the image or recognition result output by the video capture device is bound one by one to the table number of the dining table.

[0066] Step S12: Detecting key points of the human body of a person within a preset table boundary range in each table image.

[0067] In the embodiment of the present application, the recognition module in the video acquisition device extracts the key points of the human body of the person within the preset table boundary range in each table image. Then, the person is identified based on the key points of the head and shoulders, including staff and non-staff, such as diners, etc. The key points of the human body of the present application include at least the key points of the head and the key points of the shoulders.

[0068] Among them, the algorithm used by the recognition module in the video acquisition device to extract the key points of the human body of a person can adopt the existing human body key point extraction algorithm, and the model used can adopt CNN (Convolutional Neural Network), MLP (Multi Layer Perseption), RNN (Recurrent Neural Network), etc.

[0069] This application can reduce the range of human body key point extraction and improve the efficiency of personnel detection by presetting the boundary range of the dining table.

[0070] Step S13: Determine the personnel type of the personnel based on the human body key points of each dining table image.

[0071] In the embodiment of the present application, the video acquisition device uses a bird's-eye view, utilizes the head and shoulders of the person, and uses a trained large model recognition algorithm to quickly identify the person entering the personnel area, which has the following advantages over other recognition methods:

[0072] This application uses the bird's-eye view of video capture, which has a good field of view and sufficient light, and uses a head and shoulder recognition algorithm trained with large model data to quickly identify and label people in a limited personnel area.

[0073] Other solutions in the prior art rely on the human figure recognition rate, and according to industry common sense, the human figure recognition rate is not high.

[0074] Compared with other solutions in the prior art, the personnel identification of the present application is limited to the designated dining table area, no conflict occurs, no reliance is placed on secondary matching, and identification is effective, thus improving the recognition efficiency.

[0075] Furthermore, the video acquisition device can also expand the personnel type differentiation area based on the head key points and shoulder key points before identification, wherein the personnel type differentiation area is within the preset dining table boundary range and is smaller than the preset dining table boundary range, which can further improve the efficiency and accuracy of personnel detection and reduce the impact of background noise.

[0076] Specifically, the video acquisition device uses the recognition module to identify the staff member based on the head and shoulders. The special mark used in the present application to distinguish the type of personnel can be a hat, a label containing three labels of the same color on the shoulders, or a digital number. For example, the special mark can be a hat with a preset shape and / or a preset color; the special mark can also be a label on the shoulders of the personnel, such as a work sign, or a digital number, such as an employee number.

[0077] The recognition algorithm of this application recognizes from a bird's-eye view, which opens up the field of vision of the head and shoulders. Compared with other recognition algorithms for staff, it has the following advantages:

[0078] The recognition algorithm of the present application supports the recognition of restaurant staff. Compared with not recognizing the type of people, that is, not distinguishing between staff and non-staff, applying the recognition of staff to the recognition of restaurant status improves the accuracy and diversity of recognition.

[0079] The video acquisition used by the recognition algorithm of the present application is overhead recognition, and recognition can be achieved by setting special marks on the hat on the head and the shoulders. The corresponding labels are low in cost and easy to implement. These two label areas have little visual impact on the actual diners.

[0080] The recognition algorithm of the present application avoids the impact of the special dress requirements on the restaurant in the solution of relying on the dress to identify the staff, meets the restaurant's demand for customized dress, and avoids sacrificing the restaurant's personalization.

[0081] The recognition algorithm of the present application avoids the problems of high cost and easy damage of relying on wearable recognition solutions.

[0082] The recognition algorithm of this application also has the function of checking the consistency of the labels set on the hats and shoulders for the uniform dress code required to enhance the image of the restaurant. The restaurant management system can notify the management personnel to make timely corrections based on the recognition results.

[0083] In addition, the video capture equipment of the present application can also continuously input the identification information of the staff into the restaurant management system, which can be used for restaurant value purposes such as attendance.

[0084] Specifically, the video acquisition device can compare the staff identification of the staff with the preset staff standard identification to obtain the image inspection result of the staff; wherein the staff standard identification includes a work hat and / or a work shoulder tag.

[0085] In a specific implementation, the process of analyzing the behavior of a target person using the recognition module by the video acquisition device specifically includes the following steps:

[0086] The video acquisition device uses a human key point detection algorithm, such as SimpleBaseline, Yolo-Pose, etc., to detect the key point coordinates of the head, shoulder, arm, and hand of each human target in the field of view. Figure 3 As shown, Figure 3 It is a schematic diagram of the identification of an embodiment of the key points of the human body provided by the present application.

[0087] The video acquisition device expands a square area with a side length of r, taking the detected key points of the human target head and shoulders as the center, such as Figure 3 The square logo in the figure is used to identify the human target as a restaurant staff or non-staff member based on the color label or number on the head and shoulders of each human target in the three square areas.

[0088] For each non-staff member, the video acquisition device combines the coordinates of the key points of the arms and hands identified in multiple consecutive frames, and performs behavioral analysis through the motion change rules of the corresponding key points (or uses action recognition models, such as STGCN (spatial-temporal Graph Convolutional Networks), etc.) to determine the status of the non-staff member, such as the detection of the status of guests at the table in the embodiment of the present application.

[0089] In addition, this application also provides another method for retrieving the customer's table status. For details, please refer to Figure 4 , Figure 4 It is a flowchart of another embodiment of the state identification method provided by the present application.

[0090] like Figure 4 As shown, the specific steps are as follows:

[0091] Step S21: Based on the head key points, determine the left and right order of the shoulder key points.

[0092] In an embodiment of the present application, the video acquisition device uses the head key points as a base point to determine the left and right order of the shoulder key points, that is, the shoulder key points are further divided into left shoulder key points and right shoulder key points.

[0093] Step S22: Taking the shoulder key point as the base point, expand the shoulder clothes of the search person.

[0094] In the embodiment of the present application, the video acquisition device uses the left shoulder key point and the right shoulder key point as base points to expand the search for clothes on the left shoulder and clothes on the right shoulder. Because the color features of the clothes are relatively close, the color features of the clothes at the shoulder key points are used to expand the search for pixels with the same color features adjacent to the shoulder clothes, thereby obtaining the pixel range of the shoulder clothes.

[0095] Step S23: Determine the key points of the person's arms according to the left-right order of the key points of the shoulders and the clothes on the shoulders.

[0096] In the embodiment of the present application, the video acquisition device determines the left arm area of ​​the person according to the extension direction of the left shoulder clothing corresponding to the left shoulder key point, thereby obtaining the left arm key point. The video acquisition device determines the right arm area of ​​the person according to the extension direction of the right shoulder clothing corresponding to the right shoulder key point, thereby obtaining the right arm key point.

[0097] The video acquisition equipment can analyze the guest's physical condition after sitting down at the table by combining the position and changes of the key points of the head, shoulders and arms, thereby obtaining the accurate status of the person sitting down at the table.

[0098] Step S14: In response to the person being a non-staff member, the table status of the person is determined by combining the movement change patterns of key points of the human body in multiple consecutive frames of table images.

[0099] In the embodiment of the present application, when the video acquisition device detects that the person is a non-staff member, such as a dining person, the video acquisition device further combines the motion change pattern of the key points of the human body in the continuous multi-frame table image to confirm the table status of the non-staff member. The motion change pattern of the key points of the human body can be confirmed by the optical flow information of the key points of the human body in the continuous multi-frame table image, or the frame difference information of the continuous multi-frame table image.

[0100] In the embodiment of the present application, the video acquisition device acquires multiple consecutive table images; detects the key points of the human body of the person within the preset table boundary range in each table image; determines the type of the person based on the key points of the human body in each table image; in response to the person being a non-staff member, determines the table status of the person in combination with the movement change rules of the key points of the human body in the multiple consecutive table images. The above-mentioned status recognition method can accurately identify the table status of non-staff members.

[0101] Further, it can be seen from the introduction of the above embodiments that the video capture device of the present application sets the dining table boundary of the video capture screen or the video capture image, which is the boundary required to expand when the dining chair is used. The dining table boundary supports circular, rectangular, polygonal and other image designs. For details, please refer to Figure 5 and Figure 6 , Figure 5is a schematic diagram of preset boundaries and areas of the video acquisition device provided by this application, Figure 6 It is a flow chart of another embodiment of the state identification method provided by the present application.

[0102] like Figure 5 As shown, the video acquisition device of the present application sets a desktop boundary, and the desktop boundary is the actual desktop screen area. The desktop boundary supports circular, rectangular, and polygonal shapes, such as Figure 5 The recognition module supports intrusion detection outside the desktop boundary and within the desktop edge boundary range; when setting the desktop boundary, the recognition module in video acquisition automatically shrinks its center by 20%, taking 80% of the desktop boundary range as the area. Figure 5 The dish area shown in the figure. The area between the desktop boundary and the dining table boundary is used as the personnel area for detecting personnel.

[0103] like Figure 6 As shown, the specific steps are as follows:

[0104] Step S31: Detecting dish images in a preset dish area in each table image.

[0105] In an embodiment of the present application, the video acquisition device pre-captures an idle screen of the dish area, that is, the area screen when the dish area is generated. At this time, there is no dish in the dish area, and it is set to a standard screen in an idle state.

[0106] The video acquisition device scans the dish area at preset intervals, such as 1 minute, to collect multiple frames of dish images in the dish area.

[0107] Step S32: Combine the results of comparing the dish images in the continuous multiple-frame table image with the preset idle images of the dish area to determine the business status of the dish area.

[0108] In the embodiment of the present application, the video acquisition device compares the continuous multi-frame dish screen with the standard dish area idle screen to determine whether the table is in use and the service status of the dish area. In addition, since the video acquisition device acquires continuous multi-frame dish screens, the dish area can also be inspected dynamically through the timing information to determine whether the table is in the serving state, dining state, or cleaning state.

[0109] This application adopts Figure 5 Compared with other solutions in the prior art, the area division scheme shown has the following advantages:

[0110] Compared with other solutions that do not support intrusion detection at the desktop boundary, intrusion detection can be used to identify the behavior of personnel.

[0111] The setting of the dish area reduces the impact of the placement of dishes and chopsticks at the edge of the table on recognition accuracy and improves recognition accuracy.

[0112] Dynamic inspection of the dish area can be used to analyze the status of the table, indicating the diversity and accuracy of the analysis.

[0113] Based on the functions of the video capture device shown in the above embodiments, the present application also provides a restaurant management method based on a restaurant management system. Specifically, the restaurant management system provided in the present application and connected to the captured video supports online video capture, can associate the device serial number of the video capture with the table number in the restaurant management system, can receive the video stream data of the captured video, and supports setting table boundaries and desktop boundaries for the video capture device; supports receiving the table status, abnormal alarms, and equipment operating status reports of the video capture. The use of the corresponding table status information and abnormal alarms by the restaurant management system is used as a conventional method in this application, and the corresponding use can also be changed. The personnel status and table status identified by the video capture device provide sufficient basis for the restaurant management system to carry out information management.

[0114] Please combine the specific Figure 2 Continue reading Figure 7 , Figure 7 It is a flow chart of an embodiment of a restaurant management method provided by this application.

[0115] like Figure 7 As shown, the specific steps are as follows:

[0116] Step S41: Obtain the table status of the personnel from any video acquisition device.

[0117] Step S42: Acquire the business status of the dish area in the dining table from the video acquisition device, or recognize the business status of the dish area from the dining table image uploaded by the video acquisition device.

[0118] In an embodiment of the present application, the table status of personnel and the business status of the food area can be confirmed by the video acquisition device through the status recognition method introduced in the above embodiment, or can be identified and confirmed by the restaurant management system of the embodiment of the present application using the table image captured by the video acquisition device.

[0119] Step S43: Based on the status of the people at the table and the business status of the food area, determine the business status of the dining table associated with the video acquisition device. The business status of the dining table can be any one of the ordering status, serving status, meal completion status, cleaning status, and idle status.

[0120] In the embodiment of the present application, the restaurant management system determines the business status of the table by combining the status of the staff on the table and the business status of the dish area. The restaurant management system can pre-store a state correspondence table and determine the business status of the table by querying the table; or it can identify the status of the staff on the table and the business status of the dish area in real time through a large model, and the corresponding business status of the table.

[0121] In a specific implementation, when diners are identified in the personnel area and the food area is idle, the table management system is notified to the corresponding table to enter the ordering state, and the table management system notifies the corresponding table staff to provide ordering service as soon as possible.

[0122] In a specific implementation, when diners are identified in the personnel area and the food area is not idle, the table management system is notified to notify the corresponding table to enter the food serving state, and the table management system notifies the corresponding kitchen staff to provide the food as soon as possible.

[0123] In a specific implementation, when diners are identified in the personnel area and the food area is not idle, and the food area screen does not change within 5 minutes, or no diners are detected at the table for 3 minutes, the table management system is notified that the corresponding table has finished dining. The table management system notifies the hall manager to pay attention to the checkout service and arrange for the table to be cleaned.

[0124] In a specific implementation, when only staff members are identified in the personnel area and the food area is not idle, the table management system is notified of the table cleaning status.

[0125] In a specific implementation, when the dish area changes from a non-free state to a free state, the table management system is notified that the corresponding table enters a free state, and the table management system notifies the front desk that the table is free and can be used to arrange calling numbers.

[0126] The restaurant management system of the present application can also use video acquisition equipment to identify abnormal sound events, identify the rapid reduction of personnel, and trigger alarm events. After the restaurant manager identifies the alarm event, he will notify the staff to handle it in time.

[0127] The restaurant management system of the present application can collect data from nearby tables by looking down, so it is more accurate than the traditional side-mounted collection method in receiving abnormal sound events.

[0128] The restaurant management system of this application only collects data from the dining table. When an alarm occurs, it provides an effective decision-making basis for the restaurant management system to notify the corresponding staff to handle it.

[0129] The restaurant management system of the present application recognizes personnel quickly and reduces the recognition algorithm trained based on its own large model. The recognition is accurate and fast, providing a basis for the restaurant management system to notify staff to pay attention to emergencies and table checkouts.

[0130] The restaurant management system of the present application uses the lens occlusion technology used by the video acquisition device to identify whether the lens has water mist or oil stains and send status information. The restaurant management system promptly notifies the staff to handle the matter based on the status information, and the video acquisition device of the present application also reports its own operating status.

[0131] In summary, compared with other video recognition solutions, this application takes into account the frequent movement of people in the restaurant environment, and uses an oblique installation method, which makes the picture easily blocked and disturbed; this application uses a ceiling-mounted installation to provide an unobstructed field of view and sufficient light source, which simplifies the recognition picture and improves the recognition rate and accuracy.

[0132] Compared with other video recognition methods that use human figure recognition, clothing recognition, and secondary matching of dining tables, according to industry common sense, the recognition rate and accuracy are not high; this application uses head and shoulder recognition trained with a large model, with sufficient light, unobstructed field of view, high recognition rate, and accurate recognition.

[0133] Compared with other video recognition solutions, this application uses one video capture corresponding to one dining table. Compared with other video recognition solutions used for multiple dining tables, it has a high recognition rate, small impact after failure, and is easier to recover.

[0134] Compared with other solutions, this application identifies staff and diners; identifies whether diners are at the table; identifies whether the food area is idle and whether any motion detection has occurred, providing more accurate and diverse information.

[0135] Compared with other solutions of installing sensors on dining tables and chairs and using wearable devices on staff, based on common sense, this application uses video capture, which is easy to maintain, not easy to damage, low cost, and conducive to long-term use.

[0136] Compared with other solutions, this application provides abnormal event alerts accurate to the dining table.

[0137] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0138] In order to implement the above-mentioned state recognition method and / or restaurant management method, the present application also proposes a restaurant management device, for details, please refer to Figure 8 , Figure 8 It is a structural diagram of an embodiment of a restaurant management device provided in this application.

[0139] The restaurant management device 400 of this embodiment includes a processor 41 , a memory 42 , an input / output device 43 , and a bus 44 .

[0140] The processor 41, memory 42, and input / output device 43 are respectively connected to a bus 44. The memory 42 stores program data. The processor 41 is used to execute the program data to implement the state identification method and / or restaurant management method described in the above embodiments.

[0141] In the embodiment of the present application, the processor 41 may also be referred to as a CPU (Central Processing Unit). The processor 41 may be an integrated circuit chip having the ability to process signals. The processor 41 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 41 may also be any conventional processor, etc.

[0142] This application also provides a computer storage medium, please continue to refer to Fig. 9 , Fig. 9 It is a structural diagram of an embodiment of a computer storage medium provided in the present application. The computer storage medium 600 stores a computer program 61. When the computer program 61 is executed by the processor, it is used to implement the state recognition method and / or restaurant management method of the above-mentioned embodiment.

[0143] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0144] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A state recognition method, characterized in that: The state recognition method is applied to a video acquisition device, comprising: Collect multiple consecutive frames of dining table images; Detect key points of the human body of the person within the preset table boundary range in each table image; Determining the personnel type of the personnel based on the human body key points of each dining table image; In response to the fact that the person is a non-staff member, the table status of the person is determined by combining the movement change rules of key points of the human body in the multiple consecutive frames of table images.

2. The state recognition method according to claim 1, characterized in that: The human body key points include head key points and shoulder key points; The determining the type of the person based on the key points of the human body in each dining table image includes: Based on the head key points and the shoulder key points, a personnel type differentiation area is expanded; In response to searching the personnel type distinguishing area for a staff member identifier, it is determined that the personnel type of the person is a staff member.

3. The state recognition method according to claim 2, characterized in that: After detecting the key points of the human body of a person within the preset table boundary range in each table image, the state recognition method further includes: Based on the head key points, determining the left and right order of the shoulder key points; Taking the shoulder key point as a base point, expanding the search for the shoulder clothes of the person; According to the left-right order of the shoulder key points and the shoulder clothes, the arm key points of the person are determined.

4. The state recognition method according to claim 2, characterized in that: After determining that the personnel type of the personnel is a staff member, the state identification method further includes: Comparing the staff identification of the personnel with a preset staff standard identification to obtain the image inspection result of the personnel; Wherein, the standard identification of the staff includes a work hat and / or a work shoulder label.

5. The state recognition method according to claim 1, characterized in that: The table boundary includes a food area and a personnel area, wherein the food area and the personnel area are separated by a table boundary; The method of detecting key points of a person's body within a preset table boundary range in each table image includes: The human body key points of the people are detected in the preset people area in each dining table image.

6. The state recognition method according to claim 5, characterized in that: The state recognition method further includes: Detecting dish images in a preset dish area in each table image; The service status of the dish area is determined by combining the comparison results of the dish images in the continuous multiple frames of table images with the preset idle images of the dish area.

7. The state recognition method according to claim 1, characterized in that: The video acquisition device is installed above the dining table, and the video acquisition device is associated with the identification of the dining table.

8. A restaurant management method, characterized in that: The restaurant management method is applied to a restaurant management system, wherein the restaurant management system is connected to a number of video acquisition devices; Get the table status of the personnel from any video acquisition device; Acquiring the business status of the dish area in the dining table from the video acquisition device, or identifying the business status of the dish area from the dining table image uploaded by the video acquisition device; Based on the table status of the personnel and the business status of the food area, determine the business status of the table associated with the video acquisition device, where the business status of the table is any one of ordering status, serving status, dining completion status, cleaning status, and idle status; Wherein, the video acquisition device identifies the table status of the person through the status recognition method described in any one of claims 1 to 7.

9. A restaurant management device, characterized in that: The restaurant management device includes a memory and a processor coupled to the memory; Wherein, the memory is used to store program data, and the processor is used to execute the program data to implement the state identification method as described in any one of claims 1 to 7, and / or the restaurant management method as described in claim 8.

10. A computer storage medium, characterized in that: The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the state identification method according to any one of claims 1 to 7, and / or the restaurant management method according to claim 8.

Images