Intelligent meal taking and weighing system

Abstract

The invention relates to an intelligent food taking and weighing system, belongs to the technical field of intelligent restaurants, and solves the problem of inaccurate food taking behavior recognition in the prior art. The system comprises a weighing platform used for detecting the weight change of dishes; the first camera is used for shooting a human body behavior image; the second camera is used for shooting a face image; the meal taking behavior recognition module is used for processing the human body behavior image through a key point recognition model, identifying a human body coordinate frame and key points and acquiring position information of the key points; then, according to the human body coordinate frame, the position information of the key points and the weight change of the dishes, food taking behaviors are recognized; the identity recognition module is used for acquiring identity information of a meal taker when only one meal taking behavior is recognized; and the binding module is used for binding the weight change of the dishes as the meal taking weight with the identity information of the meal taker when only one meal taking behavior is identified. According to the system, the meal taking behavior can be accurately identified, and false alarm of multi-person meal taking is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent restaurants, and in particular to an intelligent meal picking and weighing system. Background Art

[0002] In existing smart restaurants, a facial recognition camera is installed above the dining table. The facial recognition camera captures the face image to determine the identity of the person picking up the meal. The face entry and exit are used as the basis for determining the start and end time of picking up the meal. The weight change on the weighing table is used to obtain the amount of food taken by the user, so as to make a settlement. However, the existing meal weighing system has the following problems:

[0003] 1. Inaccurate identification of meal-picking behaviors. In the prior art, when multiple people and multiple behaviors occur at a table at the same time, it is impossible to accurately identify the true meal-picking behaviors among the multiple behaviors, which may lead to misjudgment of multiple people's meal-picking behaviors.

[0004] 2. Tableware weight error problem. The existing intelligent weighing platform strictly requires that only one tableware can be placed. In actual use, it is inevitable that customers use tableware on other weighing platforms or do not put the tableware back on the weighing platform after taking the food, which may lead to errors in the weight of the tableware.

[0005] 3. Loss of facial signal leads to inaccurate meal weight. In the prior art, due to the small wide-angle of the facial recognition camera, the person picking up the meal may lower his head or shake his head during the meal picking process, which may cause the person's face to be out of the shooting range of the facial camera, resulting in the loss of facial signal. This will not only lead to inaccurate detected meal weight, but also may cause the meal weight or meal picking behavior to be unable to match the identity information of the person picking up the meal, resulting in failure to settle the bill. Summary of the invention

[0006] In view of the above analysis, an embodiment of the present invention aims to provide an intelligent meal pickup and weighing system to solve the problem of inaccurate recognition of meal pickup behavior in the prior art.

[0007] On the one hand, an embodiment of the present invention provides an intelligent meal weighing system, the system comprising:

[0008] A weighing platform is arranged on the dining table, and dishes are arranged on the weighing platform, and the weighing platform is used to detect the weight change of the dishes;

[0009] A first camera is used to capture images of human behavior;

[0010] The second camera is used to capture facial images;

[0011] The meal-taking behavior recognition module processes the human behavior image through a key point recognition model, identifies a human coordinate frame and key points, and obtains the position information of the key points; then recognizes the meal-taking behavior according to the human coordinate frame, the position information of the key points and the weight change of the dish; the key points include the key points of the human limb joints and the tableware;

[0012] An identity recognition module, when only one of the meal-picking behaviors is recognized, the identity recognition module obtains the identity information of the person who picks up the meal according to the coordinate frame of the human body where the meal-picking behavior occurs and the facial image;

[0013] The binding module, when only one meal-picking behavior is identified, binds the weight change of the dish as the meal-picking weight and the identity information of the person picking up the meal for settlement.

[0014] Based on the further improvement of the above system, the system comprises a plurality of the weighing platforms, a plurality of the second cameras and a plurality of the first cameras, and each of the weighing platforms is configured as a group with one of the second cameras and one of the first cameras;

[0015] Among them, the first camera is arranged above the weighing platform of the same group, and the first camera faces the weighing platform so that the weighing platform is located in the shooting picture of the first camera; the second camera is arranged above the weighing platform of the same group, and the second camera faces the head of the person located in front of the dining table, and the shooting center of the second camera and the center line of the weighing platform of the same group are located on the same vertical plane.

[0016] Based on the further improvement of the above system, the system further includes:

[0017] The alarm module sends out an alarm signal if the meal picking behavior recognition module recognizes multiple meal picking behaviors to prompt that multiple people are picking up meals.

[0018] Based on the further improvement of the above system, the meal picking behavior recognition module recognizes the meal picking behavior according to the human body coordinate frame, the key point position information and the weight change of the dish, including the following steps:

[0019] Setting a meal-taking reference line on the processed human behavior image, wherein the meal-taking reference line is located between the edge line of the weighing platform and the edge line of the dining table; and a range defined between the meal-taking reference line and the bottom edge line of the human behavior image is used as a reference area;

[0020] Determining whether a meal-taking behavior occurs within the human body coordinate frame according to the reference area;

[0021] If the preset key point in the human body coordinate frame is located in the reference area, and the weight of the dish changes and lasts for a preset time, it is determined that the meal picking behavior occurs.

[0022] Based on the further improvement of the above system, the dishes include dishes that require tableware to be taken and dishes that do not require tableware to be taken;

[0023] Wherein, when the dish is a dish that needs to be picked up with tableware, the hand joint points among the human limb joint points and the tableware key points are selected as the preset key points;

[0024] When the dish is a dish that does not require tableware to be taken, the hand joints among the human limb joints are selected as the preset key points.

[0025] Based on further improvement of the above system, the edge line of the dining table is parallel to the edge line of the weighing platform, and the interval between the edge line of the weighing platform and the edge line of the dining table is greater than or equal to a preset distance.

[0026] Based on the further improvement of the above system, before judging whether the meal picking behavior occurs in the human body coordinate frame according to the reference area, the meal picking behavior recognition module recognizes the meal picking behavior according to the human body coordinate frame, the key point position information and the weight change of the dish, and further includes the following steps:

[0027] Setting a screening line on the human behavior image, wherein the screening line is located outside the dining table;

[0028] Judging whether a human coordinate frame in the human behavior image may have a meal-taking behavior according to the screening line;

[0029] If the bottom of the human coordinate frame exceeds the screening line, it is determined that a meal-taking behavior may occur in the human coordinate frame, and then whether a meal-taking behavior occurs in the human coordinate frame is determined according to the reference area;

[0030] If the bottom of the human body coordinate frame does not exceed the screening line, it is determined that the meal-taking behavior is not possible in the human body coordinate frame, and then it is no longer determined whether the meal-taking behavior occurs in the human body coordinate frame based on the reference area.

[0031] Based on the further improvement of the above system, the identity recognition module obtains the identity information of the person who takes the meal according to the coordinate frame of the person who takes the meal and the face image, including:

[0032] Input the face image into a face recognition model for processing, identify the face coordinate frame and obtain the identity information of the person;

[0033] According to the coordinates, the coordinate frame of the human body that takes the meal is matched with the coordinate frame of the human face in the face image to obtain the identity information of the person who takes the meal.

[0034] Based on the further improvement of the above system, the method of matching the coordinate frame of the human body that takes the meal with the coordinate frame of the human face in the face image according to the coordinates includes the following steps:

[0035] Obtaining position coordinate information of a target weighing platform in the shooting picture of the first camera;

[0036] Defining the best recognition range of the target weighing platform according to the position coordinate information of the target weighing platform, and extracting the image in the best recognition range as the human behavior image;

[0037] Align the vertical center line of the human behavior image with the vertical center line of the human face image.

[0038] Based on the further improvement of the above system, the system further includes:

[0039] a dish recognition module, when the weighing platform detects that the weight of the dish increases by more than a preset weight threshold, the dish recognition module controls the first camera to capture a dish image, and recognizes dish information according to the dish image, and then the binding module binds the dish information to the weighing platform;

[0040] When only one meal-picking behavior is identified, the binding module binds the dish information with the meal-picking weight and the identity information of the person picking up the meal for settlement.

[0041] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0042] 1. In the present invention, the first camera and the second camera are used to capture human behavior images and facial images respectively, and the behavior recognition module recognizes the meal-picking behavior based on the human behavior images. When the behavior recognition module recognizes only one meal-picking behavior, the identity recognition module obtains the identity information of the person picking up the meal based on the facial image, and then the binding module binds the identity information of the person picking up the meal with the meal-picking weight, thereby accurately associating the meal-picking behavior, the identity information of the person picking up the meal, and the meal-picking weight.

[0043] Among them, when the behavior recognition module identifies the meal-taking behavior based on the human behavior image, the human body and tableware are regarded as a whole. The key points of the human limb joints and the tableware are all key points that need to be recognized by the key point recognition model. Then, the meal-taking behavior is comprehensively identified based on the position of the preset key points and the weight change of the dishes, so that the meal-taking behavior can be identified more accurately and the misjudgment of the meal-taking behavior of multiple people can be reduced.

[0044] 2. In the present invention, taking into account the different attributes of dishes, dishes are divided into two categories: dishes that require tableware to pick up and dishes that do not require tableware to pick up, and preset key point selection conditions are set for the two types of dishes, that is, different judgment conditions for picking up behaviors, which is conducive to the accurate identification of picking up behaviors.

[0045] At the same time, for dishes that require tableware to pick up, if the person picking up the food does not take the tableware, it will not be judged as a meal picking behavior, which can avoid the problem of errors in the weight of the meal due to the weight of the tableware.

[0046] 3. In the present invention, by setting a screening line to remove the human coordinate frame where it is impossible for the person to take the food, the number of key point position judgments can be reduced and the recognition efficiency can be improved.

[0047] 4. In the present invention, there is no need to strictly regulate the positions of the second camera and the first camera. The optimal recognition range is obtained by detecting the position coordinate information of the target weighing platform in the picture taken by the first camera, and the image in the optimal recognition range is extracted as the human behavior image. Then, the center line of the human behavior image and the center line of the face image are aligned along the Y-axis, and the coordinate alignment of the two images is completed, which can ensure the accurate matching of the human coordinate frame and the face coordinate frame, and accurately obtain the identity information of the person picking up the meal.

[0048] 5. In the present invention, the first camera is not only used to capture behavioral images, but also to capture dish images. The dish recognition module recognizes dish information based on dish images. At the same time, the restaurant manager can automatically control the dish recognition node by setting a weight threshold. Specifically, when the weighing platform detects that the weight of the dish has increased by more than the preset weight threshold, it can be considered that the restaurant manager is serving or adding dishes at this time. The dish recognition module controls the first camera to capture dish images and recognize dish information based on the dish images. Then, the binding module binds the dish information to the weighing platform, thereby realizing automatic update of dish information.

[0049] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0051] Figure 1This is a schematic diagram of the structure of an intelligent meal pickup and weighing system according to an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of human limb joints and a human coordinate frame according to an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of the positions of a meal pickup reference line and a screening line according to an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of an optimal recognition range defined in an embodiment of the present invention;

[0055] Figure 5 A schematic diagram of a missing shot image according to an embodiment of the present invention;

[0056] Figure 6 A schematic diagram of redundancy of shooting images according to an embodiment of the present invention;

[0057] Figure 7 This is a schematic diagram of an embodiment of the present invention in which the dining table line is not parallel to the horizontal line of the shooting screen.

[0058] Reference numerals:

[0059] 1-weighing table; 2-dining table; 3-first camera; 4-second camera; 5-display device. DETAILED DESCRIPTION

[0060] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0061] An embodiment of the present invention discloses an intelligent meal weighing system. Figure 1 The system comprises:

[0062] A weighing platform is arranged on the dining table, and dishes are arranged on the weighing platform, and the weighing platform is used to detect the weight change of the dishes;

[0063] A first camera is used to capture images of human behavior;

[0064] The second camera is used to capture facial images;

[0065] The meal-taking behavior recognition module processes the human behavior image through a key point recognition model, identifies a human coordinate frame and key points, and obtains the position information of the key points; then recognizes the meal-taking behavior according to the human coordinate frame, the position information of the key points and the weight change of the dish; the key points include the key points of the human limb joints and the tableware;

[0066] An identity recognition module, when only one of the meal-picking behaviors is recognized, the identity recognition module obtains the identity information of the person who picks up the meal according to the coordinate frame of the human body where the meal-picking behavior occurs and the facial image;

[0067] The binding module, when only one meal-picking behavior is identified, binds the weight change of the dish as the meal-picking weight and the identity information of the person picking up the meal for settlement.

[0068] During implementation, a container or space for placing dishes is provided on the weighing platform. For example, the weighing platform may be a buffet oven with a weighing function.

[0069] Compared with the prior art, in the embodiment of the present invention, the first camera and the second camera are used to capture human behavior images and facial images respectively, and the behavior recognition module recognizes the meal-picking behavior based on the human behavior image. When the behavior recognition module only recognizes one of the meal-picking behaviors, the identity recognition module obtains the identity information of the person picking up the meal based on the facial image, and then the binding module binds the identity information of the person picking up the meal with the meal-picking weight, thereby accurately associating the meal-picking behavior, the identity information of the person picking up the meal and the meal-picking weight.

[0070] Among them, when the behavior recognition module identifies the meal-taking behavior based on the human behavior image, the human body and tableware are regarded as a whole. The key points of the human limb joints and the tableware are all key points that need to be recognized by the key point recognition model. Then, the meal-taking behavior is comprehensively identified based on the position of the preset key points and the important changes in the dishes, so that the meal-taking behavior can be identified more accurately and the misjudgment of the meal-taking behavior of multiple people can be reduced.

[0071] In a specific embodiment, the system comprises a plurality of the weighing platforms, a plurality of the first cameras and a plurality of the second cameras, and each of the weighing platforms is configured as a group with one of the first cameras and one of the second cameras;

[0072] Wherein, the first camera is arranged above the weighing platform of the same group, and the first camera faces the weighing platform so that the weighing platform is located in the shooting picture of the first camera;

[0073] The second camera is arranged above the weighing platform of the same group, and the second camera is directed toward the head of the person in front of the dining table, and the shooting center of the second camera and the center line of the weighing platform of the same group are located on the same vertical plane.

[0074] In an embodiment of the present invention, a plurality of weighing platforms, a first camera and a second camera are arranged on the dining table, so that the meal picking and weighing management of a variety of dishes can be realized.

[0075] like Figure 1As shown in , the first camera is used to capture human behavior images. Therefore, the person picking up the food, the weighing platform, and the dishes on the weighing platform should be located in the first camera's shooting picture, thereby ensuring that the person picking up the food can be reflected in the human behavior images captured by the first camera. The second camera is used to capture facial images. The second camera accurately monitors the customer's head area, which helps to capture the customer's facial features. Its shooting center is on the same vertical plane as the center line of the weighing platform, ensuring the consistency and accuracy of the shooting picture and reducing the error caused by angle deviation.

[0076] Specifically, the system further includes: a calibration module, configured to calibrate the first camera before use.

[0077] The first camera is calibrated to avoid image redundancy and missing problems, and / or the problem of tilted dining table line.

[0078] Specifically, the calibration module calibrates the first camera to avoid image redundancy and missing problems, including the following steps:

[0079] Step A1, obtaining a human behavior image captured by the first camera at a current position;

[0080] Step A2, judging whether there is a problem of image redundancy or missing according to the area and position coordinates of the target weighing platform in the human behavior image;

[0081] If there is a problem of image redundancy or missing, adjust the shooting angle of the first camera along the width direction of the dining table, and then return to perform the operations from step A1 to step A2;

[0082] If there is no image redundancy or loss, the calibration is complete.

[0083] Among them, the redundancy of the picture will lead to incomplete coordinate frames of the human body, and the head of the person may be missed, such as Figure 5 As shown in ; missing images will result in failure to identify the meal pickup behavior, such as Figure 6 as shown in .

[0084] In the embodiment of the present invention, whether image redundancy and missing problems occur is automatically detected according to the area of ​​the target weighing platform in the human behavior image, and then the shooting angle of the first camera is adjusted to avoid image redundancy and missing problems.

[0085] More specifically, judging whether there is image redundancy or missing according to the area and position coordinates of the target weighing platform in the human behavior image includes:

[0086] Convert the actual area of ​​the target weighing platform into an area threshold;

[0087] If the area of ​​the target weighing platform in the human behavior image is greater than zero and less than the area threshold, it is determined that there is a picture missing;

[0088] If the area of ​​the target weighing platform in the human behavior image is equal to the area threshold, and the distance between the coordinates of the preset point at the bottom edge of the target weighing platform and the coordinates of the bottom edge of the image is greater than the preset error value, it is determined that there is image redundancy;

[0089] If the area of ​​the target weighing platform in the human behavior image is equal to the area threshold, and the distance between the coordinates of the preset point at the bottom edge of the target weighing platform and the coordinates of the bottom edge of the picture is less than or equal to the preset error value, it is determined that there is no picture missing or redundancy.

[0090] It should be noted that, since the above-mentioned area threshold is converted from the actual area of ​​the target weighing platform, the area of ​​the target weighing platform in the human behavior image will not exceed the area threshold.

[0091] During implementation, the weighing platform is generally located at the lower side of the human behavior image. Therefore, if there is a missing image in the human behavior image, the shooting angle of the behavior camera is adjusted downward; if there is a redundant image in the human behavior image, the shooting angle of the behavior camera is adjusted upward.

[0092] It should be noted that if the weighing platform cannot be detected in the human behavior image, it means that the first camera has completely missed the key image and the position of the first camera needs to be manually readjusted.

[0093] Specifically, the calibration module calibrates the first camera to avoid the problem of tilting the dining table line, including the following steps:

[0094] Step B1, obtaining a human behavior image captured by the behavior camera at a current position;

[0095] Step B2, detecting whether the dining table line in the human behavior image is parallel to the horizontal line of the shooting picture;

[0096] If not, it is determined that there is a problem of the table line being tilted, and the shooting angle of the first camera is adjusted counterclockwise or clockwise according to the angle between the table line and the horizontal line of the shooting picture, and then the operation of step B1 to step B2 is returned to be executed;

[0097] If yes, it is determined that there is no problem of table line tilt, and the calibration is completed.

[0098] like Figure 7As shown in , if the table line is not parallel to the horizontal line of the shooting screen, the behavior camera needs to be adjusted. Specifically, if the angle between the table line and the horizontal line of the shooting screen is less than 90°, the shooting angle of the behavior camera needs to be adjusted counterclockwise; if the angle between the table line and the horizontal line of the shooting screen is greater than 90°, the shooting angle of the behavior camera needs to be adjusted clockwise.

[0099] The horizontal line of the shooting picture refers to a virtual line in the horizontal direction of the shooting picture. For example, in a two-dimensional coordinate system of the shooting picture, the horizontal line of the shooting picture is a straight line parallel to the X-axis.

[0100] It should be noted that, in the embodiment of the present invention, the dining table line and the outer edge line of the weighing table are both straight lines, and when the weighing table is placed on the dining table, the outer edge line of the weighing table is parallel to the dining table line.

[0101] During implementation, when it is detected that the dining table line is parallel to the horizontal line of the shooting screen, the staff can be prompted by voice that the calibration of the tilt of the dining table line has been completed.

[0102] In addition, during implementation, the position coordinate information, area size, outer edge line of the scale body and dining table line of the target weighing platform can be obtained by using target detection models, such as the YOLOV model, Faster R-CNN model, etc.

[0103] In one embodiment, the system further comprises: an alarm module, and if the meal picking behavior recognition module recognizes a plurality of meal picking behaviors, the alarm module sends out an alarm signal to prompt that multiple people are picking up meals.

[0104] Specifically, the alarm signal may be a text signal, a light signal or a sound signal, which is respectively issued through a display device, an alarm light or an alarm.

[0105] In one embodiment, the system also includes: a dish identification module. When the weighing platform detects that the weight of the dish has increased by more than a preset weight threshold, the dish identification module controls the first camera to capture the dish image and identifies the dish information based on the dish image. Then the binding module binds the dish information to the weighing platform.

[0106] When only one meal-picking behavior is identified, the binding module binds the dish information with the meal-picking weight and the identity information of the person picking up the meal for settlement.

[0107] In the present invention, the first camera is not only used to capture behavioral images, but also to capture dish images. The dish recognition module recognizes dish information based on dish images. At the same time, the restaurant manager can automatically control the dish recognition node by setting a weight threshold. Specifically, when the weighing platform detects that the weight of the dish has increased by more than the preset weight threshold, it can be considered that the restaurant manager is serving or adding dishes at this time. The dish recognition module controls the first camera to capture dish images and recognizes dish information based on the dish images. Then, the binding module binds the dish information to the weighing platform, thereby realizing automatic update of dish information.

[0108] In a specific embodiment, the system further includes: a display device for displaying the dish information, and / or displaying the weight of the meal, and / or displaying the alarm signal issued by the alarm module.

[0109] When identifying the meal-picking behavior, the meal-picking behavior recognition module first processes the human behavior image through a key point recognition model, identifies the human coordinate frame and key points, and obtains the position information of the key points.

[0110] Specifically, the key point model is a YOLOV-POSE model, such as a YOLOV7-POSE model, a YOLOV9-POSE model, etc. The YOLOV-POSE model is an extension based on the YOLO target detection framework, and is used for human posture estimation. It adds a key point detection function based on target detection, and can identify key points in an image, such as joint points, etc. The output of the YOLOV-POSE model is a set of coordinate points and target boxes representing the key points of objects in the image.

[0111] like Figure 2 As shown in , the model can detect the limb joints of the human body, the key points of the tableware and a human coordinate frame. During implementation, 17 limb joints of the human body and 3 key points on the tableware are used as targets for recognition.

[0112] In a specific embodiment, specifically, the YOLOV7-POSE model is used as the key point recognition model. The limb joint detection algorithm of the YOLOV7-POSE model is implemented based on the top-down method, that is, first detect the target frame of all people, and then estimate the posture of each frame and output the result. The YOLOV7-POSE model accepts the input image and generates a feature map, which is then used to predict the position of each person's limb joints. It can estimate the posture of multiple people in real time and is suitable for the scene of picking up food.

[0113] First, the human behavior images are acquired by shooting the food pickup area with a behavior recognition camera, and the acquired behavior images are input into the YOLOV7-POSE model.

[0114] The specific process of the algorithm implementation of the YOLOV7-POSE model is as follows:

[0115] a. The original size of the input image is 1280*720. First, a screenshot is taken according to the framed area, and the input image is adaptively adjusted to 640x640 size and input into the backbone network. After passing through 4 CBS modules, an ELAN module is connected, and then three groups of MP modules + ELAN modules are passed to obtain feature maps C3, C4, and C5 outputs with sizes of 80*80*512, 40*40*1024, and 20*20*1024 respectively.

[0116] b. For the 32x downsampled feature map C5 output by the backbone network, it first passes through the SPPCSP module, and the number of channels is changed from 1024 to 512. It is first fused with the feature maps C4 and C3 in a top-down manner to obtain feature maps P3, P4 and P5; then it is fused with the feature maps P4 and P5 in a bottom-up manner, and then three layers of feature maps of different sizes are output through the head layer network.

[0117] c. Using feature map P5, after adjusting the number of channels through the Rep-Conv module, 1x1 convolution is finally used to predict the three key information of class, bbox and keypoint, that is, the prediction results of the human body coordinate frame and key points are output.

[0118] It should be noted that the key point model in the embodiment of the present invention is a trained model. Specifically, training the key point model includes the following steps:

[0119] Obtain several human behavior images in different food pickup scenarios;

[0120] Marking key points and human coordinate frames on each of the human behavior images; wherein the marked key points include human limb joints and tableware key points;

[0121] Divide the labeled human behavior images into training sets and test sets according to a preset ratio;

[0122] Training the key point model using the training set;

[0123] The trained key point model is verified through the test set to obtain the trained key point recognition model.

[0124] After the meal-taking behavior recognition module processes the human behavior image through a key point recognition model, the meal-taking behavior recognition module recognizes the meal-taking behavior according to the human coordinate frame, the position information of the key points and the weight change of the dish.

[0125] Specifically, the meal-picking behavior recognition module recognizes the meal-picking behavior according to the human body coordinate frame, the key point position information and the weight change of the dish, including the following steps:

[0126] Setting a meal-taking reference line on the processed human behavior image, wherein the meal-taking reference line is located between the edge line of the weighing platform and the edge line of the dining table; and a range defined between the meal-taking reference line and the bottom edge line of the human behavior image is used as a reference area;

[0127] Determining whether a meal-taking behavior occurs within the human body coordinate frame according to the reference area;

[0128] If the preset key point in the human body coordinate frame is located in the reference area, and the weight of the dish changes and lasts for a preset time, it is determined that the meal picking behavior occurs.

[0129] In the embodiment of the present invention, Figure 3 As shown in the figure, a meal-taking reference line is drawn between the edge line of the weighing table and the edge line of the dining table, and some of the identifiable key points (key points of human limb joints and tableware) are selected as preset key points. When the preset key points exceed the meal-taking reference line and enter the reference area, and the weight of the dish changes and lasts for a preset time, for example, within 1 second, it is determined that a meal-taking behavior has occurred.

[0130] Among them, in this embodiment, the meal-taking behavior is determined to have occurred only after the weight of the dish has changed and lasted for a preset time, so as to avoid misidentification of the meal-taking behavior caused by measurement errors.

[0131] More specifically, the dishes include dishes that require tableware to be served and dishes that do not require tableware to be served;

[0132] Wherein, when the dish is a dish that needs to be picked up with tableware, the hand joint points among the human limb joint points and the tableware key points are selected as the preset key points;

[0133] When the dish is a dish that does not require tableware to be taken, the hand joints among the human limb joints are selected as the preset key points.

[0134] For example, dishes that require tableware to be served include: meat dishes, vegetarian dishes, soups, etc.; dishes that do not require tableware to be served include fruits, drinks, etc.

[0135] At the same time, for dishes that require tableware to pick up, if the person picking up the food does not take a spoon, it will not be judged as a meal picking behavior, which can avoid the problem of errors in the weight of the meal due to the weight of the tableware.

[0136] Specifically, the edge line of the dining table is parallel to the edge line of the weighing table, and the interval between the edge line of the weighing table and the edge line of the dining table is greater than or equal to a preset distance.

[0137] In the embodiment of the present invention, considering that if the weighing platform is placed relatively close to the edge line of the dining table, the hand joints of people passing by this position are likely to exceed the meal picking reference lines (the first meal picking line and the second meal picking line), thereby causing misjudgment of the meal picking behavior, the weighing platform should not be placed too close to the edge line of the dining table to avoid misjudgment of the meal picking behavior.

[0138] Preferably, before judging whether a meal-taking behavior occurs in the human body coordinate frame according to the reference area, the meal-taking behavior recognition module recognizes the meal-taking behavior according to the human body coordinate frame, the key point position information and the weight change of the dish, and further comprises the following steps:

[0139] Setting a screening line on the human behavior image, wherein the screening line is located outside the dining table;

[0140] Judging whether a human coordinate frame in the human behavior image may have a meal-taking behavior according to the screening line;

[0141] If the bottom of the human coordinate frame exceeds the screening line, it is determined that a meal-taking behavior may occur in the human coordinate frame, and then whether a meal-taking behavior occurs in the human coordinate frame is determined according to the reference area;

[0142] If the bottom of the human body coordinate frame does not exceed the screening line, it is determined that the meal-taking behavior is not possible in the human body coordinate frame, and then it is no longer determined whether the meal-taking behavior occurs in the human body coordinate frame based on the reference area.

[0143] In the embodiment of the present invention, by setting a screening line to remove the human coordinate frame where it is impossible for the human body to take the food, the number of key point position judgments can be reduced and the recognition efficiency can be improved.

[0144] In one embodiment, the identity recognition module obtains the identity information of the person who takes the meal according to the coordinate frame of the person who takes the meal and the face image, including:

[0145] Input the face image into a face recognition model for processing, identify the face coordinate frame and obtain the identity information of the person;

[0146] According to the coordinates, the coordinate frame of the human body that takes the meal is matched with the coordinate frame of the human face in the face image to obtain the identity information of the person who takes the meal.

[0147] In an embodiment of the present invention, a facial image is captured by a second camera, and facial recognition is performed using a facial recognition model to obtain identity information of all persons appearing in the meal pickup area, and then position matching is performed based on coordinates, so that the meal pickup behavior can be matched with the identity information.

[0148] During implementation, the face recognition model may adopt a neural network model, such as a DeepFace model or a FaceNet model.

[0149] Specifically, the step of matching the coordinate frame of the human body that takes the meal with the coordinate frame of the human face in the face image according to the coordinates includes the following steps:

[0150] Acquire the position coordinate information of the target weighing platform in the shooting picture of the first camera; wherein the target weighing platform is a weighing platform in the same group as the first camera;

[0151] Defining the best recognition range of the target weighing platform according to the position coordinate information of the target weighing platform, and extracting the image in the best recognition range as the human behavior image;

[0152] Align the vertical center line of the human behavior image with the vertical center line of the human face image.

[0153] In the embodiment of the present invention, there is no need to strictly regulate the relative position between the face camera and the behavior camera. The optimal recognition range is obtained by detecting the position coordinate information of the target weighing platform in the picture taken by the behavior camera, such as Figure 4 As shown in the figure, the image in the optimal recognition range is extracted as the human behavior image, and then the center line of the human behavior image is aligned with the center line of the face image along the Y axis, that is, the coordinate alignment of the two images is completed, which can ensure the accurate matching of the human coordinate frame and the face coordinate frame, and accurately obtain the identity information of the person who picks up the meal.

[0154] Among them, when the optimal recognition range of the target weighing platform is defined according to the position coordinate information of the target weighing platform, the target weighing platform should be located on the center line of the optimal recognition range. Specifically, in the two-dimensional coordinate system of the shooting screen of the behavior camera, the bottom edge of the shooting screen is the X-axis, the height of the shooting screen is H, and the coordinates of the four corners of the target weighing platform are defined as (x1, y1), (x2, y1), (x1, y2), (x2, y2), wherein; then the coordinates of the four corners of the corresponding optimal recognition range are (x1-Δx, 0), (x2+Δx, 0), (x1-Δx, H), (x2+Δx, H), wherein Δx>0, Δx is the deviation value, specifically, Δx is a preset value, or Δx is the distance between the target weighing platform and the adjacent weighing platform.

[0155] It should be noted that, in actual applications, the shooting range of the face camera is smaller, and the shooting range of the behavior camera is larger. Taking the above situation into consideration, in an embodiment of the present invention, the optimal recognition range is defined in the shooting picture of the behavior camera to extract the human behavior image.

[0156] In addition, if the images captured by the face camera and the behavior camera cannot be aligned automatically, an alarm signal will be issued to prompt the staff to adjust the installation position or installation angle of the two cameras.

[0157] When installing the first camera, the weighing platform in the same group should be located in the center of the shooting screen to facilitate the identification of meal picking behavior and avoid missing key information.

[0158] In one embodiment, the binding module binds the weight change of the dish as the meal weight and the identity information of the person picking up the meal for settlement, including:

[0159] Generate an ID number according to the acquired identity information of the person picking up the meal, and dynamically bind the ID number to the coordinate frame of the person picking up the meal;

[0160] Track the human coordinate frame according to the ID serial number to obtain the time when the person who takes the meal completes taking the meal;

[0161] The weight change of the dish when the person picking up the food completes picking up the food is used as the weight of the food picked up and is bound to the identity information of the person picking up the food for settlement.

[0162] In an embodiment of the present invention, a corresponding ID number is generated when the identity information of the person picking up the meal is obtained, and the ID number is continuously and dynamically bound to the coordinate frame of the human body where the meal picking behavior occurs, thereby realizing timely binding of the meal picking behavior with the identity information of the person picking up the meal, thereby being able to accurately track the location of the person picking up the meal and the entire meal picking process, and then obtain the accurate meal picking weight of the person picking up the meal.

[0163] It should be noted that, in the embodiment of the present invention, the weighing platform, the first camera, the second camera and the display device are hardware devices of the system, and the meal picking behavior recognition module, the identity recognition module, the binding module, the alarm module, the dish recognition module, the calibration module are software modules of the system and can be installed in a computer system or an Internet platform.

[0164] The intelligent meal pickup and weighing system of the embodiment of the present invention is an intelligent solution that integrates biometric recognition software such as face recognition and behavior recognition and intelligent equipment. It is an important component of a smart restaurant. It improves the restaurant's operating efficiency and customers' dining experience through intelligent equipment and artificial intelligence algorithms.

[0165] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0166] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. An intelligent meal weighing system, characterized in that: The system comprises: A weighing platform is arranged on the dining table, and dishes are arranged on the weighing platform, and the weighing platform is used to detect the weight change of the dishes; A first camera is used to capture images of human behavior; The second camera is used to capture facial images; The meal-taking behavior recognition module processes the human behavior image through a key point recognition model, identifies a human coordinate frame and key points, and obtains the position information of the key points; then recognizes the meal-taking behavior according to the human coordinate frame, the position information of the key points and the weight change of the dish; the key points include the key points of the human limb joints and the tableware; An identity recognition module, when only one of the meal-picking behaviors is recognized, the identity recognition module obtains the identity information of the person who picks up the meal according to the coordinate frame of the human body where the meal-picking behavior occurs and the facial image; The binding module, when only one meal-picking behavior is identified, binds the weight change of the dish as the meal-picking weight and the identity information of the person picking up the meal for settlement.

2. The system according to claim 1, characterized in that The system comprises a plurality of the weighing platforms, a plurality of the second cameras and a plurality of the first cameras, wherein each of the weighing platforms is configured as a group with one of the second cameras and one of the first cameras; Among them, the first camera is arranged above the weighing platform of the same group, and the first camera faces the weighing platform so that the weighing platform is located in the shooting picture of the first camera; the second camera is arranged above the weighing platform of the same group, and the second camera faces the head of the person located in front of the dining table, and the shooting center of the second camera and the center line of the weighing platform of the same group are located on the same vertical plane.

3. The system according to claim 1 or 2, characterized in that: The system further comprises: The alarm module sends out an alarm signal if the meal picking behavior recognition module recognizes multiple meal picking behaviors to prompt that multiple people are picking up meals.

4. The system according to claim 1 or 2, characterized in that: The meal-taking behavior recognition module recognizes the meal-taking behavior according to the human body coordinate frame, the key point position information and the weight change of the dish, including the following steps: Setting a meal-taking reference line on the processed human behavior image, wherein the meal-taking reference line is located between the edge line of the weighing platform and the edge line of the dining table; and a range defined between the meal-taking reference line and the bottom edge line of the human behavior image is used as a reference area; Determining whether a meal-taking behavior occurs within the human body coordinate frame according to the reference area; If the preset key point in the human body coordinate frame is located in the reference area, and the weight of the dish changes and lasts for a preset time, it is determined that the meal picking behavior occurs.

5. The system according to claim 4, characterized in that The dishes include dishes that require tableware to be served and dishes that do not require tableware to be served; Wherein, when the dish is a dish that needs to be picked up with tableware, the hand joint points among the human limb joint points and the tableware key points are selected as the preset key points; When the dish is a dish that does not require tableware to be taken, the hand joints among the human limb joints are selected as the preset key points.

6. The system according to claim 4, characterized in that The edge line of the dining table is parallel to the edge line of the weighing table, and the interval between the edge line of the weighing table and the edge line of the dining table is greater than or equal to a preset distance.

7. The system according to claim 4, characterized in that Before determining whether a meal-taking behavior occurs in the human coordinate frame according to the reference area, the meal-taking behavior recognition module recognizes the meal-taking behavior according to the human coordinate frame, the key point position information and the weight change of the dish, and further includes the following steps: Setting a screening line on the human behavior image, wherein the screening line is located outside the dining table; Judging whether a human coordinate frame in the human behavior image may have a meal-taking behavior according to the screening line; If the bottom of the human coordinate frame exceeds the screening line, it is determined that a meal-taking behavior may occur in the human coordinate frame, and then whether a meal-taking behavior occurs in the human coordinate frame is determined according to the reference area; If the bottom of the human body coordinate frame does not exceed the screening line, it is determined that the meal-taking behavior is not possible in the human body coordinate frame, and then it is no longer determined whether the meal-taking behavior occurs in the human body coordinate frame based on the reference area.

8. The system according to claim 1 or 2, characterized in that: The identity recognition module obtains the identity information of the person who takes the meal according to the coordinate frame of the person who takes the meal and the face image, including: Input the face image into a face recognition model for processing, identify the face coordinate frame and obtain the identity information of the person; According to the coordinates, the coordinate frame of the human body that takes the meal is matched with the coordinate frame of the human face in the face image to obtain the identity information of the person who takes the meal.

9. The system according to claim 8, characterized in that The step of matching the coordinate frame of the human body that takes the meal with the coordinate frame of the human face in the face image according to the coordinates comprises the following steps: Obtaining position coordinate information of a target weighing platform in the shooting picture of the first camera; Defining the best recognition range of the target weighing platform according to the position coordinate information of the target weighing platform, and extracting the image in the best recognition range as the human behavior image; Align the vertical center line of the human behavior image with the vertical center line of the human face image.

10. The system according to claim 1 or 2, characterized in that: The system further comprises: a dish recognition module, when the weighing platform detects that the weight of the dish increases by more than a preset weight threshold, the dish recognition module controls the first camera to capture a dish image, and recognizes dish information according to the dish image, and then the binding module binds the dish information to the weighing platform; When only one meal-picking behavior is identified, the binding module binds the dish information with the meal-picking weight and the identity information of the person picking up the meal for settlement.

Images