Intelligent meal taking and weighing method

Abstract

The invention relates to an intelligent food taking and weighing method, belongs to the technical field of intelligent restaurants, and solves the problem of inaccurate food taking behavior recognition in the prior art. The method comprises the steps that dishes are arranged on a weighing table on a dining table, and the weight change of the dishes is detected through the weighing table; acquiring a human body behavior image by adopting a behavior camera, and shooting a human face image by adopting a human face camera; inputting the human body behavior image into a key point identification model for processing, and identifying a human body coordinate frame and key points; the key points comprise human limb joint points and key points of tableware; according to the human body coordinate frame, the position information of the key points and the weight change of the dishes, food taking behaviors are recognized; if only one meal taking behavior is recognized, identity information of a meal taker is obtained according to the human body coordinate frame and the face image of the meal taking behavior, and then the meal taking weight and the identity information of the meal taker are bound for settlement. The technical effect of accurately recognizing the meal taking behavior is achieved, and false alarm of multi-person meal taking is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent restaurants, and in particular to an intelligent meal picking and weighing method. Background Art

[0002] In existing smart restaurants, a combination of face recognition and behavior recognition technology is used to automatically identify the identity information and behavior of the person picking up the food, and the amount of food taken by the user is obtained through the weight change on the weighing table for settlement. However, the existing method of weighing food has the following problems:

[0003] 1. Inaccurate identification of meal-picking behavior. In the prior art, whether there is a meal-picking behavior is determined based on the relative position of the human body and the dining table. When multiple people take meals at the same dining table, an alarm will be issued. However, in the prior art, when multiple people and multiple behaviors occur at the same dining table at the same time, it is impossible to accurately identify the true meal-picking behavior among the multiple behaviors, which will also cause misjudgment of multiple people's meal-picking behaviors.

[0004] 2. Tableware weight error problem. The existing intelligent weighing platform strictly requires that only one tableware can be placed. In actual use, it is inevitable that customers use tableware on other weighing platforms or do not put the tableware back on the weighing platform after taking the food, which may lead to errors in the weight of the tableware.

[0005] 3. Loss of facial signal leads to inaccurate meal weight. In the prior art, due to the small wide-angle of the facial recognition camera, the person picking up the meal may lower his head or shake his head during the meal picking process, which may cause the person's face to be out of the shooting range of the facial camera, resulting in the loss of facial signal. This will not only lead to inaccurate detected meal weight, but also may cause the meal weight or meal picking behavior to be unable to match the identity information of the person picking up the meal, resulting in failure to settle the bill.

[0006] 4. Dual camera misalignment problem. In the prior art, a face recognition camera is used to capture a face image to detect identity information, and a behavior camera is used to capture a body image to detect body behavior information. The identity information and body behavior information are then associated and bound to determine the person who took the meal. Currently, the prior art requires strict regulation of the positions of the face camera and the behavior camera. Once the positions of the two cameras are misaligned, the system will not be able to respond, which will have a great impact on the binding effect of the face information and body behavior information, resulting in a deviation between the real person who took the meal and the person determined by the system. Summary of the invention

[0007] In view of the above analysis, an embodiment of the present invention aims to provide an intelligent meal picking and weighing method to solve the problem of inaccurate recognition of meal picking behavior in the prior art.

[0008] An embodiment of the present invention provides an intelligent meal weighing method, the method comprising:

[0009] The dishes are placed on a weighing table on the dining table, and the weight change of the dishes is detected by the weighing table;

[0010] A behavior camera is used to capture human behavior images, and a face camera is used to capture face images;

[0011] Input the human behavior image into a key point recognition model for processing, identify the human coordinate frame and key points, and obtain the position information of the key points; the key points include the key points of human limb joints and tableware;

[0012] Identifying a meal-taking behavior according to the position information of the human body coordinate frame and the key points and the weight change of the dish;

[0013] If only one meal-picking behavior is identified, the identity information of the person picking up the meal is obtained based on the coordinate frame of the human body where the meal-picking behavior occurs and the facial image, and then the weight change of the dish is used as the meal-picking weight and bound to the identity information of the person picking up the meal for settlement.

[0014] Based on the further improvement of the above method, the identifying the meal picking behavior according to the human body coordinate frame, the position information of the key points and the weight change of the dish includes:

[0015] Setting a meal-taking reference line on the processed human behavior image, wherein the meal-taking reference line is located between the edge line of the weighing platform and the edge line of the dining table; and a range defined between the meal-taking reference line and the bottom edge line of the human behavior image is used as a reference area;

[0016] Determining whether a meal-taking behavior occurs within the human body coordinate frame according to the reference area;

[0017] If the preset key point in the human body coordinate frame is located in the reference area, and the weight of the dish changes and lasts for a preset time, it is determined that a meal-taking behavior occurs in the human body coordinate frame.

[0018] Based on the further improvement of the above method, the dishes include dishes that need to be picked up with tableware and dishes that do not need to be picked up with tableware;

[0019] Wherein, when the dish is a dish that needs to be picked up with tableware, the hand joint points among the human limb joint points and the tableware key points are selected as the preset key points;

[0020] When the dish is a dish that does not require tableware to be taken, the hand joints among the human limb joints are selected as the preset key points.

[0021] Based on a further improvement of the above method, before judging whether a meal-taking behavior occurs in the human body coordinate frame according to the reference area, the method further includes:

[0022] Setting a screening line on the human behavior image, wherein the screening line is located outside the dining table;

[0023] Judging whether a human coordinate frame in the human behavior image may have a meal-taking behavior according to the screening line;

[0024] If the bottom of the human coordinate frame exceeds the screening line, it is determined that a meal-taking behavior may occur in the human coordinate frame, and then whether a meal-taking behavior occurs in the human coordinate frame is determined according to the reference area;

[0025] If the bottom of the human body coordinate frame does not exceed the screening line, it is determined that the meal-taking behavior is not possible in the human body coordinate frame, and then it is no longer determined whether the meal-taking behavior occurs in the human body coordinate frame based on the reference area.

[0026] Based on the further improvement of the above method, the weight change of the dish is used as the meal weight and is bound to the identity information of the person who takes the meal for settlement, including:

[0027] Generate an ID number according to the acquired identity information of the person picking up the meal, and dynamically bind the ID number to the coordinate frame of the person picking up the meal;

[0028] Track the human coordinate frame according to the ID serial number to obtain the time when the person who takes the meal completes taking the meal;

[0029] The weight change of the dish when the person picking up the food completes picking up the food is used as the weight of the food picked up and is bound to the identity information of the person picking up the food for settlement.

[0030] Based on the further improvement of the above method, the step of obtaining the identity information of the person who takes the meal according to the coordinate frame of the person who takes the meal and the face image includes:

[0031] Input the face image into a face recognition model for processing, identify the face coordinate frame and obtain the identity information of the person;

[0032] According to the coordinates, the coordinate frame of the human body that takes the meal is matched with the coordinate frame of the human face in the face image to obtain the identity information of the person who takes the meal.

[0033] Based on the further improvement of the above method, before position matching, the coordinates of the face image and the human behavior image are aligned, including the following steps:

[0034] Obtaining the position coordinate information of the target weighing platform in the shooting picture of the behavior camera;

[0035] Defining the best recognition range of the target weighing platform according to the position coordinate information of the target weighing platform, and extracting the image in the best recognition range as the human behavior image;

[0036] The center line of the human behavior image is aligned with the center line of the face image along the Y axis.

[0037] Based on the further improvement of the above method, the method further includes: before use, calibrating the behavior camera to avoid image redundancy and missing problems, including the following steps:

[0038] Step A1, obtaining a human behavior image captured by the behavior camera at a current position;

[0039] Step A2, judging whether there is a problem of image redundancy or missing according to the area and coordinates of the weighing platform in the human behavior image;

[0040] If there is a problem of image redundancy or missing, adjust the shooting angle of the behavior camera along the width direction of the dining table, and then return to perform the operations of step A1 to step A2;

[0041] If there is no image redundancy or loss, the calibration is complete.

[0042] Based on the further improvement of the above method, the method further includes: before use, calibrating the behavior camera to avoid the problem of table line tilt, including the following steps:

[0043] Step B1, obtaining a human behavior image captured by the behavior camera at a current position;

[0044] Step B2, detecting whether the dining table line in the human behavior image is parallel to the horizontal line of the shooting picture;

[0045] If not, it is determined that there is a problem of the table line being tilted, and the shooting angle of the behavior camera is adjusted counterclockwise or clockwise according to the angle between the table line and the horizontal line of the shooting screen, and then the operation of step B1 to step B2 is returned to be executed;

[0046] If yes, it is determined that there is no problem of table line tilt, and the calibration is completed.

[0047] Based on a further improvement of the above method, the tableware is provided with color features and / or shape features, and the positions of the color features and / or the shapes are used as key points of the tableware.

[0048] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0049] 1. In the present invention, the human body and tableware are regarded as a whole. The key points of the human limb joints and the tableware are all key points that need to be identified by the key point recognition model. Then, the meal-taking behavior is comprehensively identified based on the positions of the preset key points and the important changes in the dishes, so that the meal-taking behavior can be identified more accurately and the misjudgment of the meal-taking behavior of multiple people can be reduced.

[0050] 2. In the present invention, taking into account the different attributes of dishes, dishes are divided into two categories: dishes that require tableware to pick up and dishes that do not require tableware to pick up, and preset key point selection conditions are set for the two types of dishes, that is, different judgment conditions for picking up behaviors, which is conducive to the accurate identification of picking up behaviors.

[0051] At the same time, for dishes that require tableware to pick up, if the person picking up the food does not take the tableware, it will not be judged as a meal picking behavior, which can avoid the problem of errors in the weight of the meal due to the weight of the tableware.

[0052] 3. In the present invention, by setting a screening line to remove the human coordinate frame where it is impossible for the person to take the food, the number of key point position judgments can be reduced and the recognition efficiency can be improved.

[0053] 4. In the present invention, when the identity information of the person picking up the food is obtained, a corresponding ID number is generated, and the ID number is continuously and dynamically bound to the coordinate frame of the human body where the food picking behavior occurs, thereby realizing timely binding of the food picking behavior with the identity information of the person picking up the food, thereby being able to accurately track the location of the person picking up the food and the entire food picking process, and then obtaining the accurate food picking weight of the person picking up the food.

[0054] 5. In the present invention, there is no need to strictly regulate the positions of the face camera and the behavior camera. The optimal recognition range is obtained by detecting the position coordinate information of the target weighing platform in the shooting picture of the behavior camera, and the image in the optimal recognition range is extracted as the human behavior image. Then, the center line of the human behavior image and the center line of the face image are aligned along the Y axis, and the coordinate alignment of the two images is completed, which can ensure the accurate matching of the human coordinate frame and the face coordinate frame, and accurately obtain the identity information of the person picking up the meal.

[0055] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0057] Figure 1 This is a flow chart of a meal weighing method according to an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of human limb joints and a human coordinate frame according to an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of the positions of a meal pickup reference line and a screening line according to an embodiment of the present invention;

[0060] Figure 4 A schematic diagram of an optimal recognition range defined in an embodiment of the present invention;

[0061] Figure 5 A schematic diagram of a missing shot image according to an embodiment of the present invention;

[0062] Figure 6 A schematic diagram of redundancy of shooting images according to an embodiment of the present invention;

[0063] Figure 7 This is a schematic diagram of an embodiment of the present invention in which the dining table line is not parallel to the horizontal line of the shooting screen. DETAILED DESCRIPTION

[0064] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0065] An embodiment of the present invention provides an intelligent meal weighing method, such as Figure 1 The method comprises:

[0066] The dishes are placed on a weighing table on the dining table, and the weight change of the dishes is detected by the weighing table;

[0067] A behavior camera is used to capture human behavior images, and a face camera is used to capture face images;

[0068] Input the human behavior image into a key point recognition model for processing, identify the human coordinate frame and key points, and obtain the position information of the key points; the key points include the key points of human limb joints and tableware;

[0069] Identifying a meal-taking behavior according to the position information of the human body coordinate frame and the key points and the weight change of the dish;

[0070] If only one meal-picking behavior is identified, the identity information of the person picking up the meal is obtained based on the coordinate frame of the human body where the meal-picking behavior occurs and the facial image, and then the weight change of the dish is used as the meal-picking weight and bound to the identity information of the person picking up the meal for settlement.

[0071] In addition, if multiple meal-picking behaviors are identified, an alarm signal is issued to prompt multiple people to pick up meals.

[0072] During implementation, the key points on the tableware should have obvious features to improve the stability of identification. Specifically, the tableware is provided with color features and / or shape features, and the positions of the color features and / or the shapes are used as the key points of the tableware.

[0073] Compared with the prior art, in the embodiments of the present invention, the human body and tableware are regarded as a whole, and the key points of the human limb joints and the tableware are all key points that need to be identified by the key point recognition model. Then, the meal-taking behavior is comprehensively identified based on the positions of the preset key points and the weight changes of the dishes, thereby being able to more accurately identify the meal-taking behavior and reduce the misjudgment of the meal-taking behavior of multiple people.

[0074] Specifically, the key point model is a YOLOV-POSE model, such as a YOLOV7-POSE model, a YOLOV9-POSE model, etc. The YOLOV-POSE model is an extension based on the YOLO target detection framework, and is used for human posture estimation. It adds a key point detection function based on target detection, and can identify key points in an image, such as joint points, etc. The output of the YOLOV-POSE model is a set of coordinate points and target boxes representing the key points of objects in the image.

[0075] like Figure 2 As shown in , the model can detect the limb joints of the human body, the key points of the tableware and a human coordinate frame. During implementation, 17 limb joints of the human body and 3 key points on the tableware are used as targets for recognition.

[0076] In a specific embodiment, specifically, the YOLOV7-POSE model is used as the key point recognition model. The limb joint detection algorithm of the YOLOV7-POSE model is implemented based on the top-down method, that is, first detect the target frame of all people, and then estimate the posture of each frame and output the result. The YOLOV7-POSE model accepts the input image and generates a feature map, which is then used to predict the position of each person's limb joints. It can estimate the posture of multiple people in real time and is suitable for the scene of picking up food.

[0077] First, the human behavior images are acquired by shooting the food pickup area with a behavior recognition camera, and the acquired behavior images are input into the YOLOV7-POSE model.

[0078] The specific process of the algorithm implementation of the YOLOV7-POSE model is as follows:

[0079] a. The original size of the input image is 1280*720. First, a screenshot is taken according to the framed area, and the input image is adaptively adjusted to 640x640 size and input into the backbone network. After passing through 4 CBS modules, an ELAN module is connected, and then three groups of MP modules + ELAN modules are passed to obtain feature maps C3, C4, and C5 outputs with sizes of 80*80*512, 40*40*1024, and 20*20*1024 respectively.

[0080] b. For the 32x downsampled feature map C5 output by the backbone network, it first passes through the SPPCSP module, and the number of channels is changed from 1024 to 512. It is first fused with the feature maps C4 and C3 in a top-down manner to obtain feature maps P3, P4 and P5; then it is fused with the feature maps P4 and P5 in a bottom-up manner, and then three layers of feature maps of different sizes are output through the head layer network.

[0081] c. Using feature map P5, after adjusting the number of channels through the Rep-Conv module, 1x1 convolution is finally used to predict the three key information of class, bbox and keypoint, that is, the prediction results of the human body coordinate frame and key points are output.

[0082] It should be noted that the key point model in the embodiment of the present invention is a trained model. Specifically, training the key point model includes the following steps:

[0083] Obtain several human behavior images in different food pickup scenarios;

[0084] Marking key points and human coordinate frames on each of the human behavior images; wherein the marked key points include human limb joints and tableware key points;

[0085] Divide the labeled human behavior images into training sets and test sets according to a preset ratio;

[0086] Training the key point model using the training set;

[0087] The trained key point model is verified through the test set to obtain the trained key point recognition model.

[0088] In one embodiment, the identifying the meal picking behavior according to the human body coordinate frame, the position information of the key points and the weight change of the dish includes:

[0089] Setting a meal-taking reference line on the processed human behavior image, wherein the meal-taking reference line is located between the edge line of the weighing platform and the edge line of the dining table; and a range defined between the meal-taking reference line and the bottom edge line of the human behavior image is used as a reference area;

[0090] Determining whether a meal-taking behavior occurs within the human body coordinate frame according to the reference area;

[0091] If the preset key point in the human body coordinate frame is located in the reference area, and the weight of the dish changes and lasts for a preset time, it is determined that a meal-taking behavior occurs in the human body coordinate frame.

[0092] In the embodiment of the present invention, Figure 3 As shown in the figure, a meal-taking reference line is drawn between the edge line of the weighing table and the edge line of the dining table, and some of the identifiable key points (key points of human limb joints and tableware) are selected as preset key points. When the preset key points exceed the meal-taking reference line and enter the reference area, and the weight of the dish changes and lasts for a preset time, for example, within 1 second, it is determined that a meal-taking behavior has occurred.

[0093] Among them, in this embodiment, the meal-taking behavior is determined to have occurred only after the weight of the dish has changed and lasted for a preset time, so as to avoid misidentification of the meal-taking behavior caused by measurement errors.

[0094] Specifically, the dishes include dishes that require tableware to be picked up and dishes that do not require tableware to be picked up; when the dishes are dishes that require tableware to be picked up, the hand joint points among the human limb joints and the tableware key points are selected as the preset key points; when the dishes are dishes that do not require tableware to be picked up, the hand joint points among the human limb joints are selected as the preset key points.

[0095] In the embodiment of the present invention, taking into account the different attributes of dishes, dishes are divided into two categories: dishes that require tableware to be picked up and dishes that do not require tableware to be picked up, and preset key point selection conditions are set for the two categories of dishes, that is, different judgment conditions for picking up behaviors, which is conducive to the accurate identification of picking up behaviors.

[0096] For example, dishes that require tableware to be served include: meat dishes, vegetarian dishes, soups, etc.; dishes that do not require tableware to be served include fruits, drinks, etc.

[0097] At the same time, for dishes that require tableware to pick up, if the person picking up the food does not take a spoon, it will not be judged as a meal picking behavior, which can avoid the problem of errors in the weight of the meal due to the weight of the tableware.

[0098] Specifically, the interval between the edge line of the weighing table and the edge line of the dining table is greater than or equal to a preset distance.

[0099] In the embodiment of the present invention, considering that if the weighing platform is placed relatively close to the edge line of the dining table, the hand joints of people passing by this position are likely to exceed the meal picking reference lines (the first meal picking line and the second meal picking line), thereby causing misjudgment of the meal picking behavior, the weighing platform should not be placed too close to the edge line of the dining table to avoid misjudgment of the meal picking behavior.

[0100] Preferably, before judging whether a meal-taking behavior occurs in the human body coordinate frame according to the reference area, the method further comprises:

[0101] Setting a screening line on the human behavior image, wherein the screening line is located outside the dining table;

[0102] Judging whether a human coordinate frame in the human behavior image may have a meal-taking behavior according to the screening line;

[0103] If the bottom of the human coordinate frame exceeds the screening line, it is determined that a meal-taking behavior may occur in the human coordinate frame, and then whether a meal-taking behavior occurs in the human coordinate frame is determined according to the reference area;

[0104] If the bottom of the human body coordinate frame does not exceed the screening line, it is determined that the meal-taking behavior is not possible in the human body coordinate frame, and then it is no longer determined whether the meal-taking behavior occurs in the human body coordinate frame based on the reference area.

[0105] In the embodiment of the present invention, by setting a screening line to remove the human coordinate frame where it is impossible for the human body to take the food, the number of key point position judgments can be reduced and the recognition efficiency can be improved.

[0106] In one embodiment, the weight change of the dish is used as the meal weight and is bound to the identity information of the person who picks up the meal for settlement, including:

[0107] Generate an ID number according to the acquired identity information of the person picking up the meal, and dynamically bind the ID number to the coordinate frame of the person picking up the meal;

[0108] Track the human coordinate frame according to the ID serial number to obtain the time when the person who takes the meal completes taking the meal;

[0109] The weight change of the dish when the person picking up the food completes picking up the food is used as the weight of the food picked up and is bound to the identity information of the person picking up the food for settlement.

[0110] In an embodiment of the present invention, a corresponding ID number is generated when the identity information of the person picking up the meal is obtained, and the ID number is continuously and dynamically bound to the coordinate frame of the human body where the meal picking behavior occurs, thereby realizing timely binding of the meal picking behavior with the identity information of the person picking up the meal, thereby being able to accurately track the location of the person picking up the meal and the entire meal picking process, and then obtain the accurate meal picking weight of the person picking up the meal.

[0111] In one embodiment, the obtaining of the identity information of the person who takes the meal according to the coordinate frame of the person who takes the meal and the face image includes:

[0112] Input the face image into a face recognition model for processing, identify the face coordinate frame and obtain the identity information of the person;

[0113] According to the coordinates, the coordinate frame of the human body that takes the meal is matched with the coordinate frame of the human face in the face image to obtain the identity information of the person who takes the meal.

[0114] In an embodiment of the present invention, a facial image is acquired through a facial recognition camera, and facial recognition is performed through a facial recognition model to obtain identity information of all persons appearing in the meal pickup area, and then position matching is performed based on coordinates, so that the meal pickup behavior can be matched with the identity information.

[0115] During implementation, the face recognition model may adopt a neural network model, such as a DeepFace model or a FaceNet model.

[0116] Specifically, before position matching, aligning the face image with the human behavior image includes the following steps:

[0117] Positioning the target weighing platform on the center line of the face image;

[0118] Obtaining the position coordinate information of the target weighing platform in the shooting picture of the behavior camera;

[0119] Defining the best recognition range of the target weighing platform according to the position coordinate information of the target weighing platform, and extracting the image in the best recognition range as the human behavior image;

[0120] The center line of the human behavior image is aligned with the center line of the face image along the Y axis.

[0121] In the embodiment of the present invention, there is no need to strictly regulate the positions of the face camera and the behavior camera. The optimal recognition range is obtained by detecting the position coordinate information of the target weighing platform in the picture taken by the behavior camera, such as Figure 4As shown in the figure, the image in the optimal recognition range is extracted as the human behavior image, and then the center line of the human behavior image is aligned with the center line of the face image along the Y axis, that is, the coordinate alignment of the two images is completed, which can ensure the accurate matching of the human coordinate frame and the face coordinate frame, and accurately obtain the identity information of the person who picks up the meal.

[0122] When the optimal identification range of the target weighing platform is defined according to the position coordinate information of the target weighing platform, the target weighing platform should be located on the center line of the optimal identification range. Specifically, in the two-dimensional coordinate system of the shooting screen of the behavior camera, the bottom of the shooting screen is the X axis, the height of the shooting screen is H, and the coordinates of the four corners of the target weighing platform are defined as (x 1 ,y 1 )、(x 2 ,y 1 )、(x 1 ,y 2 )、(x 2 ,y 2 ), where; the coordinates of the four corners of the corresponding optimal recognition range are (x 1 -Δx,0)、(x 2 +Δx,0)、(x 1 -Δx,H)、(x 2 +Δx,H), wherein Δx>0, Δx is a deviation value, specifically, Δx is a preset value, or Δx is the distance between the target weighing platform and the adjacent weighing platform.

[0123] It should be noted that, in actual applications, the shooting range of the face camera is smaller, and the shooting range of the behavior camera is larger. Taking the above situation into consideration, in an embodiment of the present invention, the optimal recognition range is defined in the shooting picture of the behavior camera to extract the human behavior image.

[0124] In addition, if the images captured by the face camera and the behavior camera cannot be aligned automatically, an alarm signal will be issued to prompt the staff to adjust the installation position or installation angle of the two cameras.

[0125] In an embodiment of the present invention, during installation, the facial camera is arranged above the target weighing platform so that it shoots the head of the person to obtain a facial image, and the shooting center of the facial camera and the center line of the target weighing platform are located on the same vertical plane.

[0126] During installation, the behavior camera is arranged above the target weighing platform to shoot the body of the person, and the target weighing platform is included in the shooting picture of the behavior camera. The target weighing platform should be located in the center of the shooting picture to facilitate the identification of the meal picking behavior and avoid the loss of key information.

[0127] Specifically, the method further includes: before use, calibrating the behavior camera to avoid image redundancy and missing problems, including the following steps:

[0128] Step A1, obtaining a human behavior image captured by the behavior camera at a current position;

[0129] Step A2, judging whether there is a problem of image redundancy or missing according to the area and position coordinates of the target weighing platform in the human behavior image;

[0130] If there is a problem of image redundancy or missing, adjust the shooting angle of the behavior camera along the width direction of the dining table, and then return to perform the operations of step A1 to step A2;

[0131] If there is no image redundancy or loss, the calibration is complete.

[0132] Among them, the redundancy of the picture will lead to incomplete coordinate frames of the human body, and the head of the person may be missed, such as Figure 5 As shown in ; missing images will result in failure to identify the meal pickup behavior, such as Figure 6 as shown in .

[0133] In the embodiment of the present invention, whether image redundancy and missing problems occur is automatically detected according to the area of ​​the target weighing platform in the human behavior image, and then the shooting angle of the behavior camera is adjusted to avoid image redundancy and missing problems.

[0134] Specifically, judging whether there is image redundancy or missing according to the area and position coordinates of the target weighing platform in the human behavior image includes:

[0135] Convert the actual area of ​​the target weighing platform into an area threshold;

[0136] If the area of ​​the target weighing platform in the human behavior image is greater than zero and less than the area threshold, it is determined that there is a picture missing;

[0137] If the area of ​​the target weighing platform in the human behavior image is equal to the area threshold, and the distance between the coordinates of the preset point at the bottom edge of the target weighing platform and the coordinates of the bottom edge of the image is greater than the preset error value, it is determined that there is image redundancy;

[0138] If the area of ​​the target weighing platform in the human behavior image is equal to the area threshold, and the distance between the coordinates of the preset point at the bottom edge of the target weighing platform and the coordinates of the bottom edge of the picture is less than or equal to the preset error value, it is determined that there is no picture missing or redundancy.

[0139] It should be noted that, since the upper area threshold is converted from the actual area of ​​the target weighing platform, the area of ​​the target weighing platform in the human behavior image will not exceed the area threshold.

[0140] During implementation, the weighing platform is generally located at the lower side of the human behavior image. Therefore, if there is a missing image in the human behavior image, the shooting angle of the behavior camera is adjusted downward; if there is a redundant image in the human behavior image, the shooting angle of the behavior camera is adjusted upward.

[0141] It should be noted that if the weighing platform cannot be detected in the human behavior image, it means that the behavior camera has completely missed the key image and the position of the behavior camera needs to be manually readjusted.

[0142] Specifically, the method further includes: before use, calibrating the behavior camera to avoid the problem of tilting the dining table line, including the following steps:

[0143] Step B1, obtaining a human behavior image captured by the behavior camera at a current position;

[0144] Step B2, detecting whether the dining table line in the human behavior image is parallel to the horizontal line of the shooting picture;

[0145] If not, it is determined that there is a problem of the table line being tilted, and the shooting angle of the behavior camera is adjusted counterclockwise or clockwise according to the angle between the table line and the horizontal line of the shooting screen, and then the operation of step B1 to step B2 is returned to be executed;

[0146] If yes, it is determined that there is no problem of table line tilt, and the calibration is completed.

[0147] like Figure 7 As shown in , if the table line is not parallel to the horizontal line of the shooting screen, the behavior camera needs to be adjusted. Specifically, if the angle between the table line and the horizontal line of the shooting screen is less than 90°, the shooting angle of the behavior camera needs to be adjusted counterclockwise; if the angle between the table line and the horizontal line of the shooting screen is greater than 90°, the shooting angle of the behavior camera needs to be adjusted clockwise.

[0148] The horizontal line of the shooting picture refers to a virtual line in the horizontal direction of the shooting picture. For example, in a two-dimensional coordinate system of the shooting picture, the horizontal line of the shooting picture is a straight line parallel to the X-axis.

[0149] It should be noted that, in the embodiment of the present invention, the dining table line and the outer edge line of the weighing table are both straight lines, and when the weighing table is placed on the dining table, the outer edge line of the weighing table is parallel to the dining table line.

[0150] During implementation, when it is detected that the dining table line is parallel to the horizontal line of the shooting screen, the staff can be prompted by voice that the calibration of the tilt of the dining table line has been completed.

[0151] In addition, during implementation, the position coordinate information, area size, outer edge line of the scale body and dining table line of the target weighing platform can be obtained by using target detection models, such as the YOLOV model, Faster R-CNN model, etc.

[0152] It should be noted that in the intelligent meal picking and weighing method of the present invention, the weighing platform, face camera and behavior camera are hardware devices of the system, and the steps of meal picking behavior recognition, identity recognition and binding of meal picking weight with the identity information of the person picking up the meal are implemented based on the software module of the system and can be installed in a computer system or an Internet platform.

[0153] The intelligent meal pickup and weighing method of the embodiment of the present invention is an intelligent solution that integrates biometric recognition software such as face recognition and behavior recognition and intelligent equipment. It is an important part of a smart restaurant. It improves the restaurant's operating efficiency and customers' dining experience through intelligent equipment and artificial intelligence algorithms.

[0154] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0155] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. An intelligent meal weighing method, characterized in that: The method comprises: The dishes are placed on a weighing table on the dining table, and the weight change of the dishes is detected by the weighing table; A behavior camera is used to capture human behavior images, and a face camera is used to capture face images; Input the human behavior image into a key point recognition model for processing, identify the human coordinate frame and key points, and obtain the position information of the key points; the key points include the key points of human limb joints and tableware; Identifying a meal-taking behavior according to the position information of the human body coordinate frame and the key points and the weight change of the dish; If only one meal-picking behavior is identified, the identity information of the person picking up the meal is obtained based on the coordinate frame of the human body where the meal-picking behavior occurs and the facial image, and then the weight change of the dish is used as the meal-picking weight and bound to the identity information of the person picking up the meal for settlement.

2. The method according to claim 1, characterized in that The identifying the meal picking behavior according to the human body coordinate frame, the key point position information and the weight change of the dish includes: Setting a meal-taking reference line on the processed human behavior image, wherein the meal-taking reference line is located between the edge line of the weighing platform and the edge line of the dining table; and a range defined between the meal-taking reference line and the bottom edge line of the human behavior image is used as a reference area; Determining whether a meal-taking behavior occurs within the human body coordinate frame according to the reference area; If the preset key point in the human body coordinate frame is located in the reference area, and the weight of the dish changes and lasts for a preset time, it is determined that a meal-taking behavior occurs in the human body coordinate frame.

3. The method according to claim 2, characterized in that The dishes include dishes that require tableware to be served and dishes that do not require tableware to be served; Wherein, when the dish is a dish that needs to be picked up with tableware, the hand joint points among the human limb joint points and the tableware key points are selected as the preset key points; When the dish is a dish that does not require tableware to be taken, the hand joints among the human limb joints are selected as the preset key points.

4. The method according to claim 2 or 3, characterized in that: Before determining whether a meal-taking behavior occurs within the human body coordinate frame according to the reference area, the method further includes: Setting a screening line on the human behavior image, wherein the screening line is located outside the dining table; Judging whether a human coordinate frame in the human behavior image may have a meal-taking behavior according to the screening line; If the bottom of the human coordinate frame exceeds the screening line, it is determined that a meal-taking behavior may occur in the human coordinate frame, and then whether a meal-taking behavior occurs in the human coordinate frame is determined according to the reference area; If the bottom of the human body coordinate frame does not exceed the screening line, it is determined that the meal-taking behavior is not possible in the human body coordinate frame, and then it is no longer determined whether the meal-taking behavior occurs in the human body coordinate frame based on the reference area.

5. The method according to any one of claims 1 to 3, characterized in that: The method of binding the weight change of the dish as the meal pick-up weight and the identity information of the person picking up the dish for settlement includes: Generate an ID number according to the acquired identity information of the person picking up the meal, and dynamically bind the ID number to the coordinate frame of the person picking up the meal; Track the human coordinate frame according to the ID serial number to obtain the time when the person who takes the meal completes taking the meal; The weight change of the dish when the person picking up the food completes picking up the food is used as the weight of the food picked up and is bound to the identity information of the person picking up the food for settlement.

6. The method according to any one of claims 1 to 3, characterized in that The obtaining of the identity information of the person who takes the meal according to the coordinate frame of the person who takes the meal and the face image includes: Input the face image into a face recognition model for processing, identify the face coordinate frame and obtain the identity information of the person; According to the coordinates, the coordinate frame of the human body that takes the meal is matched with the coordinate frame of the human face in the face image to obtain the identity information of the person who takes the meal.

7. The method according to claim 6, characterized in that Before position matching, aligning the face image with the human behavior image includes the following steps: Obtaining the position coordinate information of the target weighing platform in the shooting picture of the behavior camera; Defining the best recognition range of the target weighing platform according to the position coordinate information of the target weighing platform, and extracting the image in the best recognition range as the human behavior image; The center line of the human behavior image is aligned with the center line of the face image along the Y axis.

8. The method according to any one of claims 1 to 3, characterized in that The method further includes: before use, calibrating the behavior camera to avoid image redundancy and missing problems, including the following steps: Step A1, obtaining a human behavior image captured by the behavior camera at a current position; Step A2, judging whether there is a problem of image redundancy or missing according to the area and coordinates of the weighing platform in the human behavior image; If there is a problem of image redundancy or missing, adjust the shooting angle of the behavior camera along the width direction of the dining table, and then return to perform the operations of step A1 to step A2; If there is no image redundancy or loss, the calibration is complete.

9. The method according to any one of claims 1 to 3, characterized in that: The method further includes: before use, calibrating the behavior camera to avoid the problem of table line tilt, including the following steps: Step B1, obtaining a human behavior image captured by the behavior camera at a current position; Step B2, detecting whether the dining table line in the human behavior image is parallel to the horizontal line of the shooting picture; If not, it is determined that there is a problem of the table line being tilted, and the shooting angle of the behavior camera is adjusted counterclockwise or clockwise according to the angle between the table line and the horizontal line of the shooting screen, and then the operation of step B1 to step B2 is returned to be executed; If yes, it is determined that there is no problem of table line tilt, and the calibration is completed.

10. The method according to any one of claims 1 to 3, characterized in that: The tableware is provided with color features and / or shape features, and the positions of the color features and / or the shapes are used as key points of the tableware.

Images