An intelligent meal weighing method

Abstract

The present invention relates to an intelligent meal picking and weighing method, which belongs to the field of intelligent restaurant technology and solves the problem of inaccurate identification of meal picking behavior in the prior art. The method of the present invention comprises: placing dishes on a weighing table on the dining table, and detecting the weight change of the dishes by the weighing table; using a behavior camera to obtain human behavior images, and using a face camera to shoot human face images; inputting the human behavior images into a key point recognition model for processing, and identifying the human coordinate frame and key points; the key points include the key points of human limb joints and tableware; identifying the meal picking behavior based on the position information of the human coordinate frame and key points and the weight change of the dishes; if only one meal picking behavior is identified, the identity information of the person picking up the meal is obtained based on the human coordinate frame and face image of the person picking up the meal, and then binding the meal picking weight with the identity information of the person picking up the meal for settlement. The technical effect of accurately identifying the meal picking behavior is achieved, and false alarms of multiple people picking up meals are avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent restaurants, and in particular to an intelligent meal picking and weighing method. Background Art

[0002] Existing smart restaurants use a combination of facial recognition and behavioral recognition technology to automatically identify the identity and behavior of the person picking up the food. The user's food quantity is then calculated based on the weight change on the weighing table for settlement. However, existing methods for weighing food have the following problems:

[0003] 1. Inaccurate identification of food pickup behavior. Existing technology determines whether food pickup behavior occurs based on the relative position of the person and the table. When multiple people attempt to pick up food at the same table, an alarm is triggered. However, when multiple people attempt to pick up food at the same table at the same time, this technology cannot accurately identify the actual food pickup behavior among the multiple actions, leading to misjudgment of multiple people picking up food.

[0004] Second, the problem of tableware weight error. Existing smart weighing platforms strictly require that only one tableware can be placed on them. In actual use, it is inevitable that customers will use tableware from other weighing platforms or fail to put the tableware back on the weighing platform after taking the food, which may lead to errors in the tableware weight of the food taken.

[0005] 3. Loss of facial signals leading to inaccurate meal weights. In existing technologies, due to the narrow wide-angle of facial recognition cameras, situations such as lowering or shaking the head while picking up food may cause the face to fall out of the camera's range, resulting in loss of facial signals. This not only leads to inaccurate weight detection, but also may prevent settlement due to the inability to match the meal weight or meal pickup behavior with the picker's identity information.

[0006] 4. Dual camera misalignment. In existing technologies, facial recognition cameras are used to capture facial images to detect identity information, while behavioral cameras are used to capture body images to detect body behavior information. The identity information and body behavior information are then associated and bound to determine the person who took the meal. Currently, existing technologies require strict regulations on the positions of facial cameras and behavioral cameras. Once the positions of the two cameras are misaligned, the system will not be able to respond, which will have a significant impact on the binding effect of facial information and body behavior information, resulting in a deviation between the actual person who took the meal and the person determined by the system. Summary of the Invention

[0007] In view of the above analysis, an embodiment of the present invention aims to provide an intelligent meal pickup and weighing method to solve the problem of inaccurate recognition of meal pickup behavior in the prior art.

[0008] An embodiment of the present invention provides an intelligent meal picking and weighing method, the method comprising:

[0009] The dishes are placed on a weighing table on the dining table, and the weight change of the dishes is detected by the weighing table;

[0010] Use a behavior camera to capture human behavior images, and use a face camera to capture human face images;

[0011] Inputting the human behavior image into a key point recognition model for processing, identifying a human coordinate frame and key points, and obtaining position information of the key points; the key points include key points of human limb joints and tableware;

[0012] Identifying a meal-picking behavior based on the position information of the human body coordinate frame and the key points and the weight change of the dish;

[0013] If only one meal-picking behavior is identified, the identity information of the person picking up the meal is obtained based on the coordinate frame of the human body where the meal-picking behavior occurs and the facial image, and then the weight change of the dish is used as the meal-picking weight and bound to the identity information of the person picking up the meal for settlement.

[0014] Based on a further improvement of the above method, the method of identifying the meal-picking behavior based on the human body coordinate frame, the position information of the key points, and the weight change of the dish includes:

[0015] Setting a meal-taking reference line on the processed human behavior image, wherein the meal-taking reference line is located between the edge line of the weighing platform and the edge line of the dining table; and a range defined between the meal-taking reference line and the bottom edge line of the human behavior image is used as a reference area;

[0016] Determining whether a meal-taking action occurs within the human body coordinate frame according to the reference area;

[0017] If the preset key point in the human body coordinate frame is located in the reference area, and the weight of the dish changes and lasts for a preset time, it is determined that a meal-taking behavior occurs in the human body coordinate frame.

[0018] Based on a further improvement of the above method, the dishes include dishes that require tableware to be taken and dishes that do not require tableware to be taken;

[0019] Wherein, when the dish is a dish that requires tableware to be taken, the hand joint points among the human limb joint points and the tableware key points are selected as the preset key points;

[0020] When the dish is a dish that does not require tableware to be taken, the hand joints among the human limb joints are selected as the preset key points.

[0021] Based on a further improvement of the above method, before determining whether a meal-picking behavior occurs within the human coordinate frame according to the reference area, the method further includes:

[0022] Setting a screening line on the human behavior image, wherein the screening line is located outside the dining table;

[0023] Determining whether a human coordinate frame in the human behavior image may be in a meal-taking behavior based on the screening line;

[0024] If the bottom of the human coordinate frame exceeds the screening line, it is determined that a meal-taking behavior may occur in the human coordinate frame, and then whether a meal-taking behavior occurs in the human coordinate frame is determined based on the reference area;

[0025] If the bottom of the human body coordinate frame does not exceed the screening line, it is determined that the meal-taking behavior is impossible in the human body coordinate frame, and then whether the meal-taking behavior occurs in the human body coordinate frame is no longer determined based on the reference area.

[0026] Based on a further improvement of the above method, the weight change of the dish is used as the meal weight and is bound to the identity information of the person who picks up the meal for settlement, including:

[0027] Generate an ID number based on the acquired identity information of the person picking up the meal, and dynamically bind the ID number to the coordinate frame of the person who is picking up the meal;

[0028] Track the human coordinate frame according to the ID number to obtain the time when the person who takes the meal completes taking the meal;

[0029] The weight change of the dish when the person picking up the food completes the food picking up, is used as the food picking up weight and is bound to the identity information of the person picking up the food for settlement.

[0030] Based on a further improvement of the above method, the method of obtaining the identity information of the person who takes the meal according to the coordinate frame of the person who takes the meal and the facial image includes:

[0031] Inputting the facial image into a face recognition model for processing, identifying the facial coordinate frame and obtaining the identity information of the person;

[0032] The coordinate frame of the human body that takes the meal is matched with the coordinate frame of the human face in the face image according to the coordinates to obtain the identity information of the person who takes the meal.

[0033] Based on a further improvement of the above method, before position matching, the coordinates of the face image and the human behavior image are aligned, including the following steps:

[0034] Obtaining the position coordinate information of the target weighing platform in the shooting picture of the behavior camera;

[0035] Defining an optimal recognition range of the target weighing platform according to the position coordinate information of the target weighing platform, and extracting an image in the optimal recognition range as the human behavior image;

[0036] The center line of the human behavior image is aligned with the center line of the human face image along the Y axis.

[0037] Based on a further improvement of the above method, the method further includes: before use, calibrating the behavior camera to avoid image redundancy and missing problems, including the following steps:

[0038] Step A1, obtaining a human behavior image captured by the behavior camera at a current position;

[0039] Step A2, judging whether there is a problem of image redundancy or missing according to the area and coordinates of the weighing platform in the human behavior image;

[0040] If there is a problem of image redundancy or missing, adjust the shooting angle of the behavior camera along the width direction of the dining table, and then return to perform the operations of step A1 to step A2;

[0041] If there are no redundant or missing images, the calibration is complete.

[0042] Based on a further improvement of the above method, the method further includes: before use, calibrating the behavior camera to avoid the problem of table line tilt, including the following steps:

[0043] Step B1, obtaining a human behavior image captured by the behavior camera at a current position;

[0044] Step B2, detecting whether the dining table line in the human behavior image is parallel to the horizontal line of the shooting picture;

[0045] If not, it is determined that there is a problem with the table line being tilted, and the shooting angle of the behavior camera is adjusted counterclockwise or clockwise according to the angle between the table line and the horizontal line of the shooting screen, and then the operation of step B1 to step B2 is returned to be executed;

[0046] If so, it is determined that there is no problem of table line tilt and the calibration is completed.

[0047] Based on a further improvement of the above method, the tableware is provided with color features and / or shape features, and the positions of the color features and / or the shapes are used as key points of the tableware.

[0048] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0049] 1. In the present invention, the human body and tableware are regarded as a whole. The key points of the human limb joints and the tableware are all key points that need to be identified by the key point recognition model. Then, the meal-taking behavior is comprehensively identified based on the positions of the preset key points and the important changes in the dishes, so that the meal-taking behavior can be identified more accurately and the misjudgment of the meal-taking behavior of multiple people can be reduced.

[0050] 2. In the present invention, taking into account the different attributes of dishes, dishes are divided into two categories: dishes that require tableware to pick up and dishes that do not require tableware to pick up. Preset key point selection conditions are set for the two types of dishes, that is, different judgment conditions for picking up behaviors, which are conducive to the accurate identification of picking up behaviors.

[0051] At the same time, for dishes that require tableware to pick up, if the person picking up the food does not take the tableware, it will not be judged as a meal picking behavior, which can avoid the problem of errors in the weight of the food picked up due to the weight of the tableware.

[0052] 3. In the present invention, by setting a screening line to remove the human coordinate frame where it is impossible for the person to take the food, the number of key point position judgments can be reduced and the recognition efficiency can be improved.

[0053] 4. In the present invention, when the identity information of the person picking up the meal is obtained, a corresponding ID number is generated, and the ID number is continuously and dynamically bound to the coordinate frame of the human body where the meal picking behavior occurs, thereby realizing timely binding of the meal picking behavior with the identity information of the person picking up the meal, thereby being able to accurately track the location of the person picking up the meal and the entire meal picking process, and then obtain the accurate weight of the meal picked up by the person picking up the meal.

[0054] 5. In the present invention, there is no need to strictly regulate the positions of the face camera and the behavior camera. The optimal recognition range is obtained by detecting the position coordinate information of the target weighing platform in the shooting picture of the behavior camera, and the image in the optimal recognition range is extracted as the human behavior image. Then, the center line of the human behavior image and the center line of the face image are aligned along the Y-axis, and the coordinate alignment of the two images is completed, which can ensure the accurate matching of the human coordinate frame and the face coordinate frame, and accurately obtain the identity information of the person picking up the meal.

[0055] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0057] Figure 1 This is a flow chart of a meal weighing method according to an embodiment of the present invention;

[0058] Figure 2 Schematic diagram of human limb joints and a human coordinate frame according to an embodiment of the present invention;

[0059] Figure 3 This is a schematic diagram of the positions of the meal pickup reference line and the screening line according to an embodiment of the present invention;

[0060] Figure 4 A schematic diagram of an optimal recognition range defined in an embodiment of the present invention;

[0061] Figure 5 A schematic diagram of a missing shot image according to an embodiment of the present invention;

[0062] Figure 6 A schematic diagram of redundancy of shooting images according to an embodiment of the present invention;

[0063] Figure 7 This is a schematic diagram of an embodiment of the present invention in which the dining table line is not parallel to the horizontal line of the shooting screen. DETAILED DESCRIPTION

[0064] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0065] An embodiment of the present invention provides an intelligent meal weighing method, such as Figure 1 The method comprises:

[0066] The dishes are placed on a weighing table on the dining table, and the weight change of the dishes is detected by the weighing table;

[0067] Use a behavior camera to capture human behavior images, and use a face camera to capture human face images;

[0068] Inputting the human behavior image into a key point recognition model for processing, identifying a human coordinate frame and key points, and obtaining position information of the key points; the key points include key points of human limb joints and tableware;

[0069] Identifying a meal-picking behavior based on the position information of the human body coordinate frame and the key points and the weight change of the dish;

[0070] If only one meal-picking behavior is identified, the identity information of the person picking up the meal is obtained based on the coordinate frame of the human body where the meal-picking behavior occurs and the facial image, and then the weight change of the dish is used as the meal-picking weight and bound to the identity information of the person picking up the meal for settlement.

[0071] In addition, if multiple meal-picking behaviors are identified, an alarm signal is issued to prompt multiple people to pick up their meals.

[0072] In practice, the key points on the tableware should have obvious features to improve the stability of recognition. Specifically, the tableware is provided with color features and / or shape features, and the positions of the color features and / or shapes are used as the key points of the tableware.

[0073] Compared with the prior art, in the embodiment of the present invention, the human body and tableware are regarded as a whole, and the key points of the human limb joints and the tableware are all key points that need to be identified by the key point recognition model. Then, the meal-taking behavior is comprehensively identified based on the position of the preset key points and the weight changes of the dishes, so that the meal-taking behavior can be identified more accurately and the misjudgment of the meal-taking behavior of multiple people can be reduced.

[0074] Specifically, the key point model is a YOLO V-POSE model, such as the YOLO V7-POSE model or the YOLO V9-POSE model. The YOLO V-POSE model is an extension of the YOLO object detection framework for human pose estimation. It adds key point detection to object detection and can identify key points in an image, such as joints. The output of the YOLO V-POSE model is a set of coordinate points and target bounding boxes representing key points of objects in the image.

[0075] like Figure 2 As shown in , the model can detect the human body's limb joints, the key points of the tableware, and a human coordinate frame. During implementation, the 17 limb joints of the human body and the 3 key points on the tableware are used as targets for recognition.

[0076] In a specific embodiment, the YOLOV7-POSE model is used as a key point recognition model. The limb joint detection algorithm of the YOLOV7-POSE model is implemented based on the top-down method, that is, the target frame of each person is first detected, and then the posture of each frame is estimated and the result is output. The YOLOV7-POSE model accepts the input image and generates a feature map, which is then used to predict the position of each person's limb joints. It can estimate the posture of multiple people in real time and is suitable for scenarios where people pick up food.

[0077] First, the behavior recognition camera is used to capture human behavior images in the food pickup area, and the acquired behavior images are input into the YOLOV7-POSE model.

[0078] The specific process of the algorithm implementation of the YOLOV7-POSE model is as follows:

[0079] a. The original size of the input image is 1280*720. First, a screenshot is taken based on the framed area and adaptively resized to 640x640. The image is then fed into the backbone network. After passing through four CBS modules, an ELAN module is connected. Then, three MP modules plus an ELAN module are passed through to produce feature maps C3, C4, and C5 output with sizes of 80*80*512, 40*40*1024, and 20*20*1024, respectively.

[0080] b. The 32x downsampled feature map C5, the final output of the backbone network, first passes through the SPPCSP module, reducing the number of channels from 1024 to 512. It is first fused top-down with feature maps C4 and C3 to produce feature maps P3, P4, and P5. It is then fused bottom-up with feature maps P4 and P5, and then passes through the head layer network to output three layers of feature maps of different sizes.

[0081] c. Using feature map P5, after adjusting the number of channels through the Rep-Conv module, 1x1 convolution is used to predict the three key pieces of information: class, bbox, and keypoint. The prediction results of the human coordinate frame and keypoints are output.

[0082] It should be noted that the key point model in the embodiment of the present invention is a trained model. Specifically, training the key point model includes the following steps:

[0083] Obtain several human behavior images in different food pickup scenarios;

[0084] Marking key points and human coordinate frames on each of the human behavior images; wherein the marked key points include human limb joints and tableware key points;

[0085] The labeled human behavior images are divided into training sets and test sets according to a preset ratio;

[0086] Training the key point model using the training set;

[0087] The trained key point model is verified through the test set to obtain the trained key point recognition model.

[0088] In one embodiment, the identifying the meal-picking behavior based on the human body coordinate frame, the position information of the key points, and the weight change of the dish includes:

[0089] Setting a meal-taking reference line on the processed human behavior image, wherein the meal-taking reference line is located between the edge line of the weighing platform and the edge line of the dining table; and a range defined between the meal-taking reference line and the bottom edge line of the human behavior image is used as a reference area;

[0090] Determining whether a meal-taking action occurs within the human body coordinate frame according to the reference area;

[0091] If the preset key point in the human body coordinate frame is located in the reference area, and the weight of the dish changes and lasts for a preset time, it is determined that a meal-taking behavior occurs in the human body coordinate frame.

[0092] In the embodiment of the present invention, Figure 3 As shown in , a meal-taking reference line is drawn between the edge line of the weighing table and the edge line of the dining table, and some of the identifiable key points (key points of human limb joints and tableware) are selected as preset key points. When the preset key points exceed the meal-taking reference line and enter the reference area, and the weight of the dish changes and lasts for a preset time, for example, within 1 second, it is determined that a meal-taking behavior has occurred.

[0093] In this embodiment, the meal-picking behavior is determined to have occurred only after the weight of the dish has changed and lasted for a preset time, so as to avoid misidentification of the meal-picking behavior due to measurement errors.

[0094] Specifically, the dishes include dishes that require tableware to be picked up and dishes that do not require tableware to be picked up; when the dishes are dishes that require tableware to be picked up, the hand joints among the human limb joints and the tableware key points are selected as the preset key points; when the dishes are dishes that do not require tableware to be picked up, the hand joints among the human limb joints are selected as the preset key points.

[0095] In an embodiment of the present invention, taking into account the different attributes of dishes, dishes are divided into two categories: dishes that require tableware to pick up and dishes that do not require tableware to pick up. Preset key point selection conditions are set for the two categories of dishes, that is, different judgment conditions for picking up behaviors, which is conducive to the accurate identification of picking up behaviors.

[0096] For example, dishes that require tableware include: meat dishes, vegetarian dishes, soups, etc.; dishes that do not require tableware include fruits, drinks, etc.

[0097] At the same time, for dishes that require tableware to pick up, if the person picking up the food does not take a spoon, it will not be judged as a picking up behavior, which can avoid the problem of errors in the weight of the food due to the weight of the tableware.

[0098] Specifically, the interval between the edge line of the weighing platform and the edge line of the dining table is greater than or equal to a preset distance.

[0099] In the embodiment of the present invention, considering that if the weighing platform is placed relatively close to the edge line of the dining table, the hand joints of people passing by this position are likely to exceed the meal-taking reference lines (the first meal-taking line and the second meal-taking line), thereby causing misjudgment of the meal-taking behavior, the weighing platform should not be placed too close to the edge line of the dining table to avoid misjudgment of the meal-taking behavior.

[0100] Preferably, before determining whether a meal-taking behavior occurs within the human body coordinate frame based on the reference area, the method further includes:

[0101] Setting a screening line on the human behavior image, wherein the screening line is located outside the dining table;

[0102] Determining whether a human coordinate frame in the human behavior image may be in a meal-taking behavior based on the screening line;

[0103] If the bottom of the human coordinate frame exceeds the screening line, it is determined that a meal-taking behavior may occur in the human coordinate frame, and then whether a meal-taking behavior occurs in the human coordinate frame is determined based on the reference area;

[0104] If the bottom of the human body coordinate frame does not exceed the screening line, it is determined that the meal-picking behavior is impossible in the human body coordinate frame, and then whether the meal-picking behavior occurs in the human body coordinate frame is no longer determined based on the reference area.

[0105] In the embodiment of the present invention, by setting a screening line to remove the human coordinate frame where the human body is unlikely to take the food, the number of key point position judgments can be reduced and the recognition efficiency can be improved.

[0106] In one embodiment, the weight change of the dish is used as the meal weight and is bound to the identity information of the person who picked up the dish for settlement, including:

[0107] Generate an ID number based on the acquired identity information of the person picking up the meal, and dynamically bind the ID number to the coordinate frame of the person who is picking up the meal;

[0108] Track the human coordinate frame according to the ID number to obtain the time when the person who takes the meal completes taking the meal;

[0109] The weight change of the dish when the person picking up the food completes the food picking up, is used as the food picking up weight and is bound to the identity information of the person picking up the food for settlement.

[0110] In an embodiment of the present invention, a corresponding ID number is generated when the identity information of the person picking up the meal is obtained, and the ID number is continuously and dynamically bound to the coordinate frame of the human body where the meal picking behavior occurs, thereby realizing timely binding of the meal picking behavior and the identity information of the person picking up the meal, thereby being able to accurately track the location of the person picking up the meal and the entire meal picking process, and then obtain the accurate weight of the meal picked up by the person picking up the meal.

[0111] In one embodiment, the obtaining of the identity information of the person who takes the food based on the coordinate frame of the person who takes the food and the facial image includes:

[0112] Inputting the facial image into a face recognition model for processing, identifying the facial coordinate frame and obtaining the identity information of the person;

[0113] The coordinate frame of the human body that takes the meal is matched with the coordinate frame of the human face in the face image according to the coordinates to obtain the identity information of the person who takes the meal.

[0114] In an embodiment of the present invention, facial images are acquired through a facial recognition camera, and facial recognition is performed through a facial recognition model to obtain the identity information of all persons appearing in the food pickup area. Position matching is then performed based on the coordinates, thereby matching the food pickup behavior with the identity information.

[0115] During implementation, the face recognition model may adopt a neural network model, such as a DeepFace model or a FaceNet model.

[0116] Specifically, before position matching, aligning the face image with the human behavior image includes the following steps:

[0117] Positioning the target weighing platform on the center line of the facial image;

[0118] Obtaining the position coordinate information of the target weighing platform in the shooting picture of the behavior camera;

[0119] Defining an optimal recognition range of the target weighing platform according to the position coordinate information of the target weighing platform, and extracting an image in the optimal recognition range as the human behavior image;

[0120] The center line of the human behavior image is aligned with the center line of the human face image along the Y axis.

[0121] In the embodiment of the present invention, there is no need to strictly regulate the positions of the face camera and the behavior camera. The optimal recognition range is obtained by detecting the position coordinate information of the target weighing platform in the picture taken by the behavior camera, such as Figure 4As shown in the figure, the image in the optimal recognition range is extracted as the human behavior image, and then the center line of the human behavior image is aligned with the center line of the face image along the Y axis, that is, the coordinate alignment of the two images is completed, which can ensure the accurate matching of the human coordinate frame and the face coordinate frame, and accurately obtain the identity information of the person who picks up the meal.

[0122] Among them, when the optimal recognition range of the target weighing platform is delineated according to the position coordinate information of the target weighing platform, the target weighing platform should be located on the center line of the optimal recognition range. Specifically, in the two-dimensional coordinate system of the shooting screen of the behavior camera, the bottom edge of the shooting screen is the X axis, the height of the shooting screen is H, and the coordinates of the four corners of the target weighing platform are defined as (x1, y1), (x2, y1), (x1, y2), (x2, y2), wherein; the coordinates of the four corners of the corresponding optimal recognition range are (x1-Δx, 0), (x2+Δx, 0), (x1-Δx, H), (x2+Δx, H), wherein Δx>0, Δx is the deviation value, specifically, Δx is a preset value, or Δx is the distance between the target weighing platform and the adjacent weighing platform.

[0123] It should be noted that, in actual applications, the shooting range of the face camera is smaller, and the shooting range of the behavior camera is larger. Taking the above situation into consideration, in an embodiment of the present invention, the optimal recognition range is defined in the shooting picture of the behavior camera to extract the human behavior image.

[0124] In addition, if the facial camera and the behavioral camera cannot be automatically aligned, an alarm signal will be issued to prompt the staff to adjust the installation position or installation angle of the two cameras.

[0125] In an embodiment of the present invention, during installation, the facial camera is arranged above the target weighing platform so that it shoots the head of the person to obtain a facial image, and the shooting center of the facial camera and the center line of the target weighing platform are located on the same vertical plane.

[0126] During installation, the behavior camera is placed above the target weighing platform to capture the person's body, and the target weighing platform is included in the captured image of the behavior camera. The target weighing platform should be located in the center of the captured image to facilitate identification of meal-taking behavior and avoid missing key information.

[0127] Specifically, the method further includes: before use, calibrating the behavior camera to avoid image redundancy and missing problems, including the following steps:

[0128] Step A1, obtaining a human behavior image captured by the behavior camera at a current position;

[0129] Step A2, judging whether there is a problem of image redundancy or missing according to the area and position coordinates of the target weighing platform in the human behavior image;

[0130] If there is a problem of image redundancy or missing, adjust the shooting angle of the behavior camera along the width direction of the dining table, and then return to perform the operations of step A1 to step A2;

[0131] If there are no redundant or missing images, the calibration is complete.

[0132] Among them, the redundancy of the picture will lead to incomplete coordinate frames of the human body, and the head part of the person may be missed, such as Figure 5 As shown in ; missing images will result in failure to identify the meal pickup behavior, such as Figure 6 As shown in .

[0133] In an embodiment of the present invention, whether image redundancy and missing problems occur is automatically detected based on the area of ​​the target weighing platform in the human behavior image, and the shooting angle of the behavior camera is adjusted to avoid image redundancy and missing problems.

[0134] Specifically, judging whether there is image redundancy or missing based on the area and position coordinates of the target weighing platform in the human behavior image includes:

[0135] Convert the actual area of ​​the target weighing platform into an area threshold;

[0136] If the area of ​​the target weighing platform in the human behavior image is greater than zero and less than the area threshold, it is determined that there is a picture missing;

[0137] If the area of ​​the target weighing platform in the human behavior image is equal to the area threshold, and the distance between the coordinates of the preset point at the bottom edge of the target weighing platform and the coordinates of the bottom edge of the image is greater than a preset error value, it is determined that there is image redundancy;

[0138] If the area of ​​the target weighing platform in the human behavior image is equal to the area threshold, and the distance between the coordinates of the preset point at the bottom edge of the target weighing platform and the coordinates of the bottom edge of the picture is less than or equal to the preset error value, it is determined that there is no picture missing or redundancy.

[0139] It should be noted that, since the upper area threshold is converted from the actual area of ​​the target weighing platform, the area of ​​the target weighing platform in the human behavior image will not exceed the area threshold.

[0140] During implementation, the weighing platform is generally located at the lower side of the human behavior image. Therefore, if there is a missing image in the human behavior image, the shooting angle of the behavior camera is adjusted downward; if there is a redundant image in the human behavior image, the shooting angle of the behavior camera is adjusted upward.

[0141] It should be noted that if the weighing platform cannot be detected in the human behavior image, it means that the behavior camera has completely missed the key image and the position of the behavior camera needs to be manually readjusted.

[0142] Specifically, the method further includes: before use, calibrating the behavior camera to avoid the problem of table line tilt, including the following steps:

[0143] Step B1, obtaining a human behavior image captured by the behavior camera at a current position;

[0144] Step B2, detecting whether the dining table line in the human behavior image is parallel to the horizontal line of the shooting picture;

[0145] If not, it is determined that there is a problem with the table line being tilted, and the shooting angle of the behavior camera is adjusted counterclockwise or clockwise according to the angle between the table line and the horizontal line of the shooting screen, and then the operation of step B1 to step B2 is returned to be executed;

[0146] If so, it is determined that there is no problem of table line tilt and the calibration is completed.

[0147] like Figure 7 As shown in , if the table line is not parallel to the horizontal line of the shooting screen, the behavior camera needs to be adjusted. Specifically, if the angle between the table line and the horizontal line of the shooting screen is less than 90°, the shooting angle of the behavior camera needs to be adjusted counterclockwise; if the angle between the table line and the horizontal line of the shooting screen is greater than 90°, the shooting angle of the behavior camera needs to be adjusted clockwise.

[0148] The horizontal line of the shooting picture refers to a virtual line in the horizontal direction of the shooting picture. For example, in the two-dimensional coordinate system of the shooting picture, the horizontal line of the shooting picture is a straight line parallel to the X-axis.

[0149] It should be noted that, in the embodiment of the present invention, the dining table line and the outer edge line of the weighing platform are both straight lines, and when the weighing platform is placed on the dining table, the outer edge line of the weighing platform is parallel to the dining table line.

[0150] During implementation, when it is detected that the dining table line is parallel to the horizontal line of the shooting screen, the staff can be prompted by voice that the calibration of the tilt of the dining table line has been completed.

[0151] In addition, during implementation, the position coordinate information, area size, outer edge line of the scale body and dining table line of the target weighing platform can be obtained by using target detection models, such as the YOLOV model, Faster R-CNN model, etc.

[0152] It should be noted that in the intelligent meal picking and weighing method of the present invention, the weighing platform, face camera and behavior camera are hardware devices of the system. The steps of meal picking behavior recognition, identity recognition and binding of meal picking weight with the identity information of the person picking up the meal are implemented based on the software module of the system and can be installed in a computer system or an Internet platform.

[0153] The intelligent meal pickup and weighing method of the embodiment of the present invention is an intelligent solution that integrates biometric recognition software such as facial recognition and behavior recognition and intelligent equipment. It is an important component of smart restaurants. It improves the restaurant's operating efficiency and customers' dining experience through intelligent equipment and artificial intelligence algorithms.

[0154] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0155] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. An intelligent meal weighing method, characterized in that: The method comprises: The dishes are placed on a weighing platform on the dining table, and the weight change of the dishes is detected by the weighing platform; the distance between the edge line of the weighing platform and the edge line of the dining table is greater than or equal to a preset distance; Use a behavior camera to capture human behavior images, and use a face camera to capture human face images; Inputting the human behavior image into a key point recognition model for processing, identifying a human coordinate frame and key points, and obtaining position information of the key points; the key points include key points of human limb joints and tableware; Identifying a meal-picking behavior based on the position information of the human body coordinate frame and the key points and the weight change of the dish; If only one meal-picking behavior is identified, the identity information of the person picking up the meal is obtained based on the coordinate frame of the person who took the meal and the facial image, and then the weight change of the dish is used as the meal-picking weight and is bound to the identity information of the person picking up the meal for settlement; Identifying a meal-picking behavior according to the human body coordinate frame, the key point position information, and the weight change of the dish includes: Setting a meal-taking reference line on the processed human behavior image, wherein the meal-taking reference line is located between the edge line of the weighing platform and the edge line of the dining table; and the range defined between the meal-taking reference line and the bottom edge line of the human behavior image is used as a reference area; Determining whether a meal-taking action occurs within the human body coordinate frame according to the reference area; If the preset key point in the human body coordinate frame is located in the reference area, and the weight of the dish changes and lasts for a preset time, it is determined that a meal-taking behavior occurs in the human body coordinate frame.

2. The method according to claim 1, characterized in that The dishes include dishes that require tableware to be served and dishes that do not require tableware to be served; Wherein, when the dish is a dish that requires tableware to be taken, the hand joint points among the human limb joint points and the tableware key points are selected as the preset key points; When the dish is a dish that does not require tableware to be taken, the hand joints among the human limb joints are selected as the preset key points.

3. The method according to claim 1 or 2, characterized in that Before determining whether a meal-picking behavior occurs within the human coordinate frame based on the reference area, the method further includes: Setting a screening line on the human behavior image, wherein the screening line is located outside the dining table; Determining whether a human coordinate frame in the human behavior image may be in a meal-taking behavior based on the screening line; If the bottom of the human coordinate frame exceeds the screening line, it is determined that a meal-taking behavior may occur in the human coordinate frame, and then whether a meal-taking behavior occurs in the human coordinate frame is determined based on the reference area; If the bottom of the human body coordinate frame does not exceed the screening line, it is determined that the meal-taking behavior is impossible in the human body coordinate frame, and then whether the meal-taking behavior occurs in the human body coordinate frame is no longer determined based on the reference area.

4. The method according to any one of claims 1 to 2, characterized in that The method of binding the weight change of the dish as the meal weight and the identity information of the person who picks up the dish for settlement includes: Generate an ID number based on the acquired identity information of the person picking up the meal, and dynamically bind the ID number to the coordinate frame of the person who is picking up the meal; Track the human coordinate frame according to the ID number to obtain the time when the person who takes the meal completes taking the meal; The weight change of the dish when the person picking up the food completes the food picking up, is used as the food picking up weight and is bound to the identity information of the person picking up the food for settlement.

5. The method according to any one of claims 1 to 2, characterized in that The obtaining of the identity information of the person who takes the meal according to the coordinate frame of the person who takes the meal and the facial image includes: Inputting the facial image into a face recognition model for processing, identifying the facial coordinate frame and obtaining the identity information of the person; The coordinate frame of the human body that takes the meal is matched with the coordinate frame of the human face in the face image according to the coordinates to obtain the identity information of the person who takes the meal.

6. The method according to claim 5, characterized in that Before position matching, the face image and the human behavior image are aligned, including the following steps: Obtaining the position coordinate information of the target weighing platform in the shooting picture of the behavior camera; Defining an optimal recognition range of the target weighing platform according to the position coordinate information of the target weighing platform, and extracting an image in the optimal recognition range as the human behavior image; The center line of the human behavior image is aligned with the center line of the human face image along the Y axis.

7. The method according to any one of claims 1 to 2, characterized in that The method further includes: before use, calibrating the behavior camera to avoid image redundancy and missing problems, including the following steps: Step A1, obtaining a human behavior image captured by the behavior camera at a current position; Step A2, judging whether there is a problem of image redundancy or missing according to the area and coordinates of the weighing platform in the human behavior image; If there is a problem of image redundancy or missing, adjust the shooting angle of the behavior camera along the width direction of the dining table, and then return to perform the operations of step A1 to step A2; If there are no redundant or missing images, the calibration is complete.

8. The method according to any one of claims 1 to 2, characterized in that The method further includes: before use, calibrating the behavior camera to avoid the problem of table line tilt, including the following steps: Step B1, obtaining a human behavior image captured by the behavior camera at a current position; Step B2, detecting whether the dining table line in the human behavior image is parallel to the horizontal line of the shooting picture; If not, it is determined that there is a problem with the table line being tilted, and the shooting angle of the behavior camera is adjusted counterclockwise or clockwise according to the angle between the table line and the horizontal line of the shooting screen, and then the operation of step B1 to step B2 is returned to be executed; If so, it is determined that there is no problem of table line tilt and the calibration is completed.

9. The method according to any one of claims 1 to 2, characterized in that The tableware is provided with color features and / or shape features, and the positions of the color features and / or the shapes are used as key points of the tableware.

Images