Food material turn-over control method, related equipment and cooking robot

Abstract

The embodiment of the invention discloses a food material turn-over control method, device and equipment, a cooking robot and a computer readable storage medium. According to the application, the types and pixel coordinates of the food materials in the pot can be identified; in response to the fact that the food materials are solid food materials, world coordinates of the food materials in the pot are obtained through calculation according to a mechanical arm conversion matrix on the basis of the pixel coordinates; geometric information of the food materials is obtained, and turn-over point positions of the food materials are determined according to the geometric information and the distance between the inner edge of the pot; the ripening state of the food materials is monitored in real time; and when the cooked state reaches the degree needing to be turned over, the mechanical arm is driven to turn over the food materials in the pot according to the turning-over point position and the world coordinates. According to the embodiment of the invention, the mechanical arm can be driven to turn over the food materials when the food materials are in the ripening state and need to be turned over, and the situation that the whole food materials are in the non-uniform ripening state is avoided.

Description

Technical Field

[0001] Embodiments of the present application relate to a cooking robot, and more particularly to a method, device, equipment, cooking robot, and computer-readable storage medium for controlling food flipping. Background Art

[0002] With the rapid development of society, cooking robots have emerged. Traditional cooking robots only use a simple rigid structure stirring part to turn the food in the pot, and their intelligence level is very low.

[0003] A robotic arm is set in the pot of the cooking robot. The spatula as the end effector is connected to the robotic arm, and the position, speed, movement path, etc. of the spatula can be adjusted in all directions to achieve complete anthropomorphic operation, which plays a key role in improving the intelligence of the cooking robot. Among them, how to control the cooking robot to turn over the ingredients in the pot to ensure that the ingredients in the pot are evenly cooked is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present application provide a method, device, equipment, cooking robot and computer-readable storage medium for controlling food turning over to ensure that the food in the pot is cooked evenly.

[0005] In a first aspect, an embodiment of the present application provides a method for controlling food turning, comprising:

[0006] Identify the type and pixel coordinates of the food in the pot;

[0007] In response to the type of the food being solid food, the world coordinates of the food in the pot are calculated based on the pixel coordinates and according to the robot arm transformation matrix;

[0008] Acquire geometric information of the food, and determine the turning point of the food according to the geometric information and the distance from the edge of the pot;

[0009] Real-time monitoring of the ripening status of ingredients;

[0010] When the cooked state reaches the degree that requires turning over, the robotic arm is driven to turn over the food in the pot according to the turning point and the world coordinates.

[0011] In some possible implementations, determining the turning point of the food according to the geometric information and the distance from the edge of the pot includes:

[0012] Calculate the shortest distance between the edge of the food in each direction and the edge of the pot according to the geometric information;

[0013] The longest side of the shortest distance is used as the turning point.

[0014] In some possible implementations, calculating the shortest distance between the edge of the food in each direction and the edge of the pot according to the geometric position information includes:

[0015] Obtain the outline of ingredients in the pot through vision;

[0016] Based on the outline of the food, generate a circumscribed rectangle of the food;

[0017] Calculate the shortest distance from each side of the circumscribed rectangle to the inner edge of the pot.

[0018] In some possible implementations, the real-time monitoring of the cooking state of the food includes:

[0019] Real-time monitoring of the ripening status of food materials based on visual recognition; and / or,

[0020] Monitor the ripening status of ingredients in real time based on temperature.

[0021] In some possible implementations, the driving a mechanical arm in the pot to turn over the food in the pot according to the turning point and the world coordinates includes:

[0022] Determine the corresponding material information based on the type of the food, and calculate the optimal turning force according to the material information;

[0023] The food in the pot is turned over according to the optimal turning force and turning point.

[0024] In a second aspect, the present application also provides a food turning control device, comprising:

[0025] Identification module, used to identify the type and pixel coordinates of the food in the pot;

[0026] A coordinate calculation module, configured to calculate the world coordinates of the food in the pot based on the pixel coordinates and the robot arm transformation matrix in response to the type of the food being solid food;

[0027] A distance calculation module, used to obtain geometric information of the food, and determine the turning point of the food according to the distance between the geometric information and the edge of the pot;

[0028] Status monitoring module, used to monitor the ripening status of ingredients in real time;

[0029] The flipping module is used to drive the mechanical arm in the pot to flip the food in the pot according to the flipping point and the world coordinate when the cooked state reaches the degree that requires flipping.

[0030] In a third aspect, an embodiment of the present application further provides a food turning control device, comprising a memory and a processor, wherein the processor is used to execute instructions stored in the memory so that the control device executes the food turning control method as described in any one of the above items.

[0031] In a fourth aspect, an embodiment of the present application further provides a cooking robot, comprising a robotic arm disposed in a pot, and the food flipping control device as described above.

[0032] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor enables the processor to execute the food turning control method as described in any one of the above items.

[0033] The present application identifies the type and pixel coordinates of the food in the pot; in response to the type of the food being a solid food, calculates the world coordinates of the food in the pot based on the pixel coordinates and the robotic arm conversion matrix; obtains the geometric information of the food, and determines the turning point of the food according to the geometric information and the distance from the edge of the pot; monitors the cooked state of the food in real time; and when the cooked state reaches the point where flipping is required, drives the robotic arm to flip the food in the pot according to the flipping point and the world coordinates. This enables the cooking robot to drive the robotic arm to flip the food according to the cooked state of the food and the position of the food in the pot when the cooked state of the food reaches the point where flipping is required, thereby avoiding uneven cooked state of the food as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 It is a flow chart of a food turning control method provided in an embodiment of the present application;

[0036] Figure 2 yes Figure 1 A detailed flow chart of step S130 in an embodiment;

[0037] Figure 3 yes Figure 1 A detailed flow chart of step S150 in an embodiment;

[0038] Figure 4 It is a schematic block diagram of a food turning control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described in detail by way of implementation methods with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, but not all of the embodiments. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0040] It should be noted that: in the accompanying drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions; in the description of the present application, the terms "center", "longitudinal", "lateral", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the scope of protection of the present application; in the description of the present application, "first", "second", etc. are only used to distinguish each other, rather than to indicate their importance and order, etc.

[0041] In the description of this application, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a movable connection, or a detachable connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal communication of two components, etc. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0042] Reference Figure 1 This is a flow chart of a food turning control method provided in an embodiment of the present application, the method comprising steps S110 to S150.

[0043] Step S110, identifying the type and pixel coordinates of the food in the pot;

[0044] In this embodiment, the image of the pot is collected by a visual sensor such as a surveillance camera, and then the image of the pot is recognized by a trained visual recognition algorithm to determine the type and pixel coordinates of the food in the pot. The visual recognition algorithm can be a single-stage detection target detection algorithm such as the YOLO series and the SSD series; it can also be a two-stage detection deep learning algorithm such as Fast R-CNN, Faster R-CNN and Mask R-CNN.

[0045] Step S120: In response to the type of the food being solid food, the world coordinates of the food in the pot are calculated based on the pixel coordinates and the robot arm transformation matrix;

[0046] In this embodiment, based on the recognition result of the ingredients in the pot, in response to the type of ingredients in the pot being solid ingredients, the world coordinates of the ingredients in the pot are calculated according to the robot arm conversion matrix based on the pixel coordinates of the ingredients. In the application, the first conversion matrix of the world coordinate system to the robot arm coordinate system can be obtained, and then the world coordinates of the object are calculated based on the second conversion matrix of the object to the world coordinate system, combined with the first conversion matrix of the world coordinate system to the robot arm coordinate system, that is, the world coordinates of the ingredients are obtained. The second conversion matrix of the object to the world coordinate system can specifically refer to the conversion matrix of the coordinates of the object in the object coordinate system or the camera coordinate system to the world coordinate system, and the coordinates of the object in the object coordinate system or the camera coordinate system can be converted to coordinates in the world coordinate system according to the conversion matrix.

[0047] Step S130, obtaining geometric information of the food, and determining the turning point of the food according to the geometric information and the distance from the edge of the pot;

[0048] In this embodiment, in order to avoid destroying the shape of the ingredients when turning them over, it is necessary to determine the turning point of the ingredients. Therefore, based on the recognition result obtained based on the visual recognition algorithm, the geometric information of the ingredients is extracted from it, and then based on the geometric information of the ingredients and the distance from the edge of the pot, the turning point of the ingredients is calculated.

[0049] like Figure 2 As shown, in one embodiment, step S130 includes:

[0050] Step S210, calculating the shortest distance between the edge of the food in each direction and the edge of the pot according to the geometric information;

[0051] Step S220: taking the longest side of the shortest distance as the turning point.

[0052] In some possible implementations, the type of food is considered. For example, when frying an egg, a spatula is used to turn the egg over. The spatula must be placed against the bottom of the pan and inserted under the egg before turning it over. The success rate of turning the egg over is relatively high. In order for the spatula to be able to stick to the bottom of the pan, it is generally necessary to insert it under the egg from the side of the egg that is not close to the edge of the pan. If it is inserted under the egg from the side of the egg that is close to the edge of the pan, the spatula will be blocked by the inner wall of the pan, causing the spatula to be unable to stick to the bottom of the pan, thereby causing the egg to tilt along the spatula, causing the egg on one side close to the edge of the pan to overlap with the egg on the other side after the egg tilts, causing the egg to fail to turn over. Therefore, in order to facilitate turning the food over, the side of the food that is not close to the edge of the pan is preferentially selected as the turning point. Specifically, first, the shortest distance between the edge of the food in each direction and the edge of the pan is calculated based on the geometric information of the food, wherein the geometric information of the food is the outline of the food. Next, the circumscribed rectangle of the food is generated based on the outline of the food; finally, the shortest distance from each side of the circumscribed rectangle to the edge of the pot is calculated; the side with the longest shortest distance indicates that the food on that side is farthest from the edge of the pot, so the side with the longest shortest distance is used as the turning point of the food.

[0053] For example, through calculation, the lengths of the circumscribed rectangle of the food from the edge of the pot in the upper, lower, left and right directions are 4 cm, 6 cm, 9 cm and 3 cm respectively; the longest side is the left side, so the left side of the food is selected as the turning point.

[0054] Step S140, real-time monitoring of the cooking state of the food;

[0055] Step S150: When the cooked state reaches the degree that requires turning over, the robot arm is driven to turn over the food in the pot according to the turning point and the world coordinates.

[0056] In this embodiment, the ripeness state of the food is characterized by the ripeness state attribute of the food, which is composed of a characterization parameter and a characterization parameter value. The characterization parameter of the food is composed of one of temperature, moisture, color, and texture, or of smell, or of at least one of temperature, moisture, color, and texture and smell. For different food materials, the characterization parameter of the maturity of the food material can be selected to be characterized by all the characterization parameters such as temperature, moisture, color, texture, and smell.

[0057] Considering the convenience of actual operation, the required characterization parameters can be selected for characterization based on the characteristics of the ingredients and the cooking methods, without having to use all the characterization parameters. For example, for steamed eggs, you can select temperature and texture for characterization; for baked sweet potatoes, you can select temperature, moisture, and color as three characterization parameters. Baked cakes can be characterized by color, or by smell and color. Therefore, to monitor the maturity of ingredients, the color of the ingredients can be visually identified, and / or the temperature of the ingredients can be monitored. Finally, when the maturity reaches the point where it needs to be turned over, the robotic arm is driven to turn over the ingredients in the pot according to the turning point and the world coordinates.

[0058] like Figure 3 As shown, in one embodiment, step S150 includes:

[0059] Step S310: determining corresponding material information based on the type of the food, and calculating an optimal turning force according to the material information;

[0060] Step S320: turning over the food in the pot according to the optimal turning force and turning point.

[0061] In some possible implementations, in order to avoid destroying the shape of the food in the pot when turning it over, it is also necessary to pay attention to the turning force of the robot arm when turning it over. Specifically, the material information of the food is determined according to the type of food, and then the optimal turning force of the food is calculated based on the material information. Among them, the material information of the food includes the density, softness and hardness of the food. Finally, the food in the pot is turned over based on the optimal turning force and turning point.

[0062] For example, if the food in the pot is steak, the robot arm can use a higher force to flip the food over because it has a higher density and its shape is not easily destroyed when cut. If the food in the pot is eggs, a larger force can easily destroy the egg's shape when flipping the food over. Therefore, the robot arm can use a lower force to flip the food over.

[0063] The present application identifies the type and pixel coordinates of the food in the pot; in response to the type of the food being a solid food, calculates the world coordinates of the food in the pot based on the pixel coordinates and the robotic arm conversion matrix; obtains the geometric information of the food, and determines the turning point of the food according to the geometric information and the distance from the edge of the pot; monitors the cooked state of the food in real time; and when the cooked state reaches the point where flipping is required, drives the robotic arm to flip the food in the pot according to the flipping point and the world coordinates. This enables the cooking robot to drive the robotic arm to flip the food according to the cooked state of the food and the position of the food in the pot when the cooked state of the food reaches the point where flipping is required, thereby avoiding uneven cooked state of the food as a whole.

[0064] The embodiment of the present invention further provides a food turning control device, which is used to execute any embodiment of the above-mentioned food turning control method. Figure 4 , Figure 4 1 is a schematic block diagram of a food turning control device provided by an embodiment of the present invention. The food turning control device 100 can be configured in a server.

[0065] like Figure 4 As shown, the food turning control device 100 includes a recognition module 110 , a coordinate calculation module 120 , a distance calculation module 130 , a state monitoring module 140 , and a turning module 150 .

[0066] Identification module 110, used to identify the type and pixel coordinates of food in the pot;

[0067] A coordinate calculation module 120, for calculating the world coordinates of the food in the pot based on the pixel coordinates and the robot arm transformation matrix in response to the type of the food being solid food;

[0068] The distance calculation module 130 is used to obtain geometric information of the food and determine the turning point of the food according to the geometric information and the distance from the edge of the pot;

[0069] The state monitoring module 140 is used to monitor the ripening state of the food in real time;

[0070] The turning module 150 is used to drive the robot arm in the pot to turn the food in the pot according to the turning point and the world coordinate when the cooked state reaches the degree that requires turning over.

[0071] In some possible implementations, the distance calculation module 130 includes:

[0072] A distance calculation unit, used to calculate the shortest distance between the edge of the food in each direction and the edge of the pot according to the geometric information;

[0073] The screening unit is used to use the longest side of the shortest distance as the turning point.

[0074] In some possible implementations, the distance calculation unit includes:

[0075] A contour acquisition subunit, used to obtain the contour of the ingredients in the pot through vision;

[0076] A rectangle generation subunit, used to generate a circumscribed rectangle of the food based on the outline of the food;

[0077] The shortest distance calculation subunit is used to calculate the shortest distance from each side of the circumscribed rectangle to the inner edge of the pot.

[0078] In some possible implementations, the status monitoring module 140 includes:

[0079] A visual inspection unit, used to monitor the ripening status of ingredients in real time based on visual recognition;

[0080] The temperature monitoring unit is used to monitor the cooking status of ingredients in real time based on temperature.

[0081] In some possible implementations, the flip module 150 includes:

[0082] A force calculation unit, used to determine corresponding material information based on the type of the food, and calculate the optimal flipping force according to the material information;

[0083] The turning unit is used to turn over the food in the pot according to the optimal turning force and turning point.

[0084] An embodiment of the present application also provides a food turning control device, comprising a memory and a processor, wherein the processor is used to execute instructions stored in the memory so that the control device executes the food turning control method as described above.

[0085] An embodiment of the present application also provides a cooking robot, including a robotic arm disposed in a pot, and the food turning control device as described above.

[0086] In some possible implementations, the adopted mechanical arm is a five-degree-of-freedom mechanical arm. In addition, the degrees of freedom of the mechanical arm can also be adjusted according to actual needs.

[0087] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the food turning control method as described above.

[0088] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0089] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A food turning control method, applied to a cooking robot, characterized in that: include: Identify the type and pixel coordinates of the food in the pot; In response to the type of the food being solid food, the world coordinates of the food in the pot are calculated based on the pixel coordinates and according to the robot arm transformation matrix; Acquire geometric information of the food, and determine the turning point of the food according to the geometric information and the distance from the edge of the pot; Real-time monitoring of the ripening status of ingredients; When the cooked state reaches the degree that requires turning over, the robotic arm is driven to turn over the food in the pot according to the turning point and the world coordinates.

2. The food turning control method according to claim 1, characterized in that: The step of determining the turning point of the food according to the geometric information and the distance from the edge of the pot includes: Calculate the shortest distance between the edge of the food in each direction and the edge of the pot according to the geometric information; The longest side of the shortest distance is used as the turning point.

3. The food turning control method according to claim 2, characterized in that: The step of calculating the shortest distance between the edge of the food in each direction and the edge of the pot according to the geometric position information includes: Obtain the outline of ingredients in the pot through vision; Based on the outline of the food, generate a circumscribed rectangle of the food; Calculate the shortest distance from each side of the circumscribed rectangle to the inner edge of the pot.

4. The food turning control method according to claim 1, characterized in that: The real-time monitoring of the ripening state of the food includes: Real-time monitoring of the ripening status of food materials based on visual recognition; and / or, Real-time monitoring of the ripening status of ingredients based on temperature.

5. The food turning control method according to claim 1, characterized in that: The method of driving the mechanical arm in the pot to turn over the food in the pot according to the turning point and the world coordinates includes: Determine the corresponding material information based on the type of the food, and calculate the optimal turning force according to the material information; The food in the pot is turned over according to the optimal turning force and turning point.

6. A food turning control device, characterized in that: It includes a detection module and a processing module, the detection module is used to detect the type and pixel coordinates of the food in the pot, the geometric information of the food, and the cooking state of the food, and the processing module is used to execute the food turning control method as described in any one of claims 1-5.

7. A food turning control device, characterized in that: It includes a memory and a processor, and the processor is used to execute instructions stored in the memory so that the control device executes the food turning control method as described in any one of claims 1-5.

8. A cooking robot, characterized in that: It comprises a mechanical arm arranged in the pot, and the food turning control device as claimed in claim 7.

9. The cooking robot according to claim 8, characterized in that: The robotic arm is a five-degree-of-freedom robotic arm.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the food turning control method according to any one of claims 1 to 5.

Images