Cooking method, cooking equipment, electronic equipment, storage medium and program product

Abstract

The invention provides a cooking method, cooking equipment, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring an image obtained by shooting a cooking object at at least one shooting angle; according to the image, determining the food type of the cooking object through a deep learning model; performing three-dimensional reconstruction on the cooking object according to the image to obtain a three-dimensional reconstruction effect picture, and determining size information of the cooking object based on the three-dimensional reconstruction effect picture; according to the size information of the cooking object and the food type, cooking data corresponding to the cooking object are determined, and the cooking data comprise cooking temperature and / or cooking time; and cooking the cooking object according to the cooking data. Optimized cooking of the cooking object can be achieved, and the cooking efficiency and quality are improved.

Description

Technical Field

[0001] The present disclosure relates to smart home technology, and in particular, to a cooking method, a cooking device, an electronic device, a storage medium, and a program product. Background Art

[0002] With the continuous development of technology, kitchen appliances are also constantly upgraded. Among them, for example, devices such as microwave ovens and ovens have become essential household appliances in modern kitchens.

[0003] Currently, when cooking food with cooking devices such as household ovens and grills, the cooking temperature and time set by the user are directly adopted (for simple cooking devices, there may be no cooking mode selection), or the cooking mode recommended by the cooking device is used to control the operation of the cooking device to cook the food.

[0004] However, this setting method for cooking devices is not intelligent enough, and it requires the user to rely on personal experience to set it time-consuming and laboriously, and there may be problems such as the cooked food being burnt or the cooking time being insufficient during the operation of the cooking device. Summary of the Invention

[0005] The present disclosure provides a cooking method, a cooking device, an electronic device, a storage medium, and a program product, which are used to solve the problem that the setting method for cooking devices is not intelligent enough and it is easy to occur that the cooked food is burnt or the cooking time is insufficient during the operation of the cooking device, and to realize more intelligent automatic determination of cooking data corresponding to the cooking object, so as to achieve the technical effect of improving the cooking effect of food and the user experience.

[0006] On the one hand, the present disclosure provides a cooking method, and the method includes:

[0007] Obtain images of a cooking object taken at at least one shooting angle;

[0008] According to the images, determine the food type of the cooking object through a deep learning model;

[0009] Perform three-dimensional reconstruction on the cooking object according to the images to obtain a three-dimensional reconstruction effect diagram, and determine the size information of the cooking object based on the three-dimensional reconstruction effect diagram;

[0010] Determine cooking data corresponding to the cooking object according to the size information and the food type of the cooking object, where the cooking data includes: cooking temperature and / or cooking time;

[0011] Cook the cooking object according to the cooking data.

[0012] On the other hand, the present disclosure provides another cooking method, and the method includes:

[0013] Obtain an image of a cooking object captured at at least one shooting angle;

[0014] Obtain the food type of the cooking object;

[0015] Perform three-dimensional reconstruction on the cooking object based on the image to obtain a three-dimensional reconstruction effect diagram, and determine the size information of the cooking object based on the three-dimensional reconstruction effect diagram;

[0016] Determine cooking data corresponding to the cooking object according to the size information of the cooking object and the food type, where the cooking data includes: cooking temperature and / or cooking time;

[0017] Perform cooking processing on the cooking object according to the cooking data.

[0018] An optional implementation manner, the image includes at least one of the following:

[0019] The image obtained by using a shooting device to shoot the cooking object at at least one of the shooting angles before the cooking object is placed in a cooking device;

[0020] In response to the cooking object being placed in the cooking device, the image obtained by using a shooting device arranged in the cooking device to shoot the cooking object at at least one of the shooting angles.

[0021] An optional implementation manner, the determining the size information of the cooking object based on the three-dimensional reconstruction effect diagram includes:

[0022] Obtain the characteristic size of a reference object of the cooking object;

[0023] Determine the size information of the cooking object according to the three-dimensional reconstruction effect diagram of the cooking object and the characteristic size of the reference object.

[0024] An optional implementation manner, performing three-dimensional reconstruction on the cooking object based on the image to obtain a three-dimensional reconstruction effect diagram includes:

[0025] Perform sparse three-dimensional reconstruction based on multiple pictures or multiple frames of video to obtain camera poses;

[0026] Perform dense three-dimensional reconstruction based on multiple pictures or multiple frames of video and the camera poses to obtain the three-dimensional reconstruction effect diagram of the cooking object.

[0027] An optional implementation manner, the performing sparse three-dimensional reconstruction based on multiple pictures or multiple frames of video to obtain camera poses includes:

[0028] Extract feature points from multiple pictures or multiple frames of video;

[0029] Perform feature matching on the feature points to obtain the matched feature points;

[0030] Decompose the features through the matched feature points to obtain the camera pose.

[0031] An optional implementation manner, the dense three-dimensional reconstruction based on multiple pictures or multiple frames of video and the camera pose to obtain the three-dimensional reconstruction effect diagram of the cooking object, includes:

[0032] Extract feature points from multiple pictures or multiple frames of video;

[0033] Perform feature matching on the feature points to obtain the matched feature points;

[0034] Based on the matched feature points, calculate the disparity of each pixel point;

[0035] Based on the disparity and the camera pose, calculate the three-dimensional coordinates of each pixel point;

[0036] Determine the dense point cloud data based on the three-dimensional coordinates of each pixel point;

[0037] Generate the three-dimensional reconstruction effect diagram based on the dense point cloud data.

[0038] On the other hand, the present disclosure provides a cooking device, the device includes:

[0039] A photographing device for photographing an image of a cooking object at at least one photographing angle;

[0040] A processor for determining the food type of the cooking object through a deep learning model according to the image; performing three-dimensional reconstruction on the cooking object according to the image to obtain a three-dimensional reconstruction effect diagram, and determining the size information of the cooking object based on the three-dimensional reconstruction effect diagram; determining cooking data corresponding to the cooking object according to the size information of the cooking object and the food type, where the cooking data includes: cooking temperature and / or cooking time;

[0041] A cooking device for performing cooking processing on the cooking object according to the cooking data.

[0042] On the other hand, the present disclosure provides another cooking device, the cooking device includes:

[0043] A photographing device for photographing an image of a cooking object at at least one photographing angle;

[0044] A processor, configured to obtain the food type of a cooking object; and configured to perform three-dimensional reconstruction on the cooking object based on the image to obtain a three-dimensional reconstruction effect diagram, and determine the size information of the cooking object based on the three-dimensional reconstruction effect diagram; determine cooking data corresponding to the cooking object according to the size information of the cooking object and the food type, where the cooking data includes: cooking temperature and / or cooking time;

[0045] A cooking device, configured to perform cooking processing on the cooking object according to the cooking data. On the other hand, the present disclosure provides an electronic device, including:

[0046] A photographing device, a processor, and a memory connected to the processor; the photographing device is configured to obtain images of a cooking object taken at at least one photographing angle;

[0047] The above-mentioned memory stores computer execution instructions; the above-mentioned processor executes the computer execution instructions stored in the above-mentioned memory to implement the method as described in any one of the above.

[0048] On the other hand, the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored, and when the computer execution instructions are executed by a processor, they are used to implement the method as described in any one of the above.

[0049] On the other hand, the present disclosure provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the method as described in any one of the above.

[0050] The present disclosure provides a cooking method, a cooking device, an electronic device, a storage medium, and a program product. The cooking method can determine the food type of a cooking object through a deep learning model according to an image obtained by obtaining images of the cooking object taken at at least one photographing angle; perform three-dimensional reconstruction on the cooking object based on the image to obtain a three-dimensional reconstruction effect diagram, and determine the size information of the cooking object based on the three-dimensional reconstruction effect diagram; furthermore, determine cooking data corresponding to the cooking object according to the size information of the cooking object and the food type, where the cooking data includes: cooking temperature and / or cooking time, so that cooking processing can be performed on the cooking object according to the cooking data, realizing intelligent cooking of the cooking object, thereby improving the cooking effect of food and the user experience. It can solve the problems that the setting method of the cooking device is not intelligent enough and the cooked food is prone to being burnt or the cooking time is insufficient during the operation of the cooking device. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings here are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0052] Figure 1 It is a schematic flowchart of a cooking method provided by an embodiment of the present disclosure;

[0053] Figure 2 It is a schematic flowchart of an alternative cooking method provided by an embodiment of the present disclosure;

[0054] Figure 3 It is a schematic flowchart of an alternative cooking method provided by an embodiment of the present disclosure;

[0055] Figure 4 It is a schematic flowchart of an alternative cooking method provided by an embodiment of the present disclosure;

[0056] Figure 5 It is a schematic flowchart of an alternative cooking method provided by an embodiment of the present disclosure;

[0057] Figure 6 It is a schematic flowchart of another cooking method provided by an embodiment of the present disclosure;

[0058] Figure 7 It is a structural block diagram of a cooking device provided by an embodiment of the present disclosure;

[0059] Figure 8 It is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

[0060] Through the above-mentioned drawings, specific embodiments of the present disclosure have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present disclosure in any way, but to illustrate the concept of the present disclosure to those skilled in the art by referring to specific embodiments. Detailed Embodiments

[0061] Here, exemplary embodiments will be described in detail, and their examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0062] With the continuous development of technology, kitchen appliances are also constantly upgraded. Among them, cooking devices such as ovens and grills have become essential household appliances in modern kitchens.

[0063] Currently, when cooking food with cooking devices such as household ovens and grills, the cooking data set by the user is directly adopted, such as cooking temperature and / or cooking time (for simple cooking devices, there may be no cooking mode selection, only simple cooking temperature and / or cooking time), or the cooking mode recommended by the cooking device, to control the operation of the cooking device to cook food.

[0064] However, this setting method for cooking devices is not intelligent enough. It requires the user to rely on personal experience to set it time-consuming and laboriously, and there may be problems such as the cooked food being burnt or the cooking time being insufficient during the operation of the cooking device.

[0065] The cooking method, cooking device, electronic device, storage medium and program product provided by the present disclosure aim to solve the above technical problems in the prior art. It can automatically identify the food type of the cooking object and calculate the size information of the cooking object. Before or during the cooking of the cooking object, it can determine the cooking target temperature (simply referred to as cooking temperature in the present disclosure) and / or cooking time (i.e., the time required for the cooking object to reach the above cooking target temperature) and other cooking data corresponding to the cooking object according to the food type and size information of the cooking object, so as to guide the user to set it by himself or directly control the cooking device to set appropriate cooking data, such as cooking temperature and / or cooking time, and improve the user experience of using the cooking device.

[0066] The technical solution of the present disclosure and how the technical solution of the present disclosure solves the above technical problems will be described in detail below with specific embodiments. These specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0067] Figure 1 is a schematic flow chart of a cooking method provided by an embodiment of the present disclosure, as Figure 1 shown, the method includes:

[0068] S110, obtaining images of the cooking object taken at at least one shooting angle.

[0069] S120, determining the food type of the cooking object according to the above images through a deep learning model.

[0070] S130, performing three-dimensional reconstruction on the cooking object according to the above images to obtain a three-dimensional reconstruction effect diagram, and determining the size information of the cooking object based on the three-dimensional reconstruction effect diagram.

[0071] Among them, the execution of the above steps S120 and S130 can be in any order. For example, in some embodiments, step S120 may be executed first to input the preprocessed image into the deep learning model, or directly input the original image with high quality that does not require preprocessing into the deep learning model to obtain the food type of the cooking object, and then step S130 may be executed to obtain the size information of the cooking object.

[0072] In some embodiments, step S130 may also be executed first and then step S120. In other embodiments, steps S120 and S130 may also be executed simultaneously. While inputting the original image or the preprocessed image into the deep learning model, a three-dimensional reconstruction effect diagram is also obtained by performing three-dimensional reconstruction on these original images or preprocessed images, and the size information of the cooking object is determined.

[0073] S140. Determine the cooking data corresponding to the cooking object according to the size information and food type of the cooking object, where the cooking data includes: cooking temperature and / or cooking time.

[0074] S150. Perform cooking processing on the cooking object according to the cooking data.

[0075] Optionally, the above cooking device may include: an oven, a grill, a microwave oven, a rice cooker, a pressure cooker, an automatic stir-fry machine, and so on.

[0076] Optionally, the cooking method provided by the embodiments of the present disclosure may run in a cooking device provided with a photographing device or on a smart device provided with a photographing device.

[0077] Optionally, for the cooking method provided by the embodiments of the present disclosure, it may also run in any existing cooking device. In this case, before the cooking object is placed in the cooking device, or when the cooking object is placed in the cooking device but the cooking device has not been turned on, the photographing device is used to photograph the cooking object at at least one photographing angle to obtain the above image.

[0078] It is easy to understand that deep learning is a machine learning method based on artificial neural networks. The deep learning model can automatically extract features in the image, realize automatic recognition and classification of objects, thereby improving the accuracy and robustness of object recognition.

[0079] Optionally, in the examples of the present disclosure, the deep learning model includes: a convolutional neural network (CNN), a recurrent neural network (RNN), and a model specifically designed for object recognition tasks such as YOLO, etc.

[0080] Since the convolutional layer of a deep learning model can effectively extract local features from an image, and the recurrent layer can capture dependencies between sequences, based on this property, in the embodiments of the present disclosure, the food type of a cooking object can be determined by a deep learning model.

[0081] To obtain a deep learning model that can more accurately identify the food type of a cooking object, sometimes it is necessary to select a suitable deep learning model or train some selected deep learning models. Optional deep learning models can be open-source codes with parameters trained with big data and publicly available on a network platform. For example, the Structure-from-Motion (SFM) algorithm. For such well-established deep learning models, there is no need to train them anymore. As long as an image is input to them, the food type of the cooking object included in the image can be directly obtained. However, the open-source codes of such large deep learning models require a large amount of computing power and consume a long time when running. And the method involved in the present disclosure only needs to identify images of items in a specific classification of food, only using a very small part of the algorithm of this large deep learning model. Therefore, some deep learning models with a relatively small number of parameters can also be selected, such as YOLO3, YOLO4, YOLO5, YOLO8, YOLOX, etc., and then trained with images containing various cooking objects to obtain a deep learning model that can specifically identify the food type of a cooking object. The deep learning model automatically extracts features from the preprocessed image; encodes the extracted features and converts them into a form that the deep learning model can use. Use the encoded features to optimize the parameters of the deep learning model through the backpropagation algorithm to improve the accuracy and generalization ability of the deep learning model; obtain a trained deep learning model. Use the trained deep learning model for prediction, evaluate the performance and generalization ability of the deep learning model, and adjust the parameters to optimize the deep learning model.

[0082] Specifically, an image is input into a deep learning model. Through the forward propagation process of deep learning, the predicted value of each pixel point is obtained. The loss function is calculated based on the true label and the predicted label to evaluate the prediction error of the model. According to the gradient information of the loss function, the model parameters are updated through the backpropagation algorithm, making the prediction result of the model closer to the true label. The processes of forward propagation, loss function calculation, and backpropagation are repeatedly executed until the model converges or reaches a preset number of training times, obtaining an optimized deep learning model. Among them, the loss function, also known as the cost function, is a function used to measure the gap between the model prediction result and the true result. Forward propagation means that the input data is passed into the neural network model, and after a series of linear operations and non-linear operations, the output result is finally obtained. The backpropagation algorithm is based on the chain rule of derivatives, and by continuously adjusting the parameters, the loss function is minimized, thereby continuously improving the performance and generalization ability of the model. The true label refers to the true category label of each sample in the dataset, which is known data that is manually labeled. For each input sample, the label is a predefined category label, such as "cat", "dog", "person", etc.; the predicted label refers to the prediction result output according to the input sample features and the training process, which is the predicted value of the sample category. After the above training of the deep learning model, a perfect deep learning model that can accurately identify (also known as predict in the industry) food types is finally obtained.

[0083] An optional implementation manner is as follows. First, step S110 is executed to obtain an image of a cooking object captured at at least one shooting angle. Optionally, the image may include multiple pictures or at least one video. Since a video can be regarded as composed of several frames of pictures arranged in chronological order, a video is equivalent to multiple pictures. In practice, for a video, it is also decomposed into several frames of pictures at each moment for use. The image can be obtained through at least one of the following methods:

[0084] Before the cooking object is placed in the cooking device, an image of the cooking object is captured at at least one shooting angle using a shooting device (such as a camera built in the cooking device or a separate camera, smartphone, video camera, etc.).

[0085] In response to the cooking object being placed in the cooking device, an image of the cooking object is captured at at least one shooting angle using a shooting device arranged inside the cooking device.

[0086] In the present disclosure, the shooting device is also called a camera. The spatial coordinate position of the shooting device and the orientation of its lens are also called the camera pose, where the pose includes the spatial coordinate position and the attitude (i.e., the orientation).

[0087] For example, if a user wants to cook a steak using a cooking device (i.e., the steak is the cooking object), then before putting the steak into the oven, the user can use the shooting device of a smart device such as a mobile phone or a tablet to take images of the steak from multiple different angles (such as multiple photos / pictures or at least one video), and then transmit these photos / pictures to the cooking device or the smart device (such as a mobile phone or a tablet with a built-in camera, or a computer or a server connected to the shooting device by wire or wirelessly, etc.). The cooking device or the smart device can, based on these photos or videos and the above-mentioned perfect deep learning model (such as a trained deep learning model), identify that the cooking object is a steak.

[0088] If a shooting device is provided inside the cooking device, the position of the shooting device is fixed relative to the inner cavity of the cooking device. In response to the cooking object being placed in the cooking device, the shooting device provided in the cooking device can be used to take an image of the cooking object at at least one shooting angle (such as in a rotatable microwave oven or oven, the cooking object rotates with the rotation mechanism inside the cooking device, and at this time, the shooting device can take images of the cooking object from multiple angles).

[0089] It should be noted that the advantage of providing a shooting device inside the cooking device is that not only can the image information of the real-time state of the cooking object be obtained at any time during the operation of the cooking device, but also the position of the shooting device relative to the cooking object inside the cooking device is relatively fixed, and subsequently, it is not necessary to refer to the characteristic size of the reference object as a reference for calculating the size information of the cooking object. This is because for a shooting device fixedly provided at a specific position inside the cooking device, the pixel size on each picture taken each time has a generally fixed corresponding relationship with the size of the actual space being photographed inside the cooking device. After calculating this corresponding relationship, the size information of the cooking object can be directly obtained by combining this "picture - actual space corresponding relationship" with the 3D reconstruction effect diagram.

[0090] However, it should be noted that during the operation of the cooking device, the high temperature of up to several hundred degrees inside it will damage an ordinary camera; therefore, if the camera is to be prevented from being damaged in a high-temperature environment, a special glass that is both heat-resistant and has high light transmittance can be added to protect it.

[0091] The embodiments of the present disclosure do not limit the manner of obtaining the image of the cooking object.

[0092] It should be noted that although the images in the above embodiments include multiple pictures or at least one video, it is also possible to take only one picture, because only one picture is needed to determine the food type of the cooking object through the deep learning model.

[0093] Then, optionally, the images obtained in the above steps, such as one or more pictures, can be preprocessed to obtain preprocessed pictures, and then a perfected deep learning model can be used to determine the food type of the cooking object. In the example of the present disclosure, the preprocessing of the image can remove the interference information of the objects other than the cooking object in the image, improve the quality and consistency of the image, so as to ensure that the three-dimensional reconstruction effect diagram obtained by subsequent three-dimensional reconstruction only contains the three-dimensional model of the cooking object, and avoid the calculation error caused by the interference information.

[0094] Optionally, the above preprocessing can also include operations such as image enhancement, image correction, and image registration, which can improve the contrast, brightness, clarity, color, etc. of the image, eliminate the noise, distortion, aberration, etc. of the image, and also align the size, direction, position, etc. of the images with each other.

[0095] For example, if a user wants to cook a salad, the user can use a smart device such as a mobile phone or a tablet to take pictures of the salad at different angles before putting the salad into the cooking device. Then the smart device can directly preprocess these pictures, or the smart device can also transmit these pictures to the cooking device for the cooking device to preprocess these pictures, such as adjusting the brightness, cropping the edges, scaling, rotating the angle, etc., so that these pictures are clearer, more consistent, and easier to recognize.

[0096] Further, step S130 is executed to perform three-dimensional reconstruction on the cooking object according to the image to obtain a three-dimensional reconstruction effect diagram, and determine the size information of the cooking object based on the three-dimensional reconstruction effect diagram. Three-dimensional reconstruction is a technology based on computer vision, which can use multiple two-dimensional images or at least one video to reconstruct the three-dimensional model of the object in the three-dimensional space through methods such as feature matching, geometric transformation, and depth estimation. Perform three-dimensional reconstruction on the cooking object according to the image to obtain the three-dimensional reconstruction effect diagram of the cooking object. The three-dimensional reconstruction effect diagram can be rotated and scaled to observe the three-dimensional shape of the cooking object from different perspectives, so as to obtain a more accurate size ratio of the cooking object, such as the width-to-height ratio or length-to-height ratio of the cooking object. For example, if a user wants to cook a sausage, the user can transmit the pictures of the sausage taken at different angles to the cooking device or the smart device. The cooking device or the smart device can reconstruct the three-dimensional space model of the sausage according to these two-dimensional pictures through a three-dimensional reconstruction algorithm, so as to obtain the three-dimensional reconstruction effect diagram of the sausage, and obtain the ratio relationship between the length / width and height (for the sausage, it is its diameter or thickness); then based on the three-dimensional reconstruction effect diagram of the sausage, calculate the size information of this sausage, such as length, width, height (i.e., thickness).

[0097] It should be noted that in the 3D reconstruction step, it is also possible to perform 3D reconstruction using only one image to obtain a 3D reconstruction effect diagram. For example, the method introduced in "Zero-1-to-3: Zero-shot One Image to 3D Object" can be referred to (Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, Carl Vondrick; "Zero-1-to-3: Zero-shot One Image to 3D Object"; Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), 2023, pp. 9298-9309).

[0098] Finally, perform step S140 and step S150 to determine the cooking data corresponding to the cooking object according to the size information and food type of the cooking object, where the cooking data includes: cooking temperature and / or cooking time; and perform specific cooking processing on the cooking object according to the determined cooking data to achieve a better cooking effect.

[0099] For example, if the cooking object is a steak, then according to the photo (i.e., image) of the steak, the deep learning model will automatically identify that the food type of the cooking object is steak; similarly, according to the photos of the steak at at least one shooting angle, perform 3D reconstruction on the steak to obtain a 3D reconstruction effect diagram of the steak. According to this 3D reconstruction effect diagram, the thickness of the steak is obtained as 3.3 cm. Based on the data table in the cooking database (as shown in Table 1), it is determined that the cooking temperature of the steak should be 180 degrees Celsius, and / or the cooking time should be 25 - 30 minutes. Then, inform the user of this cooking temperature and / or cooking time for the user to set it on the oven by themselves, or automatically set the oven through the control system of the oven, so that the oven can bake the steak according to this cooking temperature and / or cooking time to make the steak reach the ideal doneness and taste.

[0100] In the example of the present disclosure, through the image obtained by photographing the cooking object, it is possible to identify the food type of the cooking object, and perform 3D reconstruction on the cooking object according to this image to obtain the size information of the cooking object. Furthermore, according to the size information and food type of the cooking object, it is possible to relatively accurately automatically predict the cooking temperature and / or cooking time of the cooking object, improving the user experience; even users who have no experience in using cooking equipment such as ovens can easily handle various cooking tasks.

[0101] In an optional embodiment, part of the data of an optional cooking database is shown in Table 1 below:

[0102] Table 1

[0103]

[0104] An optional implementation manner, for example Figure 2 As shown, in the above step S130, determining the size information of the cooking object based on the above three-dimensional reconstruction effect diagram includes:

[0105] S210, obtaining the characteristic size of the reference object of the cooking object;

[0106] S220, determining the size information of the cooking object according to the three-dimensional reconstruction effect diagram of the cooking object and the characteristic size of the reference object.

[0107] For example, in one example, the characteristic size of the reference object of the cooking object can be input, and then the size information of the cooking object can be determined according to the characteristic size of the reference object and the three-dimensional reconstruction effect diagram of the cooking object. The characteristic size of the reference object can be regarded as a scale for measuring the cooking object, so the size information of the cooking object can be determined according to the characteristic size of the reference object. Optionally, the reference object can be an object with a known size photographed together with the cooking object, such as a ruler, a baking tray, a grill, a plate, a bowl, a spoon, chopsticks, etc. In the following embodiments, the cooking device is taken as an oven as an example: its reference object can be, for example, a grill, a baking tray, etc. commonly used in the oven to carry the cooking object; the characteristic size of the reference object can be, for example, the spacing between the iron bars of the grill, the size of the baking tray, etc.

[0108] It should be noted that, in the present disclosure, a grill can refer to a planar frame structure composed of a rectangular metal frame and a plurality of iron bars fixed (such as welded) inside the rectangular metal frame and arranged parallel to each other, which can carry the cooking object and be placed on the plane of the oven. Among them, the spacing between two adjacent iron bars on the grill is simply referred to as the iron bar spacing in the present disclosure; for the grill in the present disclosure, the spacing between every two adjacent iron bars, that is, the iron bar spacing, is equal, which is the characteristic size of the grill. In the present disclosure, a baking tray can refer to a plate that can carry the cooking object and be placed in the oven, and its bottom can be a whole piece of metal or a porous or hollow metal. For a baking tray, the size of the baking tray, that is, the characteristic size of the baking tray, can be the length and / or width of the baking tray.

[0109] It should be noted that the characteristic size of the reference object is known, for example, input by the user into the cooking device or the intelligent device, or pre-stored or set in the storage unit inside the cooking device.

[0110] When a reference object is determined (such as a specific reference object input by a user into a cooking device or a smart device, or a specific reference object automatically identified by a deep learning model as carrying a cooking object), the known characteristic dimensions of the above-mentioned reference object are associated with the reference object, and then the specific dimensions of the length / width of the cooking object are calculated through the dimensional relationship between the characteristic dimensions of the reference object and the cooking object (such as the proportional relationship between the length / width of the cooking object and the spacing between the iron bars of a grill); then, according to the proportional relationship between the length / width and the height (thickness) of the cooking object in the three-dimensional reconstruction effect diagram, the specific dimension of the height of the cooking object is converted.

[0111] In a feasible embodiment, it is also possible to only input a characteristic dimension into the cooking device without caring which reference object the characteristic dimension belongs to. For the sake of unified description, the characteristic dimension in this case is still referred to as the characteristic dimension of the reference object in this disclosure.

[0112] For example, if a user wants to cook a steak, when the processor of the oven runs the cooking method provided by this disclosure, first, based on the photos of the steak taken from different angles and the previous food type recognition step, the cooking object is determined to be a steak through a deep learning model. Then, the steak is three-dimensionally reconstructed based on the above-mentioned photos of the steak taken from different angles to obtain a three-dimensional reconstruction effect diagram of the steak, which includes at least a part of the carrier carrying the steak. Based on the three-dimensional reconstruction effect diagram of the steak, the ratio of the length to the thickness of the steak is obtained as 5:1. During this period, the user inputs into the processor of the oven through the input device of the oven that the reference object, that is, the carrier of the cooking object, is a grill, and inputs the characteristic dimension of the reference object, such as the spacing between every two iron bars on the grill, for example, the iron bar spacing is 3 cm; or directly inputs the characteristic dimension of the reference object, that is, the iron bar spacing of 3 cm, without caring what the specific carrier of the cooking object is. Then, through the three-dimensional reconstruction effect diagram of the steak (including a part of the grill and including at least one grill spacing), the ratio relationship between the dimension of the steak in the grill plane direction (such as the length or width of the steak) and the iron bar spacing of the grill is obtained. For example, the steak exactly spans 5 iron bars in the length direction, that is, it spans 4 iron bar spacings, so the ratio relationship between the length of the steak and the iron bar spacing is obtained as 4:1; then, based on the iron bar spacing of 3 cm, the dimension information of the steak is calculated, such as calculating that the length of the steak is 12 cm; then, according to the ratio of the length to the thickness of the steak obtained from the above-mentioned three-dimensional reconstruction effect diagram of the steak, which is 5:1, the thickness of the steak is calculated to be 2.4 cm.

[0113] For another example, if a user wants to cook a sausage, when taking a photo of the sausage, the baking tray with known characteristic dimensions that holds the sausage can be used as a reference object. Then, these photos are transmitted to the cooking device or the intelligent device. The cooking device or the intelligent device obtains a 3D reconstruction effect diagram of the sausage through 3D reconstruction of the sausage, and based on the 3D reconstruction effect diagram of the sausage, obtains the ratio of the length to the thickness of the sausage as 7:1; and based on the known characteristic dimensions of the reference object, such as the length / width of the baking tray (if the baking tray is rectangular) or the diameter (if the baking tray is circular), for example, taking the length of the rectangular baking tray as 28 cm as the characteristic dimension, and based on the above-mentioned photo of the sausage, obtains the dimensional relationship between the length of the sausage and the length of the baking tray, such as the sausage exactly occupies 1 / 2 of the length of the baking tray in its length direction, that is, the ratio of the length of the sausage to the length of the baking tray is 1:2, thereby calculating the dimensional information of the sausage, such as calculating the length of the sausage as 14 cm; then based on the ratio of the length to the thickness (i.e., radius) of the sausage obtained from the above-mentioned 3D reconstruction effect diagram of the sausage as 7:1, the thickness of the sausage is obtained as 2 cm.

[0114] In the present disclosure, as Figure 3 shown, in an optional embodiment, 3D reconstruction of the above cooking object based on the above image includes:

[0115] S310, performing sparse 3D reconstruction based on multiple pictures or multiple frames of video to obtain the camera pose;

[0116] S320, performing dense 3D reconstruction based on multiple pictures or multiple frames of video and the above camera pose to obtain the 3D reconstruction effect diagram of the above cooking object.

[0117] In an optional embodiment, as Figure 4 shown, the sparse 3D reconstruction method provided in the above step S310, that is, performing sparse 3D reconstruction based on multiple pictures or multiple frames of video to obtain the camera pose, can be implemented by the following method steps:

[0118] Step S410, extracting feature points from multiple pictures or multiple frames of video;

[0119] Step S420, performing feature matching on the above feature points to obtain the matched feature points;

[0120] Step S430, performing feature decomposition through the above matched feature points to obtain the camera pose.

[0121] The camera pose refers to the position and orientation of the camera in 3D space, and can usually be represented by the internal parameters and external parameters of the camera.

[0122] The above method steps provide a sparse 3D reconstruction method using feature point extraction and feature matching. Specifically, an algorithm such as Structure from Motion (SFM) can be used to recover the sparse 3D structure in the case of unknown camera poses. By extracting feature points from multiple pictures or multiple frames of video and performing feature matching on multiple feature points, the matching feature points are obtained. The Structure from Motion (SFM) algorithm means that motion can recover structure. It is an offline algorithm for 3D reconstruction based on various collected sequence pictures and can recover the structural information of the scene or object from the picture sequence. The Structure from Motion technology aims to recover the structural information of the scene and the poses of the shooting devices from multiple images with different viewpoints and is one of the key technologies in computer 3D vision. Of course, other algorithms can also be used for sparse 3D reconstruction. The present disclosure does not limit the algorithms for sparse 3D reconstruction. Specifically, feature matching is a technology based on computer vision. The similarity between feature points can be calculated using the description values of the feature points, so as to find the matching feature points corresponding to the same object or parts of the object in different images. For example, the key information in the image, such as edges, corners, textures, etc., can be extracted. Regarding feature points and feature matching, reference can be made to Section 7.1 of "Vision SLAM 14 Lectures: From Theory to Practice" written by Gao Xiang and Zhang Tao (published by Electronic Industry Press in March 2017). After that, the matching feature points in each picture or each frame of video are subjected to feature decomposition to obtain the initial camera pose (the pose of the camera when each picture is taken, which can be represented by the rotation matrix R and the translation vector t). Then, the mis-matched points in the initial camera pose are removed, and multiple iterative optimization calculations are performed. Finally, a more accurate camera pose is obtained to complete the sparse 3D reconstruction.

[0123] For the introduction to feature decomposition, reference can be made to Section 2.7 "Feature Decomposition" of the book "Deep Learning" written by Ian Goodfellow et al. and translated by Zhao Shenjian et al. (published by Posts and Telecommunications Press in August 2017; the original English book name is "Deep Learning").

[0124] In an optional embodiment, as Figure 5 shown, the dense 3D reconstruction method provided in the above step S320, that is, based on multiple pictures or multiple frames of video and the above camera pose for dense 3D reconstruction to obtain the 3D reconstruction effect diagram of the above cooking object, can be implemented by the following method steps:

[0125] Step S510: Extract feature points from multiple pictures or multiple frames of video;

[0126] Step S520: Perform feature matching on the above feature points to obtain the matching feature points;

[0127] Step S530: Calculate the disparity of each pixel point based on the above-matched feature points.

[0128] Step S540: Calculate the three-dimensional coordinates of each of the above pixel points based on the disparity and the camera pose.

[0129] Step S550: Form dense point cloud data based on the three-dimensional coordinates of each of the above pixel points.

[0130] Step S560: Generate the above three-dimensional reconstruction effect diagram based on the above dense point cloud data.

[0131] For the concept and description of disparity, please refer to Sections 2.1, 5.1.3, and 13.2.1 of "Vision SLAM Lectures: From Theory to Practice" written by Gao Xiang and Zhang Tao (Publishing House of Electronics Industry, March 2017).

[0132] It is easy to understand that in the examples of the present disclosure, point cloud data refers to the spatial points corresponding to the pixel points in the picture in three-dimensional space, and the point cloud data includes these spatial points and their spatial coordinates. Point cloud data can be understood as a set of points with spatial coordinate positions, or a set of vectors (vectors pointing from the camera position to the spatial coordinate positions of these spatial points, and the length of each vector is the distance between these spatial points and the camera position). For example, if a chicken wing is photographed in a picture, the points on the chicken wing in the picture are all pixel points, and the spatial points on the actual chicken wing corresponding to these pixel points are the point cloud data.

[0133] The above steps S510 to S560 for dense three-dimensional reconstruction are specifically as follows:

[0134] First, by obtaining the feature points of each picture or each frame of video in the image, the feature points in the image can be extracted, such as edges, corners, textures, etc.

[0135] Then, based on the matched feature points obtained by feature matching of the feature points, the disparity of each pixel point in each picture or each frame of video can be calculated; afterwards, using the disparity and the camera pose, through methods such as triangulation, the three-dimensional spatial coordinates of each pixel point can be calculated. After that, based on the three-dimensional coordinates of each of the above pixel points, dense point cloud data is formed, and three-dimensional reconstruction of the cooking object is performed. The size information of the cooking object can be observed from different perspectives, and then the point cloud data is converted into a mesh, and subsequent mesh optimization and texture mapping are carried out to obtain the three-dimensional reconstruction effect diagram of the cooking object.

[0136] The embodiments of the present disclosure can adopt common 3D reconstruction algorithms. For example, the Structure from Motion (SFM) algorithm can be used for sparse 3D reconstruction, and the Multi-view Stereo (MVS) algorithm can be used for dense 3D reconstruction, and so on. In this embodiment, at least one video or multiple pictures of the cooking object taken at different angles are obtained, and multiple frames of video or multiple pictures are uploaded to the 3D reconstruction algorithm interface, and the 3D reconstruction of the cooking object can be completed after waiting for a few seconds. However, the 3D reconstruction is not limited to the above algorithms, and it can also be other 3D reconstruction algorithms known to those skilled in the art, such as the deep learning model of Nerf, the Patch-Based Multi-View Stereo (PMVS) algorithm, the Large-Scale Direct SLAM (LSD-SLAM) algorithm, the DepthNet algorithm, and the Pyramid StereoMatching Network (PSMNet) algorithm, all of which can perform dense 3D reconstruction to obtain the 3D reconstruction effect diagram of the cooking object.

[0137] For the 3D reconstruction effect diagram obtained by the 3D reconstruction through the deep learning method of Nerf, reference can be made to “A Critical Analysis of NeRF-Based 3D Reconstruction” written by Remondino, F. et al. (Remondino, F.; Karami, A.; Yan, Z.; Mazzacca, G.; Rigon, S.; Qin, R. A Critical Analysis of NeRF-Based 3D Reconstruction. Remote Sens. 2023, 15, 3585).

[0138] For the above multi-view stereo (MVS) algorithm, the depth of each pixel point of each picture or each frame of video is estimated according to the relative pose and internal and external parameters of the shooting device obtained by SFM, so as to reconstruct the dense 3D structure of the scene.

[0139] The above PMVS (Patch-Based Multi-View Stereo) algorithm is a dense 3D reconstruction method based on multi-view geometry. It uses similar patches in multiple images for matching and aggregation to obtain the dense 3D reconstruction effect diagram of the scene.

[0140] The above LSD-SLAM (Large-Scale Direct SLAM) algorithm is a dense three-dimensional reconstruction method based on photometric consistency. It directly extracts and matches features from images, and uses geometric constraints for three-dimensional reconstruction to obtain a dense three-dimensional reconstruction effect diagram of the scene.

[0141] The above DepthNet algorithm is a dense three-dimensional reconstruction method based on deep learning. It uses a convolutional neural network to extract and match features from images, and uses an aggregation algorithm to fuse depth information from multiple perspectives to obtain a dense three-dimensional reconstruction effect diagram of the scene.

[0142] The above PSMNet (Pyramid Stereo Matching Network) algorithm is also a dense three-dimensional reconstruction method based on deep learning. It learns the mapping relationship between viewpoints and depth, extracts and matches features from the input images, and uses an aggregation algorithm to obtain a dense three-dimensional reconstruction effect diagram of the scene.

[0143] Finally, based on the three-dimensional reconstruction effect diagram of the cooking object, the size information of the cooking object, such as the thickness of the cooking object, is calculated by combining the characteristic sizes of the input reference object.

[0144] By adopting the cooking method provided in the examples of the present disclosure, by obtaining images of the cooking object taken at at least one shooting angle, where the images include multiple pictures or at least one video, the food type of the cooking object can be determined through a deep learning model according to the multiple pictures or at least one video; the cooking object is three-dimensionally reconstructed based on the images to obtain a three-dimensional reconstruction effect diagram, and the size information of the cooking object is determined based on the three-dimensional reconstruction effect diagram; furthermore, according to the size information and food type of the cooking object, cooking data corresponding to the cooking object is determined, where the cooking data includes: cooking temperature and / or cooking time, so that the cooking object can be cooked according to the cooking data, realizing intelligent cooking of the cooking object, thereby improving the cooking effect of the food and the user experience. It can solve the problem that the setting method of the cooking device is not intelligent enough and it is easy to burn the cooked food or the cooking time is insufficient during the operation of the cooking device.

[0145] According to one or more embodiments of the present disclosure, another cooking method is also provided. Figure 6 It is a flowchart of another cooking method provided by the embodiments of the present disclosure, as Figure 6 shown, the above method includes:

[0146] S610, obtain images of the cooking object taken at at least one shooting angle.

[0147] Optionally, the above cooking device may include: an oven, a grill, a microwave oven, a rice cooker, a pressure cooker, an automatic stir-fry machine, and so on.

[0148] Optionally, the cooking method provided by the embodiments of the present disclosure may run in a cooking device provided with a photographing device or on a smart device provided with a photographing device.

[0149] Optionally, the cooking method provided by the embodiments of the present disclosure may also run in any existing cooking device. In this case, before the cooking object is placed in the cooking device or before the cooking object is placed in the cooking device but the cooking device has not been turned on, the photographing device is used to photograph the cooking object at at least one photographing angle to obtain the above image.

[0150] S620. Obtain the food type of the cooking object.

[0151] Optionally, the food type of the cooking object, such as beef or meat, may also be informed to the processor by input. In this way, it is not necessary to automatically identify the food type of the cooking object from the input image by using a deep learning algorithm, which reduces the requirement for the computing power of the processor, thereby reducing the computing burden of the processor and enabling the final result to be obtained more quickly.

[0152] Among them, the execution of the above steps S610 and S620 may not be in a specific order. For example, in some embodiments, step S620 may be executed first, where the user first inputs the food type of the cooking object so that the cooking device or smart device obtains the food type of the cooking object, and then step S610 is executed, so that the cooking device or smart device obtains the image of the cooking object photographed at at least one photographing angle; of course, steps S610 and S620 may also be performed simultaneously.

[0153] S630. Perform three-dimensional reconstruction on the cooking object according to the above image to obtain a three-dimensional reconstruction effect diagram, and determine the size information of the cooking object based on the three-dimensional reconstruction effect diagram.

[0154] In some examples, the execution of the above steps S620 and S630 may not be in a specific order.

[0155] S640. Determine the cooking data corresponding to the cooking object according to the size information of the cooking object and the above food type, where the cooking data includes: cooking temperature and / or cooking time;

[0156] S650. Perform cooking processing on the cooking object according to the above cooking data.

[0157] Finally, according to the size information and food type of the cooking object, cooking data corresponding to the cooking object is determined, where the cooking data includes: cooking temperature and / or cooking time; and the cooking object is subjected to specific cooking processing according to the determined cooking data to achieve a better cooking effect.

[0158] In the examples of the present disclosure, a cooking device or a smart device obtains the food type of the cooking object input by the user, and the size information of the cooking object obtained by three-dimensionally reconstructing the cooking object based on the image captured of the cooking object. Then, according to the size information and food type of the cooking object, the cooking temperature and / or cooking time of the cooking object can be relatively accurately predicted automatically, improving the user experience; even users who have no experience in using cooking devices such as ovens can easily handle various cooking tasks.

[0159] It should be noted that for optional examples of the cooking method in this embodiment, one or more optional examples given in the above cooking method can be referred to, and details will not be elaborated here.

[0160] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data that have been authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0161] According to one or more embodiments of the present disclosure, a cooking device is provided. Figure 7 The following is a structural block diagram of a cooking device provided by an embodiment of the present disclosure, as Figure 7 shown. The above device includes:

[0162] A photographing device 601 for photographing an image of a cooking object at at least one photographing angle;

[0163] A processor 602 for determining the food type of the cooking object according to the above image through a deep learning model; three-dimensionally reconstructing the cooking object based on the above image to obtain a three-dimensional reconstruction effect diagram, and determining the size information of the cooking object based on the above three-dimensional reconstruction effect diagram; determining cooking data corresponding to the cooking object according to the size information and the food type of the cooking object, where the cooking data includes: cooking temperature and / or cooking time;

[0164] A cooking device 603 for performing cooking processing on the cooking object according to the above cooking data.

[0165] According to one or more embodiments of the present disclosure, a cooking device is provided, which includes:

[0166] A photographing device for photographing an image of a cooking object at at least one photographing angle;

[0167] A processor for obtaining the food type of the cooking object; and for performing three-dimensional reconstruction on the cooking object based on the above image to obtain a three-dimensional reconstruction effect diagram, and determining the size information of the cooking object based on the three-dimensional reconstruction effect diagram; determining cooking data corresponding to the cooking object according to the size information of the cooking object and the food type, wherein the cooking data includes: cooking temperature and / or cooking time;

[0168] A cooking device for performing cooking processing on the cooking object according to the cooking data.

[0169] It should be noted that for optional examples of the above cooking device, one or more optional examples given in the above cooking method can be referred to.

[0170] In an exemplary embodiment, the present disclosure also provides an electronic device, including:

[0171] A photographing device, a processor, and a memory connected to the processor;

[0172] The above photographing device is used to obtain an image of a cooking object photographed at at least one photographing angle;

[0173] The above memory stores computer execution instructions;

[0174] The above processor executes the computer execution instructions stored in the above memory to implement the method as described in any one of the above.

[0175] In an exemplary embodiment, the present disclosure also provides a computer-readable storage medium, in which computer execution instructions are stored, and when the computer execution instructions are executed by a processor, they are used to implement the method as described in any one of the above.

[0176] In an exemplary embodiment, the present disclosure also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the method as described in any one of the above.

[0177] To implement the above embodiments, the present disclosure also provides an electronic device. Refer to Figure 8, which shows a schematic structural diagram of an electronic device 700 suitable for implementing the embodiments of the present disclosure. The electronic device 700 may be a terminal device or a server, such as the above-mentioned intelligent device. Among them, the terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, messaging devices, game consoles, medical devices, fitness devices, personal digital assistants (Personal Digital Assistant, abbreviated as PDA), tablet computers (Portable Android Device, abbreviated as PAD), portable media players (Portable Media Player, abbreviated as PMP), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0178] As Figure 8 shown, the electronic device 700 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 701, which may perform various appropriate actions and processes according to the program stored in the read-only memory (Read Only Memory, abbreviated as ROM) 702 or the program loaded from the storage device 708 into the random access memory (Random Access Memory, abbreviated as RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. The input / output (I / O) interface 705 is also connected to the bus 704.

[0179] Generally, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (Liquid Crystal Display, abbreviated as LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 8 the electronic device 700 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. More or fewer devices may be alternatively implemented or had.

[0180] In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program carried on a computer-readable medium, and the computer program includes program code for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by a processing device 701, the above-described functions defined in the method of the embodiment of the present disclosure are performed.

[0181] It should be noted that the above computer-readable medium in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable signal medium can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0182] The above computer-readable medium can be included in the above electronic device; or it can exist separately without being assembled into the electronic device.

[0183] The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device is caused to perform the method shown in the above embodiment.

[0184] Computer program code for performing the operations of this disclosure may be written in one or more programming languages or combinations thereof. The foregoing programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0185] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0186] The units described in the embodiments of this disclosure may be implemented in software or in hardware. Among them, the name of the unit does not constitute a limitation on the unit itself in some cases. For example, the first acquisition unit may also be described as "the unit for acquiring at least two Internet protocol addresses".

[0187] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), system on a chip (SOC), complex programmable logic devices (CPLD), and so on.

[0188] In the context of the present disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0189] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the art that are not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0190] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the figures, and various modifications and changes may be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A cooking method, characterized in that, The method includes: Obtaining images of a cooking object captured at at least one shooting angle; Determining the food type of the cooking object through a deep learning model according to the images; Performing three-dimensional reconstruction on the cooking object according to the images to obtain a three-dimensional reconstruction effect diagram, and determining the size information of the cooking object based on the three-dimensional reconstruction effect diagram; Determining cooking data corresponding to the cooking object according to the size information and the food type of the cooking object, wherein the cooking data includes: cooking temperature and / or cooking time; Performing cooking processing on the cooking object according to the cooking data.

2. A cooking method, characterized in that, The method includes: Obtaining images of a cooking object captured at at least one shooting angle; Obtaining the food type of the cooking object; Performing three-dimensional reconstruction on the cooking object according to the images to obtain a three-dimensional reconstruction effect diagram, and determining the size information of the cooking object based on the three-dimensional reconstruction effect diagram; Determining cooking data corresponding to the cooking object according to the size information and the food type of the cooking object, wherein the cooking data includes: cooking temperature and / or cooking time; Performing cooking processing on the cooking object according to the cooking data.

3. The method according to claim 1 or 2, characterized in that, The images include at least one of the following: The images of the cooking object captured at at least one of the shooting angles by a shooting device before the cooking object is placed in a cooking device; In response to the cooking object being placed in the cooking device, the images of the cooking object captured at at least one of the shooting angles by a shooting device arranged in the cooking device.

4. The method according to claim 1 or 2, characterized in that, The determining the size information of the cooking object based on the three-dimensional reconstruction effect diagram includes: Obtaining the characteristic size of a reference object of the cooking object; Determining the size information of the cooking object according to the three-dimensional reconstruction effect diagram of the cooking object and the characteristic size of the reference object.

5. The method according to claim 1 or 2, characterized in that, The performing three-dimensional reconstruction on the cooking object according to the images to obtain a three-dimensional reconstruction effect diagram includes: Performing sparse three-dimensional reconstruction based on multiple pictures or multiple frames of video to obtain the camera pose; Performing dense three-dimensional reconstruction based on multiple pictures or multiple frames of video and the camera pose to obtain the three-dimensional reconstruction effect diagram of the cooking object.

6. The method according to claim 5, characterized in that, The performing sparse three-dimensional reconstruction based on multiple pictures or multiple frames of video to obtain the camera pose includes: Extracting feature points from multiple pictures or multiple frames of video; Performing feature matching on the feature points to obtain matching feature points; Performing feature decomposition through the matching feature points to obtain the camera pose.

7. The method according to claim 5, wherein The performing dense three-dimensional reconstruction based on multiple pictures or multiple frames of video and the camera pose to obtain the three-dimensional reconstruction effect diagram of the cooking object includes: Extracting feature points from multiple pictures or multiple frames of video; Performing feature matching on the feature points to obtain matching feature points; Calculating the disparity of each pixel point based on the matching feature points; Calculating the three-dimensional coordinates of each pixel point based on the disparity and the camera pose; Determining dense point cloud data based on the three-dimensional coordinates of each pixel point; Generating the three-dimensional reconstruction effect diagram based on the dense point cloud data.

8. A cooking device, characterized in that, The cooking device includes: A photographing device for photographing an image of a cooking object at at least one photographing angle; A processor for determining the food type of the cooking object through a deep learning model according to the image; performing three-dimensional reconstruction on the cooking object according to the image to obtain a three-dimensional reconstruction effect diagram, and determining the size information of the cooking object based on the three-dimensional reconstruction effect diagram; determining cooking data corresponding to the cooking object according to the size information and the food type of the cooking object, wherein the cooking data includes: cooking temperature and / or cooking time; A cooking device for performing cooking processing on the cooking object according to the cooking data.

9. A cooking device, characterized in that, The cooking device includes: A photographing device for photographing an image of a cooking object at at least one photographing angle; A processor for obtaining the food type of the cooking object; and for performing three-dimensional reconstruction on the cooking object according to the image to obtain a three-dimensional reconstruction effect diagram, and determining the size information of the cooking object based on the three-dimensional reconstruction effect diagram; determining cooking data corresponding to the cooking object according to the size information and the food type of the cooking object, wherein the cooking data includes: cooking temperature and / or cooking time; A cooking device for performing cooking processing on the cooking object according to the cooking data.

10. An electronic device, characterized in that, It includes: A photographing device, a processor, and a memory connected to the processor; The photographing device is used for obtaining an image of a cooking object photographed at at least one photographing angle; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to implement the method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, Computer execution instructions are stored in the computer-readable storage medium, and when the computer execution instructions are executed by the processor, they are used to implement the method according to any one of claims 1 to 7.

12. A computer program product, characterized in that, It includes a computer program, and when the computer program is executed by the processor, it implements the method according to any one of claims 1 to 7.

Images