Oven-based mode matching method and device, storage medium and electronic device

Abstract

The invention discloses a mode matching method and device based on an oven, a storage medium and an electronic device, and relates to the technical field of smart home, and the method comprises the steps: determining food material information of food materials in the oven through first image data collected by an image collection device preset in the oven; wherein the food material information comprises the category of the food material; acquiring a first priori depth value of a baking tray for placing the food material in the oven in an empty tray state; wherein the first priori depth value is determined through a depth camera in advance; determining a depth estimation map corresponding to the food material through the first priori depth value and a second priori depth value corresponding to the category of the food material; and determining an oven mode of the oven through the first image data and the depth estimation map.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and more specifically, to a pattern matching method and apparatus, storage medium and electronic device based on an oven. Background Technology

[0002] In modern family kitchens, smart ovens are gaining popularity among consumers due to their convenience and intelligent features. Ovens, equipped with integrated RGB cameras, can monitor the interior in real time, providing users with visual information about the food's cooking status. Furthermore, ovens can automatically identify the type and quantity of ingredients and select the appropriate cooking mode based on these factors, providing fundamental data support for automated cooking mode selection.

[0003] However, the current level of intelligence in smart ovens is still limited by a key technological challenge: the inability to accurately estimate the distance between the food and the heating element, thus hindering the accurate determination of the oven mode. During cooking, the distance between the food and the heating element is a crucial parameter because it directly affects the evenness of heating and the accuracy of cooking time. For example, for the same ingredient (such as sweet potatoes), if placed on the upper rack closer to the heating element, the cooking time needs to be shorter than if placed on the lower rack further away; otherwise, the food on the upper rack may be overheated, while the food on the lower rack may not be fully cooked.

[0004] There is currently no effective solution to the problem that ovens cannot accurately estimate the distance between food and the heating element, which leads to an inability to effectively determine the oven mode.

[0005] Therefore, it is necessary to improve the relevant technology to overcome the aforementioned defects. Summary of the Invention

[0006] This application provides a pattern matching method and apparatus, storage medium and electronic device based on an oven, to at least solve the problem in the related art that the oven cannot accurately estimate the distance between the food and the heating element, resulting in the inability to effectively determine the oven mode.

[0007] According to one aspect of the embodiments of this application, an oven-based pattern matching method is provided, comprising: determining food information of food in the oven by acquiring first image data from an image acquisition device pre-installed in the oven; wherein the food information includes: the category of the food; obtaining a first prior depth value of the baking tray in the oven where the food is placed in an empty state; wherein the first prior depth value is pre-determined by a depth camera; determining a depth estimation map corresponding to the food by using the first prior depth value and a second prior depth value corresponding to the category of the food; and determining the oven mode of the oven by using the first image data and the depth estimation map.

[0008] In an exemplary embodiment, determining the food information of ingredients in the oven using first image data acquired by an image acquisition device pre-installed in the oven includes: inputting the first image data into a food target segmentation model and determining the category of the food using the output of the food target segmentation model; inputting the first image data into an oven baking layer detection model and determining the target layer number of the baking tray on which the food is placed in the oven using the output of the oven baking layer detection model; wherein the food information further includes the target layer number; wherein the training data of the food target segmentation model is target image data labeled with food categories, and the training data of the oven baking layer detection model is target image data labeled with baking layer positions; the target image data are image data of different categories of food placed on different baking trays in the oven.

[0009] In an exemplary embodiment, before obtaining the first prior depth value of the baking tray containing the food in the oven in an empty state, the method further includes: determining the pixel coordinates of each corner point on a calibration plate placed in the oven in second image data when the baking tray in an empty state is placed on any layer in the oven, and determining the first three-dimensional coordinates of each corner point in the first coordinate system corresponding to the depth camera using third image data and a first depth map; determining a third prior depth value using the pixel coordinates and the first three-dimensional coordinates, wherein the third prior depth value is the prior depth value corresponding to the baking tray in an empty state being placed on any layer in the oven.

[0010] In an exemplary embodiment, determining a third prior depth value using the pixel coordinates and the first three-dimensional coordinates includes: determining the relative pose of the depth camera and the image acquisition device using the pixel coordinates and the first three-dimensional coordinates; determining the corner depth value of each corner point in the second coordinate system corresponding to the image acquisition device using the relative pose; and performing bilinear interpolation on the corner depth value to obtain the third prior depth value; wherein the third prior depth value includes: the first prior depth value.

[0011] In an exemplary embodiment, determining the depth estimation map corresponding to the food ingredient by using the first prior depth value and the second prior depth value corresponding to the category of the food ingredient includes: obtaining the segmentation region of the food ingredient determined by the food ingredient target segmentation model based on the first image data; superimposing the second prior depth value corresponding to the category of the food ingredient with the first prior depth value in the segmentation region to obtain the depth estimation map, wherein the prior depth values ​​of all categories of food ingredients that the food ingredient target segmentation model can identify are determined by the average depth of all categories of food ingredients.

[0012] In an exemplary embodiment, determining the oven mode of the oven using the first image data and the depth estimation map includes: determining the second three-dimensional coordinates of each pixel in the first image data using the first image data and the depth estimation map; determining the third three-dimensional coordinates of the food in the second coordinate system corresponding to the image acquisition device using the second three-dimensional coordinates and the segmented region of the food; wherein the segmented region is determined based on the first image data using a food target segmentation model; and determining the oven mode using the third three-dimensional coordinates of the food and the center of the heating element of the oven.

[0013] In an exemplary embodiment, determining the oven mode using the third three-dimensional coordinates of the food and the center of the oven's heating element includes: obtaining a transformation relationship between the second coordinate system corresponding to the image acquisition device and the third coordinate system corresponding to the center of the oven's heating element, wherein the transformation relationship is determined by the physical distance between the image acquisition device and the center of the heating element; converting the third three-dimensional coordinates of the food into target three-dimensional coordinates of the food in the third coordinate system corresponding to the center of the heating element using the transformation relationship; and determining the oven mode using the target three-dimensional coordinates.

[0014] According to another aspect of the embodiments of this application, an oven-based pattern matching device is also provided, comprising: a first determining module, configured to determine food information of food in the oven by means of first image data acquired by an image acquisition device preset in the oven; wherein the food information includes: the category of the food; an acquiring module, configured to acquire a first prior depth value of the baking tray in the oven where the food is placed in an empty tray state; wherein the first prior depth value is predetermined by a depth camera; a second determining module, configured to determine a depth estimation map corresponding to the food by means of the first prior depth value and a second prior depth value corresponding to the category of the food; and a third determining module, configured to determine the oven mode of the oven by means of the first image data and the depth estimation map.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the above-described oven-based pattern matching method at runtime.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the oven-based pattern matching method through the computer program.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described oven-based pattern matching method.

[0018] This application determines the food information of the food inside the oven by acquiring first image data from an image acquisition device pre-installed inside the oven. The food information includes: the category of the food; obtaining a first prior depth value of the baking tray containing the food in an empty state; wherein the first prior depth value is pre-determined by a depth camera; determining a depth estimation map corresponding to the food using the first prior depth value and a second prior depth value corresponding to the food category; and determining the oven mode using the first image data and the depth estimation map. In other words, by accurately determining the first prior depth value of the empty baking tray and the second prior depth value of the category of the food placed in the oven, and then using the depth estimation map determined based on the first and second prior depth values, and the first image data acquired by the image acquisition device inside the oven, the oven mode that the oven should use can be accurately determined. Therefore, the above technical solution solves the problem in related technologies where ovens cannot accurately estimate the distance between the food and the heating element, leading to an inability to effectively determine the oven mode; achieving an effective and accurate technical effect of determining the oven mode. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the hardware environment for an optional oven-based pattern matching method according to an embodiment of this application;

[0022] Figure 2 This is a flowchart of an optional oven-based pattern matching method according to an embodiment of this application;

[0023] Figure 3 This is another flowchart of an optional oven-based pattern matching method according to an embodiment of this application;

[0024] Figure 4 This is a structural block diagram of an optional oven-based pattern matching device according to an embodiment of this application;

[0025] Figure 5 This is another structural block diagram of an optional oven-based pattern matching device according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] According to one aspect of the embodiments of this application, an oven-based pattern matching method is provided. This oven-based pattern matching method is widely applicable to whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligencehouse ecosystems. Optionally, in this embodiment, the above-mentioned oven-based pattern matching method can be applied to, for example... Figure 1 The hardware environment shown consists of multiple terminal devices 102 and a server 104. For example... Figure 1 As shown, server 104 is connected to multiple terminal devices 102 via a network and can be used to provide services (such as application services) to terminals or clients installed on terminals. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0029] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0030] This embodiment provides a pattern matching method based on an oven, including but not limited to applications applied to ovens. Figure 2 This is a flowchart of an oven-based pattern matching method according to an embodiment of this application, the process including the following steps:

[0031] Step S202: Determine the food information of the food in the oven by using the first image data acquired by the image acquisition device preset inside the oven; wherein, the food information includes: the category of the food;

[0032] Optionally, step S202 can be performed if the food is detected to be placed in the oven.

[0033] Step S204: Obtain the first prior depth value of the baking tray containing the ingredients in the oven when it is empty; wherein, the first prior depth value is determined in advance by a depth camera;

[0034] Step S206: Determine the depth estimation map corresponding to the food ingredient by using the first prior depth value and the second prior depth value corresponding to the category of the food ingredient;

[0035] Step S208: Determine the oven mode of the oven using the first image data and the depth estimation map.

[0036] Through the above steps, the food information of the ingredients in the oven is determined by the first image data acquired by the image acquisition device pre-installed in the oven. The food information includes: the category of the food; obtaining a first prior depth value of the baking tray in the oven when the food is empty; wherein the first prior depth value is pre-determined by a depth camera; determining a depth estimation map corresponding to the food category using the first prior depth value and a second prior depth value corresponding to the food category; and determining the oven mode using the first image data and the depth estimation map. In other words, by accurately determining the first prior depth value of the empty baking tray and the second prior depth value of the category of the food placed in the oven, and then using the depth estimation map determined based on the first and second prior depth values, and the first image data acquired by the image acquisition device in the oven, the oven mode that the oven should use can be accurately determined. Therefore, by adopting the above technical solution, the problem in related technologies where ovens cannot accurately estimate the distance between the food and the heating element, resulting in an inability to effectively determine the oven mode, is solved; achieving an effective and accurate technical effect of determining the oven mode.

[0037] In an exemplary embodiment, determining the food information of ingredients in the oven using first image data acquired by an image acquisition device pre-installed in the oven includes: inputting the first image data into a food target segmentation model and determining the category of the food using the output of the food target segmentation model; inputting the first image data into an oven baking layer detection model and determining the target layer number of the baking tray on which the food is placed in the oven using the output of the oven baking layer detection model; wherein the food information further includes the target layer number; wherein the training data of the food target segmentation model is target image data labeled with food categories, and the training data of the oven baking layer detection model is target image data labeled with baking layer positions; the target image data are image data of different categories of food placed on different baking trays in the oven.

[0038] It should be noted that the target layer number refers to the layer number in which the baking tray containing the food is detected to be inside the oven. The first prior depth value obtained in step S204 refers to the prior depth value when the baking tray containing the food is placed in the oven at the target layer number in an empty state.

[0039] The image acquisition device built into a smart oven can typically be an RGB camera, also known as an RGB sensor. In this embodiment, the smart oven acquires first image data of the oven's interior using its built-in RGB camera, and then utilizes two key machine learning models to extract food information, specifically including:

[0040] The acquired image data is input into a food segmentation model, which can identify and segment the outlines and types of different foods inside the oven. The model's output will include food category information, that is, it can identify specific types of food such as sweet potatoes, bread, and chicken, which is the basis for the smart oven to select cooking modes.

[0041] Simultaneously, the acquired image data is input into the oven rack detection model. This model learns the characteristics of the oven's internal rack structure through training, enabling it to determine the rack level of the baking tray based on its relative position in the image. The output of the rack detection model provides information on the target rack level of the food or baking tray, which is crucial for fine-tuning the cooking mode by incorporating food depth information.

[0042] Finally, the food category information output by the food target segmentation model and the target layer number information of the food output by the oven baking layer detection model are combined to form complete food information.

[0043] Through the above exemplary embodiments, the smart oven can automatically identify the types of food inside the oven and determine which shelf the food (the baking tray used to hold the food) is placed on, laying the foundation for subsequent depth estimation and intelligent cooking mode selection. This automated information extraction capability greatly enhances the oven's intelligence level, enabling it to adjust cooking parameters more personalized and precisely, ensuring the quality of food cooking.

[0044] In some optional embodiments, before obtaining the first prior depth value of the baking tray containing the ingredients in the oven when it is empty, such as... Figure 3 As shown, the method further includes:

[0045] Step S302: When the baking tray in an empty state is placed on any layer in the oven, determine the pixel coordinates of each corner point on the calibration plate placed in the oven in the second image data, and determine the first three-dimensional coordinates of each corner point in the first coordinate system corresponding to the depth camera through the third image data and the first depth map;

[0046] Step S304: Determine a third prior depth value using the pixel coordinates and the first three-dimensional coordinates, wherein the third prior depth value is the prior depth value corresponding to the baking tray being placed on any layer in the oven when the baking tray is in an empty state.

[0047] The second image data is a color image of the calibration plate acquired by the image acquisition device, the third image data is a color image of the calibration plate acquired by the depth data acquisition device, and the first depth map is a depth map of the calibration plate acquired by the depth camera.

[0048] Step S302 includes: extracting the pixel coordinates of each corner point in the second image data using a checkerboard corner detection algorithm; extracting the target pixel coordinates of each corner point in the third image data using a checkerboard corner detection algorithm; and determining the first three-dimensional coordinates of each corner point in the first coordinate system corresponding to the depth camera using the target pixel coordinates corresponding to the third image data and the first depth map.

[0049] Furthermore, for step S304 above, an optional implementation includes: determining the relative pose of the depth camera and the image acquisition device using the pixel coordinates and the first three-dimensional coordinates; determining the corner depth value of each corner point in the second coordinate system corresponding to the image acquisition device using the relative pose; performing bilinear interpolation on the corner depth value to obtain the third prior depth value; wherein the third prior depth value includes: the first prior depth value.

[0050] It is understood that, in this embodiment of the application, in order to achieve accurate estimation of food depth even with only an RGB camera, a key preprocessing step is to perform camera calibration and acquire prior depth information inside the oven. Specifically, when the baking tray is empty and placed on any layer inside the oven, simultaneous acquisition is performed using an image acquisition device and a depth camera (e.g., an RGBD camera) to obtain the data required for calibration. Specifically:

[0051] A calibration plate with a checkerboard pattern is placed on a baking tray without food, and the baking tray is placed on each rack of the oven. A second image (i.e., an RGB image) of the calibration plate is captured using a pre-installed RGB camera inside the oven. Simultaneously, a third image (RGB image) and a first depth map (depth map) of the same calibration plate are captured using an external RGBD depth camera. That is, both the second and third image data are color image data.

[0052] Then, a checkerboard corner detection algorithm was applied to extract the pixel coordinates and target pixel coordinates of each corner point on the calibration board from the second image data (RGB camera) and the third image data (RGBD camera), respectively. This ensured that the corner points of the checkerboard could be accurately located during the calibration process, thus providing accurate reference points for subsequent calibration and depth estimation.

[0053] Furthermore, the first three-dimensional coordinates of each corner point in the first coordinate system corresponding to the depth camera are determined using the target pixel coordinates corresponding to the third image data and the first depth map. This includes: using the target pixel coordinates of the third image data (RGBD camera), the first depth map, and the camera intrinsic parameters of the depth camera, determining the first three-dimensional coordinates (X, Y, Z) of each corner point on the calibration board in the depth camera coordinate system. Further, based on the obtained first three-dimensional coordinates, the relative pose (rotation matrix R and translation vector t) between the RGB camera and the RGBD depth camera is calculated using a camera calibration algorithm (such as OpenCV's SolvePnP). Using the calculated relative pose, the three-dimensional coordinates of the corner point in the depth camera coordinate system are transformed to the RGB camera coordinate system to obtain the depth value of the corner point inside the oven.

[0054] Repeat the above process to calibrate each layer in the oven, thereby obtaining the prior depth value of each layer when the baking tray is empty, that is, the preset depth information relative to the heating element when the baking tray or food is placed on any layer in the oven.

[0055] Optionally, the calibration plates can be placed layer by layer according to the number of baking trays placed.

[0056] Optionally, a structured light sensor could be installed inside the oven. By projecting a specific structured light pattern and analyzing its deformation on the food or baking tray, the depth information of the food or baking tray could be estimated in real time. Alternatively, a time-of-flight (ToF) sensor could be integrated inside the oven. A ToF sensor directly measures distance by calculating the time difference between light emission and reflection back to the sensor, thus obtaining real-time depth information of the food or baking tray. However, both structured light sensors and time-of-flight sensors could potentially increase the oven's production cost.

[0057] In an exemplary embodiment, determining the depth estimation map corresponding to the food ingredient by using the first prior depth value and the second prior depth value corresponding to the category of the food ingredient includes: obtaining the segmentation region of the food ingredient determined by the food ingredient target segmentation model based on the first image data; superimposing the second prior depth value corresponding to the category of the food ingredient with the first prior depth value in the segmentation region to obtain the depth estimation map, wherein the prior depth values ​​of all categories of food ingredients that the food ingredient target segmentation model can identify are determined by the average depth of all categories of food ingredients.

[0058] In this embodiment, after inputting the first image data into the food ingredient segmentation model, the output includes both the food ingredient category and the segmented region of the food ingredient. The segmented region indicates the specific location and shape of the food ingredient in the first image data. Furthermore, in this embodiment, for all categories of food ingredients that the food ingredient segmentation model can identify, a second prior depth value for each food ingredient category is determined in advance using the average depth of a large number of samples. For example, by collecting size information of various sweet potatoes, bread, chicken legs, and other ingredients from the market, the average depth of various ingredients can be calculated, and this information will be used as a reference during cooking pattern matching.

[0059] After the food segmentation region is determined, the second prior depth value corresponding to the food category is superimposed with the first prior depth value of the baking pan in the target layer when it is empty. Specifically, for each pixel within the segmentation region, its depth value is changed to the second prior depth value corresponding to that food category; while pixels outside the segmentation region, i.e., non-food areas, retain the first prior depth value, i.e., the depth estimate of the baking pan in that layer when empty. In this way, the depth estimation map will contain the true depth information of the food in the target layer, while the non-food areas retain the depth estimate of the baking pan itself. Thus, each pixel in the depth estimation map will contain the depth value of the food or baking pan in the oven coordinate system, thereby providing the smart oven with an accurate estimate of the distance between the food and the heating element.

[0060] Through the above embodiments, the smart oven can not only identify the type and layer of food, but also generate a depth estimation map based on the prior depth information of the food's segmented areas and categories. This allows the oven to intelligently adjust the cooking mode and time according to the depth of the food during the cooking process, ensuring optimal cooking results. This method effectively overcomes the limitation of ovens with only RGB cameras that cannot directly acquire depth information, and is a key technology for improving the intelligence level of ovens.

[0061] In an exemplary embodiment, determining the oven mode of the oven using the first image data and the depth estimation map includes: determining the second three-dimensional coordinates of each pixel in the first image data using the first image data and the depth estimation map; determining the third three-dimensional coordinates of the food in the second coordinate system corresponding to the image acquisition device using the second three-dimensional coordinates and the segmented region of the food; wherein the segmented region is determined based on the first image data using a food target segmentation model; and determining the oven mode using the third three-dimensional coordinates of the food and the center of the heating element of the oven.

[0062] Furthermore, determining the oven mode using the third three-dimensional coordinates of the food and the center of the oven's heating element includes: obtaining the transformation relationship between the second coordinate system corresponding to the image acquisition device and the third coordinate system corresponding to the center of the oven's heating element, wherein the transformation relationship is determined by the physical distance between the image acquisition device and the center of the heating element; converting the third three-dimensional coordinates of the food into target three-dimensional coordinates of the food in the third coordinate system corresponding to the center of the heating element using the transformation relationship; and determining the oven mode using the target three-dimensional coordinates.

[0063] After determining the depth estimation map, the smart oven intelligently determines the oven mode by integrating RGB image data and the depth estimation map, providing the optimal cooking solution for different ingredients and locations. The specific process includes:

[0064] First, the first image data (RGB image) is combined with the generated depth estimation map. Using the depth value of each pixel in the depth estimation map, together with the pixel coordinates of the RGB image, the second three-dimensional coordinates of each pixel in the oven coordinate system are determined according to the intrinsic parameter matrix of the image acquisition device (RGB camera) inside the oven. This process is actually a three-dimensional spatial backprojection of each pixel, converting two-dimensional image information into three-dimensional spatial information.

[0065] Secondly, for the pixels within the segmented area of ​​the food ingredient, based on the second three-dimensional coordinates obtained in the previous steps, the information belonging to the food ingredient within these coordinates is further extracted, namely, the third three-dimensional coordinates of the food ingredient in the coordinate system of the RGB image acquisition device. In this way, the smart oven can accurately obtain the three-dimensional position information of the food ingredient inside the oven.

[0066] Finally, based on the third and third-dimensional coordinates of the food, the transformation relationship (R',t') between the coordinate system of the image acquisition device and the coordinate system of the oven heating element center is obtained. This relationship has been determined by physical distance measurement before the oven leaves the factory. By applying this transformation relationship, the third and third-dimensional coordinates of the food are converted into the target three-dimensional coordinates of the food in the coordinate system of the heating element center. According to the depth distance of the food from the heating element (i.e., the Z value in the target three-dimensional coordinates), combined with information such as the type, size, and quantity of the food, the smart oven can automatically select the most suitable oven mode, including adjusting parameters such as temperature setting and time control, to achieve precise cooking of different foods at different depths. Specifically, the smart oven can have a pre-stored database of oven modes based on depth distance, food type, size, and quantity. Optionally, a preset baking mode for that type of food in the database can be selected first, and the temperature and time settings of each stage in the preset baking mode can be adjusted by increasing or decreasing a preset percentage (e.g., 20%) based on the Z value in the target three-dimensional coordinates. When the depth distance is relatively short (e.g., determining the average Z value in the three-dimensional coordinates of all targets, which is less than the first value), a higher temperature and a shorter cooking time are needed to avoid overheating. The temperature settings for each stage can be increased by a preset percentage, and the time settings can be decreased by a preset percentage. When the depth distance is moderate, the temperature and time can be set according to the standard (e.g., the average value is greater than the first value and less than the second value (the first value is less than the second value)). When the depth distance is long, the temperature needs to be appropriately reduced and the cooking time extended to ensure that the inside of the food is also fully heated (e.g., the average distance is greater than the second value and less than the third value (the second value is less than the third value)).

[0067] The embodiments described above in this application take into account the actual distance between the food and the heating source, enabling the oven to perform personalized and precise cooking, effectively avoiding uneven cooking caused by varying distances between the food and the heating source. This function of the smart oven greatly improves cooking results and user experience, enabling more intelligent and precise selection of cooking modes without increasing costs, utilizing existing hardware and deep information.

[0068] Obviously, the embodiments described above are only some embodiments of this application, and not all embodiments. To better understand the oven-based pattern matching method described above, the process is explained below with reference to embodiments, but this is not intended to limit the technical solutions of the embodiments of this application. Specifically:

[0069] In this embodiment, an RGBD camera (equivalent to the depth camera in the above embodiment) is matched with an RGB camera (equivalent to the image acquisition device in the above embodiment) to estimate the prior depth information within the field of view of the RGB camera. This information is then combined with the food category obtained from food target detection, and based on the prior depth information of the food category, more accurate depth estimation information is obtained.

[0070] Since ovens already manufactured typically have only one RGB camera, meaning they cannot obtain depth information, the technical solution of this application is generally divided into two parts: the first part is before the oven leaves the factory but is delivered to the consumer (equivalent to the user in the above embodiment), and the second part is after the oven leaves the factory and is delivered to the consumer. The specific steps are as follows:

[0071] Part 1: Before the oven leaves the factory but is delivered to the consumer:

[0072] 1) The RGB camera inside the oven has a fixed structure at the factory and is installed inside the oven, so it cannot be moved arbitrarily. However, the camera calibration board can be moved without moving the RGB camera, and the intrinsic parameter matrix K of the RGB camera can be obtained through the camera calibration algorithm.

[0073] 2) There is another RGBD camera with known intrinsic parameter matrix K'. Place it very close to the RGB camera to ensure that neither camera can see the other's camera within its field of view, and both can see the complete checkerboard calibration board at a certain distance.

[0074] 3) Keep RGB and RGBD stationary and take one picture each. RGB will only get one color image (equivalent to the second image data in the above embodiment), while RGBD will get one color image (equivalent to the third image data in the above embodiment) and one depth image (equivalent to the first depth image in the above embodiment). Usually, the output image of RGBD is aligned, that is, the number of pixels in the RGBD color image and the depth image are the same. However, it is not required that the number of pixels in the RGB and RGBD color images be the same. Also, since the field of view of RGBD is wider than that of RGB camera, the field of view of RGBD color image usually includes the field of view of RGB color image.

[0075] 4) Using the OpenCV checkerboard corner detection method, extract the checkerboard corners on the RGBD and RGB color images. It is required that the pixel coordinates of all corners of the checkerboard be obtained; otherwise, the current checkerboard position is not feasible and the position needs to be readjusted until the pixel coordinates of all corners can be obtained.

[0076] 5) Based on the pixel coordinates of RGBD (equivalent to the pixel coordinates corresponding to the third image data mentioned above), camera intrinsics, and depth map (i.e. the first depth map mentioned above), the 3D coordinate position of each corner point based on the RGBD camera coordinate system (equivalent to the first coordinate system mentioned above) can be calculated (equivalent to the first three-dimensional coordinates mentioned above).

[0077] 6) Using OpenCV's SolvePnP method, the 3D coordinates of RGBD (equivalent to the first coordinate system mentioned above) and the 2D coordinates of RGB (equivalent to the pixel coordinates corresponding to the second image data mentioned above) are matched to optimize the relative pose (R,t) between RGBD and the RGB camera.

[0078] 7) The corner depth value in the RGB camera coordinate system (equivalent to the second coordinate system in the above embodiment) can be calculated based on the relative pose (R,t). However, since the number of pixels in the RGB and RGBD cameras is inconsistent, bilinear interpolation is required for the RGB corner depth to obtain a depth map consistent with the RGB camera pixels (equivalent to the third prior depth value mentioned above).

[0079] It should be noted that all the above methods 1)-7) are based on the premise that the oven is empty (i.e. there is no food or baking tray).

[0080] Then, place the baking tray on different layers (equivalent to any layer in the above embodiments) and repeat the above operation in sequence to obtain the depth distance value when there is no food when the baking tray is placed on different layers (equivalent to the third prior depth value mentioned above).

[0081] Provide prior depth information (equivalent to the prior depth value of all the above food categories) for the categories that the food object segmentation model can detect. For example, for the purple sweet potato category, randomly select 100 purple sweet potatoes from the market and use their average depth as the prior depth information of the purple sweet potato category.

[0082] Physical distance is measured between the RGB camera inside the oven and the center of the heating element (the physical distance in the above embodiment is obtained), thereby obtaining the transformation relationship (R',t') between the RGB camera coordinate system and the heating element center coordinate system.

[0083] Part Two: After the oven leaves the factory and is delivered to the consumer:

[0084] After the user places the food into the oven, the image data captured by the RGB camera (i.e., the first image data) is used to detect the corresponding category of the food according to the food target segmentation model. At the same time, the oven baking layer detection model is called to obtain the current baking tray layer number (i.e., the target layer number). The depth prior information under the empty tray (equivalent to the first prior depth value) is directly extracted according to the food category. The food area segmented by the segmentation model (i.e., the segmented area) is superimposed with the food category prior depth value (non-food areas do not need to be superimposed) to obtain the depth estimation map after the food is placed on the Nth layer of the oven.

[0085] Based on the image data captured by the RGB camera, the depth estimation map after the food is placed on the Nth layer of the oven, and the RGB intrinsic parameter matrix, the 3D coordinate information (i.e., the second three-dimensional coordinates) of all current RGB pixels can be obtained.

[0086] Based on the regions segmented by the food target segmentation model, the 3D coordinates (i.e., third-dimensional coordinates) of the corresponding food in the RGB camera coordinate system can be extracted.

[0087] By combining the pre-measured transformation relationship (R',t'), the 3D coordinates of the food in the coordinate system of the heating element center (i.e., the third coordinate system) can be obtained (i.e., the target 3D coordinates). Thus, Z (the depth distance of the food from the oven heating element) can be directly extracted, and the target value is obtained. The oven mode can then be determined based on the target value.

[0088] In summary, this embodiment uses an RGBD camera and an RGB camera to estimate prior depth information within the RGB camera's field of view before the oven leaves the factory but is delivered to the consumer. The overall technical solution does not increase costs; the RGBD camera only needs to be used once before delivery (throughout the entire process) and does not need to be designed and installed on the oven, thus not increasing oven costs. In other words, prior depth information is obtained without increasing oven costs, which is a significant innovation. Furthermore, after the oven leaves the factory and is delivered to the consumer, the prior depth information for the oven without food (only the baking tray) is combined with the prior depth information for the food to obtain the desired distance between the food and the oven's heating element. Ultimately, this embodiment effectively estimates the distance between the food and the center of the heating element, reducing errors in the oven's automated judgment and improving the overall intelligence level of the oven.

[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0090] This embodiment also provides an oven-based pattern matching device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0091] Figure 4 This is a structural block diagram of an optional oven-based pattern matching device according to an embodiment of this application, the device comprising:

[0092] The first determining module 42 is used to determine the food information of the food in the oven by using first image data acquired by an image acquisition device preset inside the oven; wherein, the food information includes: the category of the food;

[0093] The acquisition module 44 is used to acquire a first prior depth value of the baking tray in the oven where the food is placed, in an empty state; wherein, the first prior depth value is determined in advance by a depth camera;

[0094] The second determining module 46 is used to determine the depth estimation map corresponding to the food ingredient by using the first prior depth value and the second prior depth value corresponding to the category of the food ingredient;

[0095] The third determining module 48 is used to determine the oven mode of the oven using the first image data and the depth estimation map.

[0096] Using the aforementioned device, the food information of the ingredients in the oven is determined by first image data acquired by an image acquisition device pre-installed inside the oven. This food information includes: the category of the food; a first prior depth value of the baking tray containing the food in an empty state; wherein the first prior depth value is pre-determined by a depth camera; a depth estimation map corresponding to the food is determined by the first prior depth value and a second prior depth value corresponding to the food category; and the oven mode is determined by the first image data and the depth estimation map. In other words, by accurately determining the first prior depth value of the empty baking tray and the second prior depth value of the category of the food placed in the oven, and then using the depth estimation map determined based on the first and second prior depth values, and the first image data acquired by the image acquisition device inside the oven, the oven mode that the oven should use can be accurately determined. Therefore, the above technical solution solves the problem in related technologies where ovens cannot accurately estimate the distance between the food and the heating element, leading to an inability to effectively determine the oven mode; and achieves the technical effect of effectively and accurately determining the oven mode.

[0097] In an exemplary embodiment, the first determining module 42 is further configured to: input the first image data into a food ingredient target segmentation model and determine the category of the food ingredient based on the output result of the food ingredient target segmentation model; input the first image data into an oven baking layer detection model and determine the target layer number of the baking tray on which the food ingredient is placed in the oven based on the output result of the oven baking layer detection model; wherein, the food ingredient information further includes: the target layer number; wherein, the training data of the food ingredient target segmentation model is target image data labeled with food ingredient categories, and the training data of the oven baking layer detection model is target image data labeled with baking layer positions; the target image data are image data of different categories of food ingredients placed on different baking trays in the oven.

[0098] In one exemplary embodiment, such as Figure 5 As shown, the device further includes a fourth determining module 50, used to: before obtaining the first prior depth value of the baking tray containing the ingredients in the oven in an empty state: when the baking tray in an empty state is placed on any layer in the oven, determine the pixel coordinates of each corner point on the calibration plate placed in the oven in the second image data, and determine the first three-dimensional coordinates of each corner point in the first coordinate system corresponding to the depth camera through the third image data and the first depth map; determine the third prior depth value through the pixel coordinates and the first three-dimensional coordinates, wherein the third prior depth value is the prior depth value corresponding to the baking tray in an empty state being placed on any layer in the oven.

[0099] In an exemplary embodiment, the fourth determining module 50 is further configured to: determine the relative pose of the depth camera and the image acquisition device using the pixel coordinates and the first three-dimensional coordinates; determine the corner depth value of each corner point in the second coordinate system corresponding to the image acquisition device using the relative pose; and perform bilinear interpolation on the corner depth value to obtain the third prior depth value; wherein the third prior depth value includes: the first prior depth value.

[0100] In an exemplary embodiment, the second determining module 46 is further configured to: obtain the segmented region of the food determined by the food target segmentation model based on the first image data; and superimpose the second prior depth value corresponding to the category of the food with the first prior depth value in the segmented region to obtain the depth estimation map, wherein the prior depth values ​​of all categories of food that the food target segmentation model can identify are determined by the average depth of all categories of food.

[0101] In an exemplary embodiment, the third determining module 48 is further configured to: determine the second three-dimensional coordinates of each pixel in the first image data using the first image data and the depth estimation map; determine the third three-dimensional coordinates of the food ingredient in the second coordinate system corresponding to the image acquisition device using the second three-dimensional coordinates and the segmented region of the food ingredient; wherein the segmented region is determined based on the first image data using a food ingredient target segmentation model; and determine the oven mode using the third three-dimensional coordinates of the food ingredient and the center of the heating tube of the oven.

[0102] In an exemplary embodiment, the third determining module 48 is further configured to: obtain the transformation relationship between the second coordinate system corresponding to the image acquisition device and the third coordinate system corresponding to the center of the heating tube of the oven, wherein the transformation relationship is determined by the physical distance between the image acquisition device and the center of the heating tube; convert the third three-dimensional coordinates of the food into the target three-dimensional coordinates of the food in the third coordinate system corresponding to the center of the heating tube through the transformation relationship; and determine the oven mode through the target three-dimensional coordinates.

[0103] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0104] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0105] S1, determine the food information of the food in the oven by using the first image data acquired by the image acquisition device preset in the oven; wherein, the food information includes: the category of the food;

[0106] S2, obtain the first prior depth value of the baking tray in the oven where the ingredients are placed in the oven in an empty state; wherein, the first prior depth value is determined in advance by a depth camera;

[0107] S3, determine the depth estimation map corresponding to the food ingredient by using the first prior depth value and the second prior depth value corresponding to the category of the food ingredient;

[0108] S4, determine the oven mode of the oven using the first image data and the depth estimation map.

[0109] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0110] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0111] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0112] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0113] S1, determine the food information of the food in the oven by using the first image data acquired by the image acquisition device preset in the oven; wherein, the food information includes: the category of the food;

[0114] S2, obtain the first prior depth value of the baking tray in the oven where the ingredients are placed in the oven in an empty state; wherein, the first prior depth value is determined in advance by a depth camera;

[0115] S3, determine the depth estimation map corresponding to the food ingredient by using the first prior depth value and the second prior depth value corresponding to the category of the food ingredient;

[0116] S4, determine the oven mode of the oven using the first image data and the depth estimation map.

[0117] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0118] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0119] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0120] Embodiments of this application also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0121] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0122] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0123] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A pattern matching method based on an oven, characterized in that, include: The food information of the food in the oven is determined by first image data acquired by an image acquisition device pre-installed in the oven; wherein, the food information includes: the category of the food; Obtain a first prior depth value of the baking tray containing the food in the oven when it is empty; wherein, the first prior depth value is determined in advance by a depth camera; The depth estimation map corresponding to the food ingredient is determined by the first prior depth value and the second prior depth value corresponding to the category of the food ingredient; The oven mode of the oven is determined using the first image data and the depth estimation map.

2. The oven-based pattern matching method according to claim 1, characterized in that, The food information of the ingredients in the oven is determined by first image data acquired by an image acquisition device pre-installed inside the oven, including: The first image data is input into the food target segmentation model, and the category of the food is determined by the output of the food target segmentation model; The first image data is input into the oven baking layer detection model, and the target layer number of the baking tray on which the food is placed is determined in the oven based on the output of the oven baking layer detection model; wherein, the food information further includes: the target layer number; The training data for the food ingredient segmentation model consists of target image data labeled with food ingredient categories, and the training data for the oven baking layer detection model consists of target image data labeled with baking layer positions. The target image data consists of image data of different types of food ingredients placed on different baking trays in the oven.

3. The oven-based pattern matching method according to claim 1, characterized in that, Before obtaining the first prior depth value of the baking tray containing the ingredients in the oven when the tray is empty, the method further includes: When an empty baking tray is placed on any layer inside the oven, the pixel coordinates of each corner point on the calibration plate placed inside the oven in the second image data are determined, and the first three-dimensional coordinates of each corner point in the first coordinate system corresponding to the depth camera are determined by the third image data and the first depth map. A third prior depth value is determined by the pixel coordinates and the first three-dimensional coordinates, wherein the third prior depth value is the prior depth value corresponding to the baking tray being placed on any layer in the oven when the baking tray is in an empty state.

4. The oven-based pattern matching method according to claim 3, characterized in that, Determining the third prior depth value using the pixel coordinates and the first three-dimensional coordinates includes: The relative pose of the depth camera and the image acquisition device is determined by the pixel coordinates and the first three-dimensional coordinates. The relative pose is used to determine the corner depth value of each corner point in the second coordinate system corresponding to the image acquisition device; The corner depth value is bilinearly interpolated to obtain the third prior depth value; wherein the third prior depth value includes the first prior depth value.

5. The oven-based pattern matching method according to claim 1, characterized in that, Determining the depth estimation map corresponding to the food ingredient by using the first prior depth value and the second prior depth value corresponding to the food ingredient category includes: Obtain the segmented region of the food determined by the food target segmentation model based on the first image data; In the segmented region, the second prior depth value corresponding to the category of the food ingredient is superimposed with the first prior depth value to obtain the depth estimation map. The prior depth values ​​of all categories of food ingredients that the food ingredient target segmentation model can identify are determined by the average depth of all categories of food ingredients.

6. The oven-based pattern matching method according to claim 1, characterized in that, Determining the oven mode of the oven using the first image data and the depth estimation map includes: The second three-dimensional coordinates of each pixel in the first image data are determined using the first image data and the depth estimation map; The third three-dimensional coordinates of the food ingredient in the second coordinate system corresponding to the image acquisition device are determined by the second three-dimensional coordinates and the segmented region of the food ingredient; wherein, the segmented region is determined by the food ingredient target segmentation model based on the first image data; The oven mode is determined by the third-dimensional coordinates of the food ingredients and the center of the oven's heating element.

7. The oven-based pattern matching method according to claim 6, characterized in that, The oven mode is determined by the third-dimensional coordinates of the food ingredients and the center of the oven's heating element, including: The transformation relationship between the second coordinate system corresponding to the image acquisition device and the third coordinate system corresponding to the center of the heating tube is obtained, wherein the transformation relationship is determined by the physical distance between the image acquisition device and the center of the heating tube; The transformation relationship is used to convert the third three-dimensional coordinates of the food ingredient into the target three-dimensional coordinates of the food ingredient in the third coordinate system corresponding to the center of the heating tube. The oven mode is determined by the target's three-dimensional coordinates.

8. A pattern matching device based on an oven, characterized in that, include: The first determining module is used to determine the food information of the food in the oven by using first image data acquired by an image acquisition device preset inside the oven; wherein, the food information includes: the category of the food; The acquisition module is used to acquire a first prior depth value of the baking tray in the oven where the food is placed, in an empty state; wherein, the first prior depth value is determined in advance by a depth camera; The second determining module is used to determine the depth estimation map corresponding to the food ingredient by using the first prior depth value and the second prior depth value corresponding to the category of the food ingredient; The third determining module is used to determine the oven mode of the oven using the first image data and the depth estimation map.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 7 through the computer program.

Images