Steaming oven automatic cooking method and system based on machine vision

Abstract

The invention discloses an automatic cooking method and system of a steaming oven based on machine vision, relates to the field of steaming ovens, and aims to solve the problems that an existing steaming oven lacks the capability of automatically identifying food material characteristics and intelligently adjusting cooking parameters, so that the operation is tedious and the cooking effect is unstable. The method comprises the following steps: acquiring an image of a food material in a cavity of the steaming oven; the collected images are preprocessed; the image is input to a food material identification and classification module, a YOLO algorithm is used to carry out preliminary food material target detection, a ShuffleNet algorithm is used to carry out fine classification on a detection result, and food material category information is generated; a menu matched with the recognized food materials is searched, and recommended menu information is reflected to an interaction screen for a user to select; and according to the selection of the user and based on the volume and the characteristics of the food materials, adjusting the temperature, the time and the cooking mode, and starting cooking. According to the technical scheme, automatic food material recognition and cooking parameter adjustment based on machine vision are achieved, the use process of the steaming oven is simplified, and the cooking efficiency and quality are improved.

Description

Technical Field

[0001] The present invention relates to the field of steam ovens, and particularly to an automatic cooking method and system for steam ovens based on machine vision. Background Art

[0002] With the continuous improvement of modern families' demands for cooking convenience and intelligence, intelligent household appliances are increasingly widely used in the kitchen. Although traditional steam ovens can provide certain automation functions, users still need to manually select cooking modes, times, and temperatures, and the operation steps are cumbersome. Especially for users who are not familiar with the characteristics of ingredients, incorrect operations are likely to occur. This manual adjustment not only increases the cognitive burden on users but also easily leads to overcooking or undercooking of ingredients due to improper settings, affecting the quality and taste of food. In addition, due to the large differences in the cooking requirements of different ingredients, existing steam ovens are difficult to automatically adjust cooking parameters according to the characteristics of ingredients, resulting in unstable cooking effects.

[0003] In order to improve the cooking experience of users and increase cooking efficiency, there is an urgent need for a system that can intelligently identify ingredients, recommend the best recipes, and automatically adjust cooking parameters. Summary of the Invention

[0004] The technical problem to be solved and the technical task proposed by the present invention are to improve and refine the existing technical solutions, and provide an automatic cooking method and system for steam ovens based on machine vision, so as to achieve automatic ingredient recognition and cooking parameter adjustment based on machine vision, with the aim of simplifying the use process of steam ovens and improving cooking efficiency and quality. To this end, the present invention adopts the following technical solutions.

[0005] An automatic cooking method for steam ovens based on machine vision includes the following steps: 1) Collect images of ingredients in the steam oven cavity through an optical camera at the top inside the steam oven; 2) Preprocess the collected images, and the preprocessing includes denoising, color adjustment, and size normalization; 3) Input the preprocessed images into the ingredient recognition and classification module, and the ingredient recognition and classification module uses the YOLO algorithm to perform preliminary ingredient target detection, and uses the ShuffleNet algorithm to perform fine classification on the detection results to generate ingredient category information; 4) According to the recognized ingredient category information, search for recipes that match the recognized ingredients from the established recipe database to obtain recommended cooking recipes, and reflect the recommended recipe information on the interactive screen for users to select; 5) According to the user's selection and based on the volume and characteristics of the ingredients, the steam oven adjusts the temperature, time, and cooking mode according to the target recipe and starts cooking.

[0006] Using an optical camera combined with advanced YOLO algorithm and ShuffleNet algorithm to achieve precise identification and classification of ingredients. The YOLO algorithm can quickly perform global detection and preliminary localization of ingredient targets, and ShuffleNet can carefully distinguish ingredient categories on this basis. The two cooperate to accurately identify a variety of ingredients. Even ingredients with similar appearances (such as different types of green vegetables, fish, etc.) can be effectively distinguished, greatly reducing the cumbersome operation of users manually selecting ingredient types and improving the intelligence level of the cooking process.

[0007] Not only based on the ingredient category, but also comprehensively considering the volume and characteristics of the ingredients to dynamically adjust the cooking temperature, time, and mode. For the same ingredient of different sizes, it can automatically adapt appropriate cooking parameters to avoid overcooking or undercooking due to the difference in the amount of ingredients, ensuring that each dish can achieve an ideal cooking effect, saving energy and improving food quality.

[0008] The interactive screen displays recommended recipes matched according to the ingredients in real time. Users do not need to have professional cooking knowledge. They only need to simply select their favorite recipes, and the steam oven can automatically complete the subsequent complex cooking parameter setting and startup operations, making the cooking process easy and efficient, especially suitable for people who are not familiar with cooking skills or pursue a convenient life.

[0009] The preprocessing steps of denoising, color adjustment, and size normalization after image acquisition can effectively overcome the influence of factors such as the complex lighting environment inside the steam oven, the placement angle of ingredients, and the shooting distance on the image quality, providing a high-quality image data basis for subsequent precise ingredient identification and classification, and ensuring the stability and reliability of the entire system.

[0010] The established recipe database covers a rich variety of recipes, which can meet the diverse needs of different users for various cooking methods of ingredients. Whether it is home-cooked dishes or special delicacies, corresponding cooking solutions can be quickly found, broadening the applicable scenarios of the steam oven.

[0011] As an optimized technical means: in step 1), the optical camera is used in cooperation with a diffused LED lighting lamp, and the diffused LED lighting lamp is located at the rear of the top of the steam oven, and it is ensured that during the operation of the optical camera, the lighting conditions of the ingredients are stable, avoiding the decline of image acquisition quality caused by uneven light sources or shadows.

[0012] The astigmatic LED lighting lamp is located at the rear of the top of the steam oven and works in conjunction with the optical camera to provide uniform and stable lighting for the ingredients. During the cooking process, the surface details of the ingredients (such as texture, color, etc.) are crucial for accurate ingredient recognition. Stable lighting conditions can ensure that these details are clearly presented in the image, avoiding the masking or distortion of some features of the ingredients due to factors such as insufficient light, excessive light, or shadows. For example, when identifying ingredients with fine surface textures (such as the gills of shiitake mushrooms) or ingredients with similar colors but slight differences (such as different varieties of potatoes), stable lighting helps to better capture these details, thereby improving the accuracy of ingredient recognition.

[0013] Uneven light sources are a common problem in the image acquisition process. It may cause uneven bright and dark areas in the image, making it difficult to identify the true features of the ingredients. By providing uniform light through the astigmatic LED lighting lamp, this unevenness can be reduced, and the interference factors in the image acquisition process can be lowered. For example, when cooking larger ingredients, without uniform lighting, shadows may be generated at the edges or bottoms of the ingredients, affecting the judgment of the overall shape and size of the ingredients. Stable lighting can effectively avoid this situation, ensure the image quality, and provide more reliable image data for subsequent ingredient recognition and classification.

[0014] High-quality image acquisition is the premise for the accurate operation of the subsequent ingredient recognition and classification module. The images obtained under stable lighting conditions have characteristics such as color and shape that are closer to the true state of the ingredients, enabling the YOLO algorithm and the ShuffleNet algorithm to function better. For example, when the YOLO algorithm performs preliminary ingredient target detection, it requires clear image boundaries and accurate color information to determine the position and category of the ingredients. The ShuffleNet algorithm also relies on high-quality image features during fine classification. The images acquired under good lighting conditions can provide more accurate input data for these algorithms, thereby improving the accuracy and efficiency of ingredient recognition.

[0015] The reliability of the entire automatic cooking system depends to a large extent on the image acquisition link. The stability of the lighting conditions can reduce the fluctuations in image quality caused by environmental factors (such as changes in light sources), enabling the system to stably acquire high-quality ingredient images in different cooking scenarios and time periods. This helps to enhance the stability of the entire steam oven automatic cooking system based on machine vision, reduce problems such as incorrect ingredient recognition or improper cooking parameter settings caused by image quality issues, and improve the user experience.

[0016] As a preferred technical means: in step 2), the image preprocessing includes: using the histogram equalization method to optimize the brightness distribution of the image; using the Retinex algorithm to reduce the influence caused by light changes, and by separating the brightness information and reflection information in the image to enhance the stability and robustness of the image under different lighting conditions.

[0017] The histogram equalization of this technical solution can improve the brightness distribution of the image. Inside the steam oven, due to factors such as the placement position of the ingredients, the light source angle, and the reflection characteristics of the ingredients themselves, the collected images may have uneven brightness. Through histogram equalization, it will stretch the grayscale histogram of the image, making the distribution of the grayscale values of the pixels in the image more uniform. For example, for an image of ingredients that is overall darker or brighter, after histogram equalization, the dark details will become brighter, and the bright details will also be displayed more clearly, thus enhancing the contrast of the image and making details such as the outline and internal structure of the ingredients more obvious, providing a clearer image for subsequent ingredient recognition.

[0018] The Retinex algorithm effectively reduces the influence of light changes on the image by separating the brightness information and reflection information. In the actual use of the steam oven, the brightness of the light source during the cooking process may change due to various factors (such as power fluctuations of the heating element, interference from other electrical equipment, etc.). The Retinex algorithm can separate the influence of this light change from the image and highlight the reflection characteristics of the ingredients themselves, that is, the true color and texture and other characteristics of the ingredients. For example, when the light intensity inside the steam oven suddenly becomes brighter or darker, for the image processed by the Retinex algorithm, key information such as the color and shape of the ingredients will not deviate greatly due to the light change, enabling the image to maintain relatively stable characteristics under different lighting conditions and improving the image quality.

[0019] The image after optimizing the brightness distribution and enhancing the lighting stability provides more accurate input for the YOLO algorithm and ShuffleNet algorithm in the ingredient recognition and classification module. When the YOLO algorithm performs preliminary ingredient target detection, a clearer brightness distribution and stable image features help accurately locate the bounding box of the ingredient and determine the position of the target. For example, when identifying ingredients with small volume or irregular shapes, a good brightness distribution can avoid misjudgment of the target due to shadows or overly bright areas. The ShuffleNet algorithm also relies on stable image features such as color and texture when performing fine classification of ingredients. The preprocessed image enables it to better extract the fine-grained features of the ingredients, thereby improving the accuracy of ingredient recognition.

[0020] The robustness of the entire automatic cooking system has been improved. Whether in strong light, weak light, or uneven lighting conditions, the preprocessed images can maintain good quality, enabling the system to adapt to more variable lighting environments. In this way, when users use the steam oven at different times and in different cooking scenarios (such as during the day and at night, different kitchen lighting conditions, etc.), the system can stably identify the ingredients, reduce ingredient recognition errors caused by lighting changes, and ensure the smooth progress of the automatic cooking process.

[0021] As a preferred technical means: Step 3) includes the following: 301) Use the YOLO algorithm to perform preliminary detection on the image, divide the image into multiple grids, and evaluate the confidence of each grid to generate multiple bounding boxes; 302) Screen out the areas most likely to contain ingredient targets based on the confidence, and generate candidate ingredient areas to complete the preliminary ingredient detection; 303) Use the ShuffleNet algorithm to perform fine-grained classification on the selected candidate ingredient areas, extract the detailed features of the ingredients, and output the class probabilities of each candidate area; the detailed features include color, shape, and texture for accurate ingredient classification; 304) Select the highest classification result based on the class probabilities, determine the category of the ingredient, and generate the category label and confidence information of the ingredient for subsequent recipe recommendation and automatic adjustment of the cooking process.

[0022] This technical solution divides the image into multiple grids by the YOLO algorithm for detection. This method can achieve a comprehensive coverage analysis of the image. Each grid serves as an independent detection unit, which can effectively capture ingredient targets at different positions in the image. Whether the ingredient is located in the center or edge area of the image, there is a corresponding grid responsible for detection, avoiding the omission of ingredient targets, thus laying a good foundation for accurately identifying ingredients subsequently.

[0023] Evaluate the confidence of each grid and generate bounding boxes, and then screen out the areas most likely to contain ingredient targets based on the confidence. This process can accurately locate the approximate range where the ingredients are located. It can exclude misjudged areas caused by factors such as image noise and similar backgrounds, and only focus on the areas that are truly likely to be ingredients, greatly improving the accuracy and efficiency of ingredient detection, and enabling the subsequent classification work to be carried out in a more targeted area.

[0024] The ShuffleNet algorithm is used to extract detailed features such as color, shape, and texture from the selected candidate food areas. These detailed features are critical for distinguishing ingredients with similar appearances. For example, different types of mushrooms may have similar shapes but subtle differences in texture, or different species of fish have their own characteristics in color and texture. By extracting these fine-grained features, their specific categories can be accurately identified, achieving accurate food classification and avoiding the problem of mismatching cooking recipes due to rough classification.

[0025] The probability of each candidate area is output, and then the highest classification result is selected based on the probability, which provides a quantitative basis for food classification. Compared with simple qualitative judgment, determining categories based on probability is more scientific and accurate, and can fully consider the similarities between different food features and the uncertainty in the recognition process, further improving the reliability of food classification and providing accurate food category information for subsequent recipe recommendations and automatic adjustment of cooking parameters.

[0026] Accurately generated ingredient category labels and confidence information enable the system to quickly and accurately find matching recipes in the huge recipe database. Different ingredients require different cooking methods, time, temperature and other parameters. Accurate ingredient classification ensures that the recipes recommended to users are the most suitable for the ingredients currently placed in the steam oven, which improves the convenience of users obtaining suitable cooking plans, so that users do not need to judge how to cook the ingredients themselves, simplifying the cooking process.

[0027] During the cooking process, based on the accurate food category and corresponding confidence information, the system can automatically adjust the cooking parameters according to the characteristics of the food, such as determining the appropriate baking time and temperature for different meat ingredients according to their categories, or adjusting the steaming time for different vegetables, etc. This not only ensures the cooking effect of the food and achieves the best taste and quality, but also further reduces the cooking failures caused by improper manual parameter settings, and improves the success rate and intelligence of the entire automatic cooking process.

[0028] Whether the ingredients are placed in a messy manner, blocking each other, or the ingredients appear at different angles and sizes in the image, this technical solution can deal with it well by first conducting comprehensive detection and screening, and then extracting fine-grained feature classification. It can accurately identify ingredients from complex image scenes, allowing the system to function stably in various actual cooking scenarios, with strong adaptability, ensuring the reliable implementation of the automatic cooking function of the steam oven based on machine vision.

[0029] This technical solution reduces the result deviation caused by external interferences (such as changes in light, slight blurring of images, etc.), can stably output accurate ingredient recognition results, ensures the smooth progress of subsequent recipe recommendation and cooking adjustment, etc., and improves the stability and robustness of the entire automatic cooking system.

[0030] As a preferred technical means: the ingredient recognition and classification module is obtained through training with a large-scale dataset; during the training process, image data containing different ingredients are collected and classified and labeled to construct an ingredient image dataset. The dataset covers at least X major categories and Y minor categories of ingredients, and the total sample size of the dataset is not less than Z, where X, Y, and Z are set according to common ingredient types and model accuracy requirements.

[0031] This technical solution constructs a dataset by collecting image data covering at least X major categories and Y minor categories of ingredients, which means that the system can learn extremely rich and diverse ingredient appearance features. Different types of ingredients have their own unique manifestations in terms of shape, color, texture, etc. For example, there are huge morphological differences between leafy greens and root vegetables in the vegetable category, and the textures of different parts of meat are also very different. The large-scale and diverse dataset allows the model to fully expose to various features of these different ingredients, so that in actual recognition, it can more accurately judge which category the ingredients in the collected image specifically belong to, reduce the occurrence of misjudgment, and effectively improve the accuracy of ingredient recognition.

[0032] Setting the specific values of X, Y, and Z according to common ingredient types and model accuracy requirements can ensure that the construction of the dataset meets the needs of the actual application scenario. For some scenarios with high requirements for ingredient recognition accuracy, such as professional cooking occasions or home users pursuing high-quality cooking effects, increasing the sample size and the number of ingredient categories covered can make the model training more refined and better distinguish the subtle differences between similar ingredients. For general cooking scenarios, it can also ensure that there are enough samples for the model to learn the main features of common ingredients, so that the model can output relatively accurate ingredient recognition results under different actual usage conditions with different complexities.

[0033] When collecting image data containing different ingredients, different shooting angles, lighting conditions, background environments, etc. are covered. After the model is trained based on such a large-scale dataset, it can learn the commonalities and differences of ingredient features in various changing environments. Therefore, in actual applications, even when the user places ingredients in different placement methods in the steam oven and there are certain fluctuations in the light in the steam oven, etc., the ingredient category can still be accurately recognized, and there will be no problem of recognition failure due to changes in these external environmental factors, greatly enhancing the generalization ability of the model in different image environments.

[0034] With the increasing richness of people's diet choices, new types of ingredients are constantly emerging. Since the constructed large-scale dataset covers a relatively large number of ingredient categories, the model will gradually master the general methods for identifying the characteristics of different ingredients during the learning process. When encountering new ingredients that have not been trained but have similar characteristics, it can also make certain identification judgments based on the learned rules, which helps to better adapt to the constantly updated ingredient types in the market, extend the effective usage cycle of the model, and enable it to cope with the identification challenges brought by new ingredients without frequent retraining.

[0035] Accurate ingredient identification is the basis for realizing precise recipe recommendations. Only when the ingredient identification and classification module can accurately judge the ingredient category can the most suitable cooking recipe for the ingredient be found for the user from the recipe database. The model trained with a large-scale dataset ensures the accuracy of ingredient identification, and further guarantees that the recipes recommended to users are highly matched with the ingredients in terms of cooking methods, time, temperature and other parameters, enabling users to obtain an ideal cooking plan easily without relying on their own experience, and improving the convenience and satisfaction of users when using the steam oven for cooking.

[0036] Based on the accurate ingredient identification results, the steam oven can automatically adjust the cooking parameters for different ingredients, making the cooking process more scientific and reasonable, and ultimately ensuring better food taste and quality. Whether it is common ingredients or some special ingredients, good cooking effects can be achieved through accurate ingredient identification and corresponding parameter adjustment, avoiding the influence on food quality caused by improper cooking parameter settings due to incorrect ingredient identification, and continuously providing users with a stable high-quality cooking experience.

[0037] As an optimal technical means: by means of rotation, flipping, scaling, and brightness change, the image sample size of the training set of the ingredient identification and classification module is increased; The training set also includes ingredient image samples with water vapor and oil stain occlusion in different degrees and positions; these training samples are used to train the deep convolutional neural network and the region proposal network to optimize the identification performance of the model under different occlusion conditions.

[0038] This technical solution increases the number of image samples in the training set through methods such as rotation, flipping, scaling, and brightness change, enabling the model to see the presentation forms of ingredients under various different angles, sizes, and lighting conditions during the training process. For example, the rotated ingredient images show different postures, scaling changes the proportion of the ingredient in the image, and brightness change simulates the visual effects under different lighting environments. After learning from these diverse samples, the model can extract more generalizable features of the ingredients. In practical applications, regardless of the angle at which the user places the ingredient in the steam oven or the natural fluctuations in the internal lighting of the steam oven, the model can stably and accurately identify the ingredient, greatly enhancing its robustness to different image change situations and improving its generalization ability.

[0039] The training set contains image samples of ingredients with water vapor and oil stains at different degrees and positions, allowing the model to fully learn the variation rules of ingredient image features under the interference of occlusion. In the actual use scenario of the steam oven, it is a relatively common situation that the lens is blocked by substances such as water vapor and oil stains. After training with such occlusion samples, the model can learn how to distinguish the key features of the ingredient from the occluded image and accurately judge the ingredient category, rather than being unable to identify due to the existence of occlusion, effectively solving the recognition problem caused by occlusion and enabling it to adapt to complex and variable actual use environments.

[0040] The diverse image samples expand the range of materials for the model to learn, enabling it to learn the essential features of ingredients from more dimensions. Samples after operations such as rotation and flipping can enable the model to comprehensively master the common features of ingredients from different perspectives, samples with brightness change help understand the core features of ingredients under different lighting performances, and for occlusion samples, the model can learn how to capture the key recognition information of the unoccluded area of the ingredient through the occluded part, thereby classifying the ingredient more accurately. For example, for ingredients with similar appearances, through learning a large number of different samples, the model can distinguish them based on more subtle and accurate feature differences, effectively improving the accuracy of ingredient recognition and reducing the occurrence of misjudgment.

[0041] Training the deep convolutional neural network and region proposal network with samples with water vapor and oil stain occlusion optimizes the recognition performance of the model under occlusion conditions. The deep convolutional neural network can better extract the deep features of the occluded ingredient image, and the region proposal network can more accurately locate the regions in the occluded image that may contain ingredients. The two work together, enabling the model to still accurately identify the ingredient category when facing occlusion, further ensuring the accuracy of ingredient recognition under various complex working conditions and improving the overall recognition accuracy.

[0042] This technical solution improves the recognition performance of the model under different conditions, ensuring that the category of ingredients can be accurately identified in both normal unobstructed image acquisition and complex situations such as lens obstruction, thereby providing a reliable basis for subsequent recipe recommendations, automatic adjustment of cooking parameters, etc. In this way, when users use the steam oven to cook, they do not need to worry about unreasonable cooking parameters due to recognition errors, and can enjoy high-quality cooking results stably, improving the stability and satisfaction of the user experience.

[0043] Since the model has been trained to adapt to complex situations such as image changes and occlusions, users do not need to deliberately maintain the ideal state inside the steam oven (such as completely avoiding water vapor and oil stains) during daily use. This reduces many restrictions and concerns during use, making it more convenient and worry-free to use, creating a more relaxed and flexible cooking environment for users.

[0044] As a preferred technical means: the food identification and classification module adopts local saliency detection and attention mechanism based on the color characteristics of dishes to focus on the key color characteristics of the food; and analyzes through the HSV color space, calculates the two-dimensional hue and saturation histogram, and uses the image comparison algorithm to evaluate the similarity between the food color and the standard model, thereby improving the accuracy of food identification.

[0045] This technical solution uses local saliency detection and attention mechanism, which enables the system to focus on the key color features of ingredients in a targeted manner. In actual food images, there are often many interference factors, such as background color, changes in light and shadow on the surface of ingredients, etc. Through this mechanism, the discriminative color features of the ingredients themselves can be highlighted, and irrelevant visual information can be filtered out. For example, when identifying leafy green vegetables, it is possible to focus on the specific green hue and color distribution of the leaves without being disturbed by the color of the surrounding steam oven cavity or other non-critical color elements, thereby more accurately capturing the color features that play a key role in distinguishing ingredients, effectively improving the accuracy of recognition.

[0046] Compared with the traditional RGB color space, the HSV color space is more in line with the way humans perceive color and has unique advantages in describing color characteristics. By analyzing in the HSV color space and calculating the two-dimensional hue and saturation histogram, the color characteristics of the ingredients can be more delicately portrayed. For example, for fruits of different maturity, there will be obvious differences in hue and saturation, which can be more clearly reflected in the HSV space. In this way, the system can dig out more detailed information about the color of the ingredients, which is richer and more distinguishable, laying a good foundation for the subsequent accurate identification of the ingredients.

[0047] The similarity between the color of food ingredients and the standard model is evaluated using an image comparison algorithm, which is particularly important for differentiating food ingredients that are similar in appearance but have subtle color differences. Many food ingredients may be similar in shape and size, such as Chinese cabbage and Shanghai green vegetables. It is difficult to accurately distinguish them only from their appearance. However, they have slight differences in aspects such as color tone and saturation. By calculating the color similarity through this technical solution, the system can keenly detect these differences, and then accurately determine which specific food ingredient it is, greatly enhancing the ability to distinguish similar food ingredients and avoiding inappropriate recipe recommendations or incorrect cooking parameter settings caused by misjudgment.

[0048] Accurate food ingredient recognition is a prerequisite for achieving reasonable recipe matching. By improving the accuracy of food ingredient recognition, the system can more accurately screen cooking recipes suitable for the current food ingredient from the recipe database for users. Different food ingredients have specific requirements for cooking methods, time, temperature and other parameters. Only by accurately identifying the food ingredient can the recommended recipe better match the characteristics of the food ingredient, ensuring that the taste and quality of the cooked food reach an ideal state, enabling users to select recipes without relying on their own experience, simplifying the cooking process, and enhancing the convenience and success rate of cooking.

[0049] Based on accurate food ingredient recognition, the steam oven can accurately set and automatically adjust cooking parameters according to the characteristics of the food ingredient. For example, according to the identified specific food ingredient and its accurate category, reasonable parameters such as baking temperature and time are set to avoid improper cooking parameters caused by incorrect food ingredient recognition, which in turn affects the food quality. In this way, it can continuously provide users with stable high-quality cooking effects and enhance users' satisfaction with the automatic cooking function of the steam oven.

[0050] Different food ingredients have their own unique color characteristics. This technical solution is based on color characteristics for recognition and can adapt to a wide variety of food ingredient types. Moreover, whether under different lighting conditions or in the face of situations such as changes in the placement position and angle of the food ingredient, color characteristics are relatively stable and recognizable factors. Therefore, this technical solution can still maintain good food ingredient recognition ability in a variety of complex actual usage environments, enhancing the adaptability and versatility of the entire automatic cooking system and ensuring its reliable operation in various scenarios.

[0051] As a preferred technical means: when the collected food ingredient image has problems such as blurring or occlusion, the food ingredient recognition and classification module locates the boundary of the blurred area in the image through the Canny edge detection algorithm to clarify the affected area; subsequently, an adapted deblurring technique is used to estimate the blur kernel and repair the blurred area through deconvolution operations to improve the image clarity, ensuring that the outline and details of the food ingredient can be effectively recognized.

[0052] This technical solution uses the Canny edge detection algorithm to locate the boundaries of blurred regions in images. This operation can accurately determine which parts of the image are difficult to clearly present the relevant information of the ingredients due to blurring or occlusion. In the actual scenario of collecting images of steam ovens, the image may be blurred due to reasons such as water vapor, oil stains blocking the lens, or the movement of the ingredients themselves. By accurately finding the blurred regions through this algorithm, it is like delimiting the "key attention scope" for subsequent restoration work, avoiding unnecessary operations on the originally clear and non-processed parts of the image, thus more targeted to solve the problem and laying a foundation for accurate ingredient recognition.

[0053] Adopting an appropriate deblurring technique, performing blurred kernel estimation and restoring the blurred region through deconvolution operations can effectively improve the clarity of the image. A clear image is crucial for ingredient recognition. Details such as the contours and textures of ingredients are more easily captured and analyzed in a clear image. For example, a clear ingredient contour can help determine its shape category, and texture details help distinguish similar ingredients (such as different types of mushrooms having differences in texture). By restoring these key information through repairing the blurred region, it greatly improves the ability of the ingredient recognition and classification module to accurately judge the types of ingredients and reduces recognition errors caused by image blurring.

[0054] During the daily use of steam ovens, it is a relatively common and difficult-to-fully-avoid problem that the image is blurred or occluded, such as water vapor generated during cooking condensing on the lens, or part of the ingredients being blocked by other items when placed. This technical solution specifically addresses these interference factors, so that the system will not be unable to work properly due to these common and complex situations, but can automatically take measures to repair the image, ensuring that the entire automatic cooking process of the steam oven based on machine vision can proceed continuously and stably, enhancing the adaptability and robustness of the system in the actual complex environment.

[0055] Accurate ingredient recognition is a prerequisite for subsequent recipe recommendation and automatic adjustment of cooking parameters. By solving the problem of image blurring or occlusion and ensuring that the ingredients can be effectively recognized, the system can accurately match the corresponding recipes according to the recognized ingredient categories and reasonably set parameters such as cooking temperature, time, and mode, avoiding cooking errors caused by incorrect ingredient recognition due to image quality problems, ensuring the smooth progress of the cooking process, and enhancing the user experience of using the automatic cooking function of the steam oven.

[0056] When using the steam oven, users do not need to worry about cooking not being able to proceed normally due to blurred or blocked images, nor do they need to manually deal with these image problems or readjust the placement of ingredients. The system can automatically detect and repair images, so that the entire cooking process can still proceed smoothly according to the automated process, greatly reducing the user's operating burden and troubles during use, allowing users to enjoy the convenience brought by automatic cooking more conveniently and easily, and improving user satisfaction with the product.

[0057] As a preferred technical means: in step 5), the difference between the current volume of the ingredients and the standard volume of the ingredients in the model is determined according to the volume ratio of the ingredients, and the cooking time and temperature are adjusted according to the difference. A smaller volume of ingredients corresponds to a shorter cooking time, and a larger volume of ingredients corresponds to a longer cooking time. At the same time, the cooking temperature is optimized according to the volume ratio of the ingredients. A higher temperature is used for larger ingredients, and a lower temperature is used for smaller ingredients, so as to optimize the cooking effect and output the adjusted cooking parameters.

[0058] The cooking time and appropriate cooking temperature required for ingredients of different volumes are different. By judging the difference between the current volume of ingredients and the standard volume of ingredients, the cooking time and temperature can be adjusted to ensure that the ingredients are cooked just right in the steam oven. For example, if smaller ingredients are cooked according to the conventional length of time and at a higher temperature, they are prone to overcooking, excessive water loss, and poor taste; while larger ingredients will not be fully cooked if the cooking time is too short or the temperature is not high enough. By making corresponding adjustments based on the volume ratio of the ingredients, both small side dishes and large main ingredients can be cooked at the appropriate time and temperature, which maximizes the retention of the taste, texture, and nutrients of the food and improves the overall cooking effect.

[0059] The cooking temperature is optimized according to the volume ratio of the ingredients. For large ingredients, a higher temperature can ensure that the heat is fully transferred to the inside of the ingredients and cooked thoroughly; for small ingredients, a lower temperature can prevent the surface from burning too quickly, allowing the heat to penetrate evenly and ensure uniform overall doneness. This refined temperature and time control can make ingredients of different sizes present the best cooking state. For example, meat can be charred on the outside and tender and juicy on the inside, and vegetables can maintain a suitable crisp or soft taste, thereby significantly improving the quality of the cooked dishes.

[0060] Users do not need to rely on their own experience to estimate the volume of ingredients and manually adjust cooking parameters. The system automatically optimizes the time and temperature according to the actual volume ratio of the ingredients. The entire process does not require manual intervention, which greatly simplifies the cooking process. This reflects the intelligence of the automatic cooking function of the steam oven. Whether it is a professional user who is familiar with cooking skills or a kitchen novice, it can be easily used. Just put the ingredients into the steam oven and select the corresponding recipe. The system can automatically complete the subsequent complex parameter adjustment work, which brings great convenience to users.

[0061] Due to the different density, texture and other characteristics of different types of food, the cooking parameters required for the same volume are also different. This technical solution is adjusted based on the volume ratio of the food, and can adapt to various types of food, whether it is block-shaped root vegetables, sliced ​​meat, or scattered granular food, etc., the cooking parameters can be flexibly optimized according to their actual volume, so that the system can achieve intelligent cooking control when facing a variety of food, and broaden the scope of application of automatic cooking in steam ovens.

[0062] If the cooking parameters are not adjusted according to the volume of the food, it may be necessary to use high temperature for a long time to cook small food, resulting in energy waste. However, according to the technical solution, the cooking time and temperature can be reasonably shortened according to the actual volume of the food, which can effectively avoid unnecessary energy consumption and achieve energy saving. For example, when baking small chicken wings, lowering the temperature and shortening the time can not only bake the chicken wings to perfection, but also reduce the use of electricity or gas.

[0063] Accurately adjust the cooking time to ensure that the ingredients are cooked within the appropriate time, avoiding repeated adjustments or extended cooking cycles due to improper parameters, and improving cooking efficiency. Users can enjoy cooked food faster, especially when preparing multiple dishes or when time is tight. This precise control of cooking time is particularly important, which can help users arrange the cooking process more reasonably and improve the efficiency of overall kitchen work.

[0064] As a preferred technical means: when the food identification and classification module processes food images, if the number of food targets detected in the image does not match the expectation, it will be processed through a preset fault-tolerant mechanism; if no food is detected, or multiple ingredients are detected, the 2-3 targets with the highest confidence levels are selected for subsequent processing by sorting by confidence level, and possible recipes are recommended based on these targets for the user to choose.

[0065] During the actual image acquisition process of the steam oven, due to various complex factors such as the way of placing ingredients, light reflection, and partial occlusion of some ingredients, it is very likely that the number of detected ingredient targets does not match the expectation. Through a preset fault tolerance mechanism, the system can properly handle these unexpected situations and will not be completely unable to work normally due to momentary detection deviations. For example, when the ingredients are stacked in a messy manner, resulting in some ingredients being occluded and the situation of detecting multiple ingredients or not detecting ingredients occurs, the system can still rely on the fault tolerance mechanism to continue running, ensuring the stability of the entire automatic cooking process of the steam oven based on machine vision and enabling it to function reliably in complex and variable actual usage scenarios.

[0066] Without such a fault tolerance mechanism, once the number of ingredient targets does not meet the expectation, the entire automatic cooking process may be forced to interrupt, requiring the user to manually intervene to troubleshoot problems, re-operate, etc., which brings great inconvenience to the user. And this fault tolerance mechanism reduces the risk of system interruption caused by abnormal detection results, enabling the system to more robustly handle various possible image recognition problems and enhancing the fault tolerance ability and overall robustness of the system.

[0067] Sort by confidence level and select the 2 - 3 targets with the highest confidence levels for subsequent processing, and recommend possible recipes based on these targets for the user to choose. This provides the user with more possibilities for cooking choices. Sometimes, due to the complexity of image recognition, there may be certain uncertainties. However, by presenting the recipes corresponding to several more likely ingredient options, the user can select according to their actual ingredient situation or cooking preferences, avoiding the embarrassing situation of having no suitable recipes available due to inaccurate single recognition results and improving the flexibility and satisfaction of the user when using the automatic cooking function of the steam oven.

[0068] Even when the number of ingredient targets is abnormal, the user does not need to spend a lot of time on cumbersome operations such as re - arranging the ingredients and adjusting the steam oven settings. The system automatically recommends possible recipes based on the fault tolerance mechanism, and the user only needs to make a simple selection from the recommended recipes to continue the cooking process, greatly simplifying the operation process, saving the user's time and energy, and allowing the user to experience a more convenient and intelligent cooking experience.

[0069] When faced with less-than-ideal image recognition results, instead of directly discarding the data, it filters out the more likely food ingredient targets based on confidence levels, and makes the best use of the existing recognition information to drive subsequent steps such as recipe recommendations. This approach fully takes into account the reality that image recognition in practical applications is difficult to achieve 100% accuracy. By reasonably utilizing the limited and somewhat uncertain information, it maximizes the role of the food ingredient recognition and classification module, enabling the entire automatic cooking system to output relatively reasonable results in various complex recognition scenarios, and improving the practicality and value of the food ingredient recognition step in the actual cooking process.

[0070] Another object of the present invention is to provide an automatic cooking system for a steam oven based on machine vision. The automatic cooking system for a steam oven includes: An image acquisition module: used to acquire images of food ingredients in the steam oven cavity through an optical camera at the top inside the steam oven; An image preprocessing module: used to preprocess the acquired images, including denoising, color adjustment, size normalization, and optimizing the brightness distribution of the images using the histogram equalization method, and adopting the Retinex algorithm to reduce the influence brought by light changes, so as to enhance the stability of the images under different lighting conditions; A food ingredient recognition and classification module: used to receive the preprocessed images and perform food ingredient recognition and classification, including: a preliminary detection unit, used to divide the image into multiple grids using the YOLO algorithm, evaluate the confidence level of each grid, generate multiple bounding boxes, screen out the areas most likely to contain food ingredient targets according to the confidence level, and generate candidate food ingredient areas; a fine classification unit, used to perform fine-grained classification on the selected candidate food ingredient areas using the ShuffleNet algorithm, extract the detailed features of the food ingredients, including color, shape, and texture, and output the class probability of each candidate area, select the highest classification result according to the class probability, determine the category of the food ingredient, and generate the category label and confidence information of the food ingredient; an image restoration unit, when the acquired food ingredient image has problems such as blurring or occlusion, locates the boundary of the blurred area in the image through the Canny edge detection algorithm, estimates the blur kernel using an appropriate deblurring technique, and restores the blurred area through deconvolution operations; A recipe recommendation module: according to the food ingredient category information output by the food ingredient recognition and classification module, searches for recipes matching the recognized food ingredients in the recipe database, and displays the recommended recipes on the interactive screen for the user to select; A cooking parameter adjustment module: automatically adjusts the temperature, time, and cooking mode of the steam oven according to the selected recipe and the volume and characteristics of the food ingredients, and starts the cooking process.

[0071] The image acquisition module can obtain the images of the ingredients inside the steam oven cavity. Subsequently, the subsequent image preprocessing module, through operations such as denoising, color adjustment, size normalization, histogram equalization, and Retinex algorithm, effectively overcomes the influence brought by the complex environment inside the steam oven, such as reducing noise interference, optimizing brightness distribution, and reducing the impact of light changes, providing a high-quality image basis for ingredient recognition and classification. High-quality images help to more accurately capture various features of the ingredients, avoid recognition errors caused by poor image quality, and improve the accuracy of ingredient recognition.

[0072] The preliminary detection unit in the ingredient recognition and classification module uses the YOLO algorithm for comprehensive and targeted preliminary detection. Through grid division and confidence evaluation, candidate ingredient regions are screened out. Then, the fine-grained classification unit uses the ShuffleNet algorithm to extract detailed features such as color, shape, and texture for fine-grained classification, determine the ingredient category, and generate corresponding labels and confidence information. This hierarchical detection and classification process can give full play to the advantages of different algorithms, accurately distinguish ingredients with similar appearances, and effectively capture even subtle feature differences, greatly improving the accuracy of ingredient recognition and classification, and providing a reliable basis for the subsequent cooking process.

[0073] The image restoration unit plays a role when there are problems such as blurring or occlusion in the ingredient images. It locates the boundaries of the blurred areas through the Canny edge detection algorithm and then uses an appropriate deblurring technique to restore the blurred areas. In the actual use of the steam oven, situations such as water vapor, oil stains blocking the lens or the movement of ingredients causing image blurring often occur. The existence of this unit ensures that the system will not interrupt the ingredient recognition process due to these common image problems, and guarantees that ingredients can still be continuously and stably recognized under complex image conditions, enhancing the ability of the entire system to cope with actual complex usage scenarios.

[0074] Based on the results of ingredient recognition and classification, the recipe recommendation module quickly and accurately searches for matching recipes from a rich recipe database and displays them on the interactive screen for the user. Users do not need to have professional cooking knowledge or judge the suitable cooking methods for the ingredients themselves. They only need to easily select the recommended recipes, which greatly simplifies the cooking process and makes cooking more convenient, especially suitable for kitchen novices or busy housewives / husbands and other groups.

[0075] The cooking parameter adjustment module will automatically adjust the temperature, time, and cooking mode of the steam oven according to the selected recipe by the user and the volume and characteristics of the ingredients and start cooking. This process does not require the user to manually set complex parameters, avoiding problems caused by the user's unfamiliarity with ingredient characteristics or lack of cooking experience in setting cooking parameters incorrectly. It not only ensures that the food can achieve the ideal cooking effect but also further improves the convenience and satisfaction of the user using the steam oven for automatic cooking.

[0076] Adjust the cooking parameters considering the volume and characteristics of different ingredients, so that regardless of the differences in the size and texture of the ingredients, appropriate cooking conditions can be obtained. For example, for larger ingredients, appropriately extend the cooking time and increase the temperature, and for small, easily cooked ingredients, correspondingly shorten the time and lower the temperature, ensuring that the ingredients are thoroughly cooked and have a good taste, avoiding overcooking or undercooking, stably outputting high-quality cooking effects, and enhancing the user's trust in the automatic cooking function of the steam oven.

[0077] The entire system ranges from image preprocessing to handle light changes, to image restoration to deal with blurring and occlusion problems, to precise ingredient recognition and classification, and subsequent parameter adjustment. Each link cooperates with each other, enabling it to adapt to the working requirements of the steam oven under various complex conditions such as different usage times (e.g., day and night with different lighting), different usage states (e.g., different lens cleanliness levels and different ingredient placement situations), etc. It has strong versatility and adaptability and can be widely applied to different scenarios such as various household kitchens or professional cooking places.

[0078] Beneficial effects: This technical solution can not only capture ingredient images in real time through a camera and intelligently identify the types of ingredients, but also accurately distinguish ingredients with similar appearances, especially in terms of detailed features (such as shape, color, and texture). By combining local saliency detection and attention mechanisms, the recognition accuracy is significantly improved. After identifying the ingredients, the system will intelligently recommend the most suitable recipes and automatically adjust cooking parameters (such as time, temperature, and mode), optimize according to the characteristics of the ingredients, avoid the risk of improper user settings, ensure the best cooking effects for each dish, and greatly improve the user experience and food quality. Description of the Drawings

[0079] Figure 1 It is a schematic structural diagram of the integrated stove steam oven hardware based on machine vision of the present invention; Figure 2 It is a schematic flow diagram of the present invention.

[0080] Description of the Drawings: 1. Steam oven cavity; 2. Optical camera; 3. Temperature sensor; 4. Heat-resistant anti-scattering LED lamp; 5. Ingredient recognition and classification processing module; 6. Interactive screen; 7. System terminal; 8. Steam oven baking tray. Detailed Description of the Invention

[0081] The technical solution of the present invention will be further described in detail below with reference to the drawings in the specification.

[0082] Example 1: Refer to Figure 1 , the first embodiment of the present invention provides a structure of an integrated stove steam oven hardware based on machine vision, including: The steam oven cavity 1 is located below the integrated stove. In this embodiment, the steam oven cavity serves as the hardware carrier for ingredient recognition and classification. Only when the user places the ingredients in the baking tray of the steam oven cavity can recipe recommendations for ingredient recognition be made.

[0083] The optical camera 2 is located in front of the top of the steam oven cavity 1. From the top-down perspective, it can more clearly capture the shape, size, and distribution of the ingredients. It is used to detect and collect the ingredients placed by the user in the steam oven cavity 1, and the collected ingredient images are transmitted to the ingredient recognition and classification processing module 5.

[0084] The ingredient recognition and classification processing module 5 receives the ingredient images collected by the optical camera, recognizes and classifies the ingredient images, recommends ingredient recipes through the system terminal, and the interactive screen 6 displays the recommended recipes.

[0085] The temperature sensor 3 is located on one side inside the steam oven cavity 1, and it continuously detects the temperature during the cooking process of the steam oven. After receiving the cooking temperature parameter information of the target recipe, it automatically detects and adjusts the temperature change inside the steam oven cavity 1 to meet the cooking temperature requirements.

[0086] The heat-resistant anti-glare LED 4 is located at the rear of the top of the steam oven cavity 1, and it is used to ensure stable lighting conditions during the operation of the optical camera 2 and to observe the cooking situation of the ingredients in a timely manner after the user places the ingredients.

[0087] The interactive screen 6 is located on the top of the integrated stove and is used to control the relevant parameters of the integrated stove cooktop, blower, and steam oven. After the ingredients are placed in the steam oven, the ingredient-related recipes recommended by the optical camera 2 through the ingredient recognition and classification processing module and the system terminal are displayed on the interactive screen 6. After the user selects the target cooking recipe, the steam oven will cook based on the target cooking parameters, so that the user does not need cooking knowledge.

[0088] Embodiment 2: An automatic cooking method for a steam oven based on machine vision provided by an embodiment of the present invention; Its process is as Figure 2 shown. The method is applied to the integrated stove steam oven. The method uses a machine vision depth algorithm to be connected with the integrated stove steam oven. By collecting the ingredients in the steam oven, the recommended recipes matching the target ingredients are determined, so that the user no longer needs cooking knowledge matching the ingredients. The user only needs to select the recipe to complete automatic cooking. The method can specifically include the following steps: S1: Collect images of the ingredients in the steam oven cavity.

[0089] In this embodiment, an image acquisition device installed at the front of the top inside the steam oven cavity, specifically an optical camera, is used to obtain the image of the target food ingredient in order to comprehensively acquire the image information of the target food ingredient. Therefore, the target image in this embodiment is an image or real-time video of the target food ingredient placed on the baking tray inside the steam oven cavity.

[0090] After the user activates the steam oven of the integrated stove, the food ingredient is placed inside the steam oven and the oven door is closed. Then the steam oven automatically activates the optical camera and the diffused LED light to collect the pictures of the food ingredient inside the steam oven cavity.

[0091] S2: Preprocess the collected image. The preprocessing includes denoising, color adjustment, and size normalization.

[0092] S3: Identify the type of the food ingredient picture according to the trained food ingredient recognition and classification processing module.

[0093] After receiving the target food ingredient picture collected by the optical camera, the food ingredient classification and recognition processing module identifies the type of the collected target food ingredient picture according to the food ingredient classification and recognition processing algorithm.

[0094] The food ingredient classification and recognition processing module inputs the image collected by the camera into the YOLO model. YOLO will divide the image into S×S grids, and each grid is independently responsible for detecting whether there is a food ingredient target in this area.

[0095] The YOLO model generates one or more bounding boxes for the food ingredient target in each grid and calculates the confidence rate of each bounding box.

[0096] The YOLO model outputs the confidence rate of each bounding box, which is calculated by the following formula: Confidence rate = P (Object) × IOU (predicted box, ground truth box) The final confidence rate multiplied by the class probability represents the possibility that the object in this detection box belongs to a specific class:

[0097] YOLO predicts multiple bounding boxes in each grid and calculates the confidence of the bounding boxes, indicating whether these boxes contain the target object. At the same time, YOLO makes a preliminary classification for each detected target to obtain the class label.

[0098] The shooting of the dish image is often affected by different lighting environments and plate materials, which may cause changes in the color, texture, and shape of the dish. In order to improve the robustness of the target detection and classification recognition algorithm, it must have strong generalization ability and be able to accurately identify the dish under different lighting conditions, different plate materials, and shapes.

[0099] By performing illumination preprocessing on the input image, methods such as histogram equalization and Retinex are used to reduce the impact of illumination changes. In particular, using the Retinex algorithm (based on the color reflection model) can effectively separate the brightness information and reflection information in the image, thereby improving the stability under different illuminations.

[0100] Train using a large-scale and diverse training dataset, including samples of different plate materials and different illumination conditions. Through transfer learning, transfer features from existing large-scale datasets (such as ImageNet), enabling the model to adapt to different environmental changes in practical applications. Data augmentation methods, such as random rotation, scaling, brightness change, etc., can simulate various changes in different environments.

[0101] Furthermore, during the steaming and baking process or in the usage process, the lens may be blocked by substances such as water vapor and oil stains, resulting in the loss or distortion of visual information. To solve this problem, an image processing and recognition algorithm with anti-interference ability must be designed to ensure that the system can still efficiently identify ingredients under damaged visual conditions.

[0102] Adopt an edge detection algorithm (such as Canny edge detection) to detect the blurred areas in the image, and then use deblurring technology for restoration. This technology can improve the blurred areas caused by water vapor or oil stains and enhance the clarity of the image.

[0103] Adopt a deep convolutional neural network (CNN) and a region proposal network (RPN) to detect the occluded areas in the image. During the training process, train by collecting samples with water vapor and oil stain occlusions, enabling the model to identify the occluded areas.

[0104] Furthermore, receive the ingredient region image from YOLO detection, and after preprocessing, input it into ShuffleNet. ShuffleNet efficiently extracts fine-grained ingredient features through group convolution and channel shuffling techniques. Through the fully connected layer and Softmax classification, output the probability distribution of each category, representing the specific classification result and confidence of the ingredient. Judge the reliability of the classification result according to the confidence and threshold, and output the final classification result.

[0105] Furthermore, by combining local saliency detection and attention mechanism, improve the recognition accuracy of ingredients in the case of similar appearances. The system calculates the hue (H) and saturation (S) histograms of the ingredient region, and uses an image comparison algorithm (such as cv::compareHist) to evaluate the similarity between the ingredient and a pre-set standard model. In this way, ingredients can be further accurately distinguished according to color features and the accuracy of accurate recognition can be improved.

[0106] The ShuffleNet finally outputs the probability distribution of the ingredients belonging to each category, and selects the category with the highest probability as the classification result of the ingredients. For example, the model may output a probability of 0.75 for the "leg" category, indicating a 75% confidence that the ingredient is a leg in poultry. According to the confidence threshold set by the system, the system will determine whether the classification result is credible. In the present invention, if the confidence is higher than the set threshold of 0.8, the classification result is considered reliable and the category is finally output.

[0107] The results processed by the ingredient recognition and classification processing module will be sorted according to the ingredient recognition confidence rate, and the target ingredient with the highest confidence rate will proceed to the next step.

[0108] Store the targets with a confidence greater than 0.8.

[0109] The ingredient recognition and classification processing module displays the top three with higher confidence rates to solve the problem that users cannot select other ingredients when there is an ingredient recognition error.

[0110] S4: Recommend relevant recipes in the recipe recommendation system according to the recognized ingredient types.

[0111] In the system terminal, a recipe database matching all the classified ingredients in the training dataset is established. This database contains detailed information on various ingredients and their corresponding recipes, including the ingredients required for each dish, cooking time, temperature, mode (such as steaming, baking, or a combination of steaming and baking), nutritional value, etc. Each recipe is also associated with one or more ingredient categories for identification and matching.

[0112] Each recipe is encoded by the labels and classifications of the required ingredients. The ingredient labels can be transformed into vectors and combined with the ingredient list of each recipe.

[0113] Furthermore, calculate the similarity between the recognized ingredient feature vector and the vector of the ingredients required in the recipe.

[0114] According to the similarity calculation result, select the recipes with a higher matching degree to the current ingredient. A threshold can be set or the Top-N recipes can be selected.

[0115] The algorithm for this step is as follows: Determine whether there are recognized ingredient vectors and recipe vectors, or the feature vectors of the ingredients input by the user. If so, calculate the similarity (cosine similarity) between the ingredients and the recipes, sort the recommended recipes in descending order of similarity, and output the recommended results.

[0116] Each recipe is bound to a specific ingredient category. By mapping the recognized target category of the ingredient to the recipe database in the database, the recipes that match the recognized ingredient can be found, and 2-3 recipes for the target ingredient can be recommended for the user to choose from.

[0117] Reflect the recipe information on the interactive screen according to the relevant recipes of the recommendation system.

[0118] The system terminal extracts the recommended recipe information from the recipe database, including content such as time, temperature, mode, ingredients, etc.

[0119] S5: The user selects the target recipe recommended.

[0120] On the main interface of the interactive screen, the user can see the thumbnails and brief information of multiple recommended recipes. By swiping and selecting, the user can confirm the recipe and choose to start cooking. The system allows the user to adjust the time and temperature parameters.

[0121] According to the target recipe selected by the user, the steam oven performs corresponding time, temperature, and cooking mode settings and starts cooking.

[0122] In this step, to ensure that the ingredients can achieve the best cooking effect at different volumes, thereby improving the quality and taste of the food, the comparison of the ingredient volume and the dynamic adjustment of the parameters can be added; specifically, it includes the following steps: calculate the proportion of the dish in the picture; volume ratio comparison and cooking parameter adjustment; adjust the cooking time according to the volume ratio: if it is small-volume ingredients, the cooking time is shortened; if it is large-volume ingredients, the cooking time is extended; and for small-volume ingredients, the temperature is slightly lower; for large-volume ingredients, the temperature is slightly higher; otherwise, cook normally.

[0123] When the user selects the recommended recipe, the system will transfer the extracted parameters (time, temperature, mode) to the automatic control module of the steam oven.

[0124] The system sends the temperature parameter to the temperature controller of the steam oven, and adjusts the heating element through the internal sensor to make the cavity temperature reach the temperature specified in the recipe.

[0125] The system sets the time parameter to the timer module of the steam oven to ensure that the steam oven completes the cooking task within the specified time.

[0126] According to the mode specified in the recipe, the system will automatically select the steaming, baking or combination mode and start the relevant control module.

[0127] The recipe contains multiple cooking stages (for example, steaming first and then baking). The system will automatically switch the cooking mode after the end of the first stage. For example, after completing the steaming, the system will switch to the baking mode and automatically adjust the temperature and time.

[0128] The integrated range hood steam oven system based on machine vision of the present invention provides users with a brand-new intelligent cooking experience by combining deep learning algorithms with automated cooking technologies. The system uses an optical camera to collect images of the ingredients in the steam oven in real time, and uses the YOLO model and ShuffleNet algorithm to accurately identify and classify the ingredients. The identified ingredients will be cooked according to the recipe recommendations preset by the system, without the need for users to have cooking knowledge, thus effectively reducing the operation complexity and cognitive load of users.

[0129] The working process of the system includes multiple key steps: First, the ingredients are collected by the camera, then classified and identified using machine vision algorithms, and finally the recommended recipes are displayed on the interactive screen according to the identification results. Users only need to select a recipe, and the steam oven will automatically adjust the temperature and cooking parameters to complete the entire cooking process. Hardware devices such as temperature sensors and heat-resistant anti-scattering LED lights ensure a good cooking environment, ensuring the stability of image acquisition and the controllability of the cooking effect of the ingredients.

[0130] The present invention also introduces a number of technical optimizations. For example, under different lighting conditions, the Retinex algorithm is used for image lighting preprocessing to improve the robustness of target recognition. At the same time, for the situation where the image is blocked by water vapor, oil stains, etc., deblurring technology and deep neural networks are used to restore the damaged image, thus ensuring the stability and efficiency of the system in various environments. In addition, the system also combines the dynamic adjustment function of the ingredient volume. By calculating the proportion of the ingredient in the picture, the cooking time and effect are further optimized, improving the taste and quality of the food.

[0131] Generally speaking, the present invention not only greatly simplifies the cooking process, but also realizes a cooking experience that combines automation and intelligence through accurate ingredient recognition and intelligent cooking parameter adjustment, providing users with a more convenient and comfortable kitchen solution.

[0132] Embodiment 3: Provide an automatic cooking system for a steam oven based on machine vision. The automatic cooking system for a steam oven includes: An image acquisition module: used to collect images of the ingredients in the steam oven cavity through an optical camera at the top inside the steam oven; An image preprocessing module: used to preprocess the collected images, including denoising, color adjustment, size normalization, and optimizing the brightness distribution of the images using the histogram equalization method, and using the Retinex algorithm to reduce the influence of lighting changes to enhance the stability of the images under different lighting conditions; Ingredient Recognition and Classification Module: Used to receive the preprocessed image and perform ingredient recognition and classification, including: A preliminary detection unit, which uses the YOLO algorithm to divide the image into multiple grids, evaluate the confidence of each grid, generate multiple bounding boxes, screen out the regions most likely to contain ingredient targets according to the confidence, and generate candidate ingredient regions; A fine-grained classification unit, which uses the ShuffleNet algorithm to perform fine-grained classification on the selected candidate ingredient regions, extract the detailed features of the ingredients, including color, shape, and texture, and output the class probabilities of each candidate region. Select the highest classification result according to the class probabilities to determine the category of the ingredient, generate the category label and confidence information of the ingredient; An image restoration unit, when the collected ingredient image has problems such as blurring or occlusion, locates the boundary of the blurred region in the image through the Canny edge detection algorithm, estimates the blur kernel using an appropriate deblurring technique, and repairs the blurred region through deconvolution operations. Recipe Recommendation Module: According to the ingredient category information output by the ingredient recognition and classification module, search for recipes that match the recognized ingredients in the recipe database, and display the recommended recipes on the interactive screen for the user to select. Cooking Parameter Adjustment Module: Automatically adjusts the temperature, time, and cooking mode of the steam oven according to the recipe selected by the user and the volume and characteristics of the ingredients, and starts the cooking process.

[0133] The image acquisition module can obtain the ingredient image inside the steam oven cavity. The subsequent image preprocessing module effectively overcomes the influence brought by the complex environment inside the steam oven through operations such as denoising, color adjustment, size normalization, histogram equalization, and Retinex algorithm. For example, it reduces noise interference, optimizes the brightness distribution, and reduces the influence of light changes, providing a high-quality image basis for ingredient recognition and classification. High-quality images help to more accurately capture various features of the ingredients, avoid recognition errors caused by poor image quality, and improve the accuracy of ingredient recognition.

[0134] The preliminary detection unit in the ingredient recognition and classification module uses the YOLO algorithm for comprehensive and targeted preliminary detection. It screens out candidate ingredient regions through grid division and confidence evaluation. Then, the fine-grained classification unit uses the ShuffleNet algorithm to extract detailed features such as color, shape, and texture for fine-grained classification, determines the ingredient category, and generates the corresponding label and confidence information. This hierarchical detection and classification process can give full play to the advantages of different algorithms, accurately distinguish ingredients with similar appearances, and effectively capture even subtle feature differences, greatly improving the accuracy of ingredient recognition and classification, and providing a reliable basis for subsequent cooking steps.

[0135] The image restoration unit plays a role when facing problems such as blurring or occlusion in the food ingredient image. It locates the boundary of the blurred area through the Canny edge detection algorithm and then uses an appropriate deblurring technique to restore the blurred area. In the actual use of the steam oven, situations such as water vapor, oil stains obscuring the lens or the movement of food ingredients causing image blurring often occur. The existence of this unit ensures that the system will not interrupt the food ingredient recognition process due to these common image problems, guaranteeing continuous and stable food ingredient recognition under complex image conditions and enhancing the ability of the entire system to handle actual complex usage scenarios.

[0136] Based on the results of food ingredient recognition and classification, the recipe recommendation module quickly and accurately searches for matching recipes from a rich recipe database and displays them to the user on the interactive screen. Users do not need to have professional cooking knowledge or judge the suitable cooking methods for the ingredients themselves. They only need to easily select the recommended recipes, greatly simplifying the cooking process and making cooking more convenient, especially suitable for kitchen novices or busy housewives / husbands and other groups.

[0137] The cooking parameter adjustment module automatically adjusts the temperature, time, and cooking mode of the steam oven according to the selected recipe by the user and the volume and characteristics of the food ingredients and then starts cooking. This process does not require the user to manually set complex parameters, avoiding problems caused by improper cooking parameter settings due to the user's unfamiliarity with the ingredient characteristics or lack of cooking experience. It not only ensures that the food can achieve the ideal cooking effect but also further improves the convenience and satisfaction of the user using the steam oven for automatic cooking.

[0138] Adjusting the cooking parameters considering the volume and characteristics of different food ingredients enables appropriate cooking conditions to be obtained regardless of the differences in the size and texture of the food ingredients. For example, for larger food ingredients, the cooking time is appropriately extended and the temperature is increased, while for small and easily cooked food ingredients, the time is correspondingly shortened and the temperature is decreased to ensure that the food ingredients are cooked through and have a good taste, avoiding situations such as overcooking or undercooking, stably outputting high-quality cooking effects, and enhancing the user's trust in the automatic cooking function of the steam oven.

[0139] The entire system, from image preprocessing to handle light changes, to image restoration to handle blurring and occlusion problems, to accurate food ingredient recognition and classification, and subsequent parameter adjustment, all links cooperate with each other, enabling it to adapt to the working requirements of the steam oven under various complex conditions such as different usage times (such as day and night with different lighting), different usage states (such as different lens cleaning degrees and different food ingredient placement situations), etc. It has strong versatility and adaptability and can be widely applied to different scenarios such as various household kitchens or professional cooking places.

[0140] It can be understood that the detailed function implementation of the above modules and units can be referred to the introduction in the foregoing method embodiments and will not be elaborated here.

[0141] The automatic cooking method and system of a steam oven based on machine vision shown above are specific embodiments of the present invention, which have reflected the substantial features and progress of the present invention. According to actual usage needs, equivalent modifications can be made to its shape, structure, etc. under the inspiration of the present invention, and all of them are within the protection scope of this solution.

Claims

1. A steam oven automatic cooking method based on machine vision, characterized in that The following steps are involved: 1) The optical camera on the top of the steam oven collects images of the food in the steam oven cavity; 2) Preprocessing the collected images, including denoising, color adjustment and size normalization; 3) The preprocessed image is input into the food identification and classification module, which uses the YOLO algorithm to perform preliminary food target detection, and uses the ShuffleNet algorithm to finely classify the detection results to generate food category information; 4) According to the identified ingredient category information, a recipe matching the identified ingredient is searched from an established recipe database to obtain a recommended cooking recipe, and the recommended recipe information is reflected on an interactive screen for the user to select; 5) According to the user's selection and based on the volume and characteristics of the ingredients, the steam oven adjusts the temperature, time and cooking mode according to the target recipe and starts cooking.

2. The automatic cooking method of a steam oven based on machine vision according to claim 1, characterized in that: In step 1), the optical camera is used in conjunction with a diffuse light LED lighting lamp, which is located at the top rear of the steam oven and ensures that the lighting conditions of the food are stable during the operation of the optical camera to avoid a decrease in image acquisition quality due to uneven light sources or shadows.

3. The automatic cooking method of a steam oven based on machine vision according to claim 1, characterized in that: In step 2), image preprocessing includes: using the histogram equalization method to optimize the brightness distribution of the image; using the Retinex algorithm to reduce the impact of lighting changes, and separating the brightness information and reflection information in the image to enhance the stability and robustness of the image under different lighting conditions.

4. The automatic cooking method of a steam oven based on machine vision according to claim 1, characterized in that: Step 3) includes the following: 301) Use the YOLO algorithm to perform preliminary detection on the image, divide the image into multiple grids, and perform confidence assessment on each grid to generate multiple bounding boxes; 302) Filtering out the area most likely to contain the food target according to the confidence level, and generating candidate food areas, thereby completing preliminary food detection; 303) Using the ShuffleNet algorithm to perform fine-grained classification on the selected candidate food regions, extracting detailed features of the food, and outputting the category probability of each candidate region; the detailed features include color, shape, and texture for accurate food classification; 304) Select the highest classification result according to the category probability, determine the category of the ingredient, and generate the category label and confidence information of the ingredient for subsequent recipe recommendation and automatic adjustment of the cooking process.

5. The automatic cooking method of a steam oven based on machine vision according to claim 4, characterized in that: The food identification and classification module is obtained through training of a large-scale data set. During the training process, image data containing different food ingredients are collected and classified and labeled to construct a food image data set, which covers at least X major categories and Y minor categories of food ingredients, and the total sample size of the data set is not less than Z, where X, Y, and Z are set according to common food types and model accuracy requirements.

6. The automatic cooking method of a steam oven based on machine vision according to claim 4, characterized in that: Increase the image sample size of the food identification and classification module training set by methods including rotation, flipping, scaling, and brightness changes; The training set also includes food image samples with different degrees and positions of water vapor and oil stains; These training samples are used to train the deep convolutional neural network and region proposal network to optimize the recognition performance of the model under different occlusion conditions.

7. The automatic cooking method of a steam oven based on machine vision according to claim 4, characterized in that: The food identification and classification module uses local saliency detection and attention mechanism based on the color features of dishes to focus on the key color features of the food; The HSV color space is used for analysis, two-dimensional hue and saturation histograms are calculated, and the image comparison algorithm is used to evaluate the similarity between the color of the food and the standard model, thereby improving the accuracy of food identification. When the collected food images are blurred or occluded, the food recognition and classification module uses the Canny edge detection algorithm to locate the boundaries of the blurred areas in the image and clarify the affected areas. Subsequently, it uses an adapted deblurring technique to estimate the blur kernel and repairs the blurred areas through deconvolution operations to improve image clarity and ensure that the outlines and details of the food can be effectively identified.

8. The automatic cooking method of a steam oven based on machine vision according to claim 4, characterized in that: In step 5), the difference between the current volume of the ingredients and the standard volume of the ingredients in the model is determined based on the volume ratio of the ingredients, and the cooking time and temperature are adjusted based on the difference. A smaller volume of ingredients corresponds to a shorter cooking time, and a larger volume of ingredients corresponds to a longer cooking time. At the same time, the cooking temperature is optimized based on the volume ratio of the ingredients. A higher temperature is used for larger ingredients, and a lower temperature is used for smaller ingredients, so as to optimize the cooking effect and output the adjusted cooking parameters.

9. The automatic cooking method of a steam oven based on machine vision according to claim 7, characterized in that: When the ingredient recognition and classification module processes an ingredient image, if the number of ingredient targets detected in the image does not match expectations, it will be processed through a preset fault-tolerant mechanism; if no ingredient is detected, or if multiple ingredients are detected, the 2-3 targets with the highest confidence levels will be selected for subsequent processing by confidence level, and possible recipes will be recommended based on these targets for the user to choose from.

10. A steam oven automatic cooking system based on machine vision, characterized in that include: Image acquisition module: used to collect images of food in the steam oven cavity through an optical camera on the top of the steam oven; Image preprocessing module: used to preprocess the collected images, including denoising, color adjustment, size normalization, and use the histogram equalization method to optimize the brightness distribution of the image. The Retinex algorithm is used to reduce the impact of illumination changes to enhance the stability of the image under different illumination conditions. Food identification and classification module: used to receive pre-processed images and perform food identification and classification, including: a preliminary detection unit, used to divide the image into multiple grids using the YOLO algorithm, and perform confidence assessment on each grid, generate multiple bounding boxes, filter out the area most likely to contain food targets according to the confidence, and generate candidate food areas; a fine classification unit, used to perform fine-grained classification of the selected candidate food areas using the ShuffleNet algorithm, extract detailed features of the food, including color, shape, texture, and output the category probability of each candidate area, select the highest classification result according to the category probability, determine the category of the food, and generate the category label and confidence information of the food; an image restoration unit, when the collected food image is blurred or occluded, locates the boundary of the blurred area in the image through the Canny edge detection algorithm, uses the adapted deblurring technology to estimate the blur kernel, and repairs the blurred area through the deconvolution operation; Recipe recommendation module: based on the ingredient category information output by the ingredient identification and classification module, it searches for recipes matching the identified ingredients from the recipe database and displays the recommended recipes on the interactive screen for users to choose from; Cooking parameter adjustment module: automatically adjusts the temperature, time and cooking mode of the steam oven according to the recipe selected by the user and the volume and characteristics of the ingredients, and starts the cooking process.

Images