A visual recognition-based intelligent control method and system for kitchenware

Abstract

The application discloses a kind of based on visual identification's kitchen ware intelligent control method and system, the method includes: based on integrated in kitchen ware integration visual identification device real-time acquisition cooking scene image data;The cooking scene image data collected is corrected and preprocessed, and output standardization image data;Cooking scene exclusive dataset is built, and the cooking scene exclusive dataset is labeled, iteration and optimization processing;The cooking scene exclusive dataset processed is loaded, and the standardization image data is carried out real-time deep learning inference, and output structured recognition result;The structured recognition result is analyzed, generates control instruction, and control instruction is issued to kitchen ware control mainboard.The application is compatible with oil fume, food material dual recognition, after multiple training optimization offline inference, low cost realizes kitchen ware automatic regulation and control and safety early warning.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for kitchen utensils, and in particular to an intelligent control method and system for kitchen utensils based on visual recognition. Background Technology

[0002] With the rapid development of the smart home industry, smart cooking appliances are gradually becoming the mainstream in the market. Consumers' demands for intelligent appliances such as automation, manual operation, and safety warnings continue to increase. However, traditional kitchen appliances still have many shortcomings in intelligent control technology: traditional range hoods rely on manual adjustment, and the accompanying oil fume sensors are easily contaminated by oil stains and have delayed detection, making it impossible to predict smoke-generating actions such as adding food or stir-frying; air fryers, steam ovens, and other appliances require users to manually set the temperature and time based on experience, which can easily lead to problems such as undercooked or burnt food; there is no protection or model optimization for kitchen oil stains, high temperatures, and cluttered backgrounds, resulting in high rates of missed detections and false detections and large inference delays under complex working conditions, making it difficult to achieve offline fully automatic and accurate control of kitchen appliances. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art. This invention provides a method and system for intelligent control of kitchen utensils based on visual recognition, which is compatible with dual recognition of oil fumes and food ingredients. After multiple training optimizations, it performs offline inference to achieve automatic control and safety warning of kitchen utensils at low cost.

[0004] This invention provides a method for intelligent control of kitchen utensils based on visual recognition, the method comprising:

[0005] Real-time acquisition of cooking scene image data based on an integrated visual recognition device on kitchen utensils; The collected cooking scene image data is corrected and preprocessed to output standardized image data; A dataset specifically designed for cooking scenarios is constructed, and the dataset is then labeled, iterated, and optimized. Load the processed cooking scene-specific dataset, perform real-time deep learning inference on the standardized image data, and output structured recognition results; The structured recognition results are analyzed to generate control commands, which are then sent to the kitchen appliance control motherboard.

[0006] Furthermore, the real-time acquisition of cooking scene image data based on the integrated visual recognition device integrated on the kitchenware includes: The acquisition mode of the integrated visual recognition device is matched according to the category of kitchenware, and the imaging verification is completed; The camera is controlled to cover the shooting area based on a matching acquisition mode and continuously acquire image data at a fixed frame rate; The light transmittance of the camera lens is detected in real time, and the recognition weight of the image data is generated synchronously based on the light transmittance.

[0007] Furthermore, the annotation, iteration, and optimization processing of the cooking scenario-specific dataset includes: Separate datasets were built for oil fume recognition and food ingredient recognition. The SAM3 image segmentation model is called to standardize and pre-label all image data in the cooking scene-specific dataset, generating initial bounding boxes and corresponding annotation labels. Review the standardized pre-annotation results and correct any missing, incorrect, or offset target boxes. Further differentiated labeling is set based on the type of the cooking scenario-specific dataset.

[0008] Furthermore, the annotation, iteration, and optimization processing of the cooking scenario-specific dataset also includes: A training framework based on the YOLO-V8-nano lightweight network was built to generate a dataset specifically for cooking scenarios and iteratively optimize the processing model. Perform data augmentation on the image data to be iterated in the dataset; The image data after data augmentation is trained iteratively, and the optimal weight file is output after a preset number of training iterations.

[0009] Furthermore, the training iteration on the data-augmented image data includes: Configure a dynamic loss weighting mechanism during training iterations on image data, including: The distribution of sample numbers for various labeled targets in the training set is statistically analyzed to obtain real-time loss feedback. Based on the real-time loss feedback, the loss weight coefficients of the corresponding target boxes are configured.

[0010] Furthermore, the training iteration on the data-augmented image data also includes: The learning rate value of image data during the training iteration process is dynamically presented using the cosine function change curve, generating the learning rate training gradient change curve. The system monitors the changes in the learning rate training gradient curve in real time. When it is determined that the learning rate training gradient curve is oscillating continuously, the optimizer parameters are adaptively switched to fit the current learning rate range.

[0011] Furthermore, the training iteration on the data-augmented image data also includes: After each iteration, the current model weights are saved, the real-time recognition accuracy and loss value are read, and the real-time recognition accuracy and loss value are compared with a preset threshold. When an overfitting trend occurs after multiple iterations, where the recognition accuracy on the validation set does not improve, the training loss value decreases, and the validation loss value increases, an early stopping decision is triggered. After triggering the early stopping decision, the current iteration process is terminated, and the model weights with the best recognition accuracy on the validation set before the early stopping are selected as the candidate weight file.

[0012] Furthermore, the training iteration on the data-augmented image data also includes: An auxiliary loss constraint branch is set in the P4 output layer of the backbone network in the training framework based on the YOLO-V8-nano lightweight network, and the dynamic weight coefficients of the auxiliary loss constraint branch are configured independently. The auxiliary loss constraint branch is used to apply structured similarity constraints to the image features extracted by the model.

[0013] Furthermore, the loaded and processed cooking scene-specific dataset is used to perform real-time deep learning inference on the standardized image data, and the output structured recognition results include: Read the kitchenware type representation bound to the standardized image data and automatically switch to either the oil fume recognition reasoning mode or the food ingredient recognition reasoning mode. In the oil fume recognition reasoning mode, after simultaneously recognizing the oil fume concentration level, smoke-generating actions and dangerous and abnormal scenarios, the oil fume dimension structured recognition data is output. In the food ingredient recognition and reasoning mode, the food ingredient category, quantity and maturity status are identified simultaneously, and the structured recognition data of the food ingredient dimension is output. By combining the structured recognition data of oil fume dimension and the structured recognition data of food ingredient dimension, the structured recognition result is output.

[0014] This invention also provides a vision-based intelligent control system for kitchen utensils, which is used to implement the aforementioned vision-based intelligent control method for kitchen utensils. The system includes: The data acquisition module is used to acquire cooking scene image data in real time based on an integrated visual recognition device integrated on the kitchenware; The data preprocessing module is used to perform correction preprocessing on the acquired cooking scene image data and output standardized image data. A recognition model building module is used to build a cooking scenario-specific dataset and to annotate, iterate, and optimize the cooking scenario-specific dataset. The structured recognition module is used to load the processed cooking scene-specific dataset, perform real-time deep learning inference on the standardized image data, and output the structured recognition result. A control command generation module is used to analyze the structured recognition results, generate control commands, and send the control commands to the kitchen appliance control motherboard.

[0015] This invention provides a visual recognition-based intelligent control method and system for kitchen utensils. It employs a model compatible with both oil fume and food ingredient recognition and adaptable to various types of kitchen utensils, effectively reducing software and hardware development and production adaptation costs. During the training phase, it integrates optimization techniques such as dynamic loss weighting, cosine annealing learning rate, early stopping to prevent overfitting, backbone network-assisted loss constraints, and multiple data augmentation methods. Combined with category-specific lens protection and adaptive adjustment of light transmission weights, it significantly improves recognition accuracy and generalization ability in complex kitchen scenarios involving oil stains and high temperatures. The device enables local offline low-latency inference, automatically switching recognition modes and outputting structured data. It can automatically adjust the fan speed of range hoods and match cooking parameters with steaming and baking equipment, simultaneously providing safety warnings for overflow, fire, and burning without manual intervention. The overall automation, reliability, and safety of the system are significantly superior to traditional control schemes, achieving fully automatic and precise offline control of kitchen utensils. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the intelligent kitchenware control method based on visual recognition in Embodiment 1 of the present invention; Figure 2 This is a flowchart of the process of acquiring cooking scene image data in Embodiment 1 of the present invention; Figure 3 This is a flowchart illustrating the annotation of a cooking scenario-specific dataset in Embodiment 1 of the present invention; Figure 4 This is a flowchart illustrating the iteration of a cooking scenario-specific dataset in Embodiment 1 of the present invention; Figure 5 This is a flowchart of real-time deep learning inference in Embodiment 1 of the present invention; Figure 6 This is a diagram of the architecture of the intelligent kitchenware control system based on visual recognition in Embodiment 2 of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] In this invention, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of features, figures, steps, behaviors, components, portions or combinations thereof disclosed in this specification, and are not intended to exclude the possibility that one or more other features, figures, steps, behaviors, components, portions or combinations thereof are present or added.

[0020] It should also be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] In traditional intelligent control systems for kitchen appliances, image recognition technology has shortcomings. For example, the oil fume sensor is easily contaminated by kitchen grease, resulting in delayed detection signals and an inability to predict dynamic smoke-generating actions such as adding food or stir-frying. Temperature control parameters are manually set by the user, lacking automatic perception of the food's condition, which can easily lead to undercooked or burnt food. In addition, existing systems have not been specifically optimized for the oil, high temperature, and cluttered background of the kitchen environment, resulting in high false alarm rates, high false alarm rates, and large inference delays in image recognition under complex working conditions, which affects the realization of offline fully automatic and precise control of kitchen appliances.

[0022] If the aforementioned technical problems are not resolved, the reliance of kitchen appliance control systems on human intervention cannot be eliminated, and users will be required to continuously monitor and adjust equipment parameters, increasing operational complexity. The reliability of the system under harsh conditions such as oil stains and high temperatures will be reduced, safety warning functions may fail, and kitchen safety accidents may occur. In addition, insufficient recognition accuracy and response delays will hinder the evolution of kitchen appliances towards intelligence and automation.

[0023] Example 1 Embodiment 1 of the present invention provides a method for intelligent control of kitchen utensils based on visual recognition, the method comprising: Real-time acquisition of cooking scene image data based on an integrated visual recognition device on kitchen utensils; The collected cooking scene image data is corrected and preprocessed to output standardized image data; A dataset specifically designed for cooking scenarios is constructed, and the dataset is then labeled, iterated, and optimized. Load the processed cooking scene-specific dataset, perform real-time deep learning inference on the standardized image data, and output structured recognition results; The structured recognition results are analyzed to generate control commands, which are then sent to the kitchen appliance control motherboard.

[0024] In one optional implementation of this embodiment, such as Figure 1 As shown, Figure 1 The flowchart of the intelligent kitchen utensil control method based on vision recognition in Embodiment 1 of the present invention is shown, including the following steps: S101. Real-time acquisition of cooking scene image data based on an integrated visual recognition device integrated on kitchen utensils; In one optional implementation of this embodiment, such as Figure 2 As shown, Figure 2 The flowchart illustrating the acquisition of cooking scene image data in Embodiment 1 of the present invention is shown, including the following steps: S201. Match the acquisition mode of the integrated visual recognition device according to the category of kitchenware, and complete the image verification; In one optional implementation of this embodiment, different acquisition modes are matched for different types of kitchen appliances, including but not limited to range hoods, air fryers, steam ovens, and microwave ovens. For example, a downward wide-angle lens is configured on the range hood, paired with a wipeable oil-proof glass, and the acquisition mode is telephoto shooting. For steaming and baking appliances such as air fryers, steam ovens, and microwave ovens, an in-cavity short-focus lens is configured, paired with double-layer heat-insulating glass, and the acquisition mode is short-focus shooting.

[0025] Furthermore, after the matching of the perspective recognition device is completed, the imaging verification is carried out. The specific verification process includes: acquiring several blank background reference images, statistically analyzing three indicators in the images: average brightness, edge sharpness, and transmittance, and comparing them with preset thresholds. If all three indicators meet the standards, the imaging verification is completed.

[0026] Specifically, considering that the visual recognition device in this embodiment is used in kitchen utensils, especially range hoods, the lens is prone to being obscured by oil fumes and grease. Therefore, to address the lens blurring problem that may be caused by oil fumes and grease, three indicators are used to represent the image: average brightness, edge sharpness, and transmittance of the blank background reference image, and corresponding verification is performed. If the three indicators meet the standards, it means that the imaging verification is passed.

[0027] S202. Control the camera to cover the shooting area based on the matching acquisition mode, and continuously acquire image data at a fixed frame rate; In one optional implementation of this embodiment, for cameras installed on different types of kitchen utensils, the shooting area coverage is performed based on a matching acquisition mode, mainly including: When shooting in telephoto mode with a range hood, the lens is tilted downwards, and the field of view completely encompasses the two burners on the left and right of the stove, the cookware, and the surrounding operating area, with no blind spots in the image; When using the short-focus shooting mode for steaming and baking appliances, aim the lens at the center of the cavity to capture all the food, paper lining, and charred areas inside.

[0028] Furthermore, after the camera's shooting area coverage is set, the adaptive focal length and exposure parameters are locked, and the original RGB image is captured frame by frame at a preset fixed frame rate.

[0029] S203. Real-time detection of the light transmittance of the camera lens, and synchronous generation of the recognition weight of image data based on the light transmittance.

[0030] In one optional implementation of this embodiment, after completing the camera shooting settings, the pixel grayscale value of the protective glass area in front of the camera lens is calculated based on the captured test image, and the real-time transmittance value is obtained.

[0031] Furthermore, the transmittance is divided into three threshold ranges: high, medium, and low. For low-transmittance oil stains, the weight of pixels in the deep learning inference calculation is automatically reduced to weaken the recognition interference caused by oil stains. For medium-transmittance oil stains, no adjustment is considered. For high-transmittance oil stains, the corresponding weight is increased.

[0032] Through the above technical solution, this application can intelligently optimize the image data acquisition process according to different types of kitchen utensils and complex cooking environments. By matching the acquisition mode and imaging verification, the initial quality of the acquired data and its matching degree with the scene are ensured. Continuous acquisition at a fixed frame rate ensures the stability and real-time performance of the data stream. Most importantly, real-time detection of lens transmittance and generation of recognition weights effectively address the impact of environmental factors such as oil fumes and water vapor on image quality during cooking, reducing the interference of low-quality image data on recognition accuracy. This significantly improves the reliability of subsequent deep learning inference and the accuracy of control commands, avoiding misidentification and misoperation caused by image data quality issues.

[0033] S102. Perform correction and preprocessing on the collected cooking scene image data, and output standardized image data; In one optional implementation of this embodiment, the correction preprocessing includes a three-stage serial processing flow of image cropping, image quality correction, and image enhancement.

[0034] Specifically, the image cropping process includes: after acquiring the original large-size image output by the camera, defining the effective region of interest based on the currently matched kitchen appliance acquisition mode. Specifically, in the range hood mode, redundant images of cabinets and walls that are not in the background of the stove are cropped, leaving only the area where the double stove cookware is captured; in the steam oven mode, invalid pixels of the cavity shell and door frame are cropped, leaving only the image inside the cavity where the food is located, removing irrelevant background interference and reducing the amount of computation required for subsequent model inference.

[0035] Furthermore, the image quality correction includes: compensating for the brightness of areas obscured by oil stains based on lens transmittance parameters generated synchronously with the image; correcting overexposure / underexposure caused by strong light from the stove and high-temperature reflections from the oven through a grayscale equalization algorithm; and correcting image color shifts to restore the true color characteristics of oil fumes and food ingredients, eliminating recognition biases caused by ambient light.

[0036] Furthermore, the image enhancement process includes: using an adaptive contrast enhancement algorithm to strengthen the edge features of targets and improve the image recognition of weak targets such as thin oil fumes, small pieces of food, and burnt marks; and simultaneously performing noise reduction filtering on the image to filter out image noise caused by kitchen water vapor and oil particles.

[0037] Furthermore, the cropped, corrected, and enhanced images are uniformly scaled to the fixed input size for model inference, generating standardized image data with uniform format, size, and image quality, which is then fed into the subsequent deep learning model to complete inference calculations.

[0038] By sequentially performing three-level preprocessing operations—image cropping, image quality correction, and image enhancement—irrelevant and redundant images are eliminated to reduce the model's computational power consumption. Image quality defects caused by oil stains, reflections, and uneven lighting are specifically compensated for. Weak target features such as oil fumes, food, and burnt marks are enhanced, and environmental noise is filtered out. Standardized images with uniform size and image quality are output, effectively reducing the interference of complex kitchen environments on model inference and improving the accuracy and speed of subsequent target recognition.

[0039] S103. Build a dedicated dataset for cooking scenarios, and perform annotation, iteration, and optimization on the dedicated dataset for cooking scenarios; In one optional implementation of this embodiment, a dedicated dataset for cooking scenarios is built by collecting real images of cooking scenarios, and pre-labeling is performed first.

[0040] Specifically, such as Figure 3 As shown, Figure 3 The flowchart illustrating the annotation of a cooking scenario-specific dataset according to Embodiment 1 of the present invention is shown, including the following steps: S301. Construct separate datasets for oil fume recognition and food ingredient recognition; In one optional implementation of this embodiment, considering that the visual recognition device in step S101 is mainly used on range hoods and steam ovens, it is necessary to build corresponding oil fume recognition datasets and food ingredient recognition datasets respectively.

[0041] Specifically, two types of cooking-specific real-world datasets will be independently constructed, with specific requirements including: The oil fume recognition dataset collects images of the stove in all scenarios, including different concentrations of oil fumes, adding food, stir-frying, lifting the lid, overflowing, dry burning, and fire, covering various kitchen lighting conditions such as natural daylight, nighttime overhead light, oil stains, and water vapor interference. The food ingredient recognition dataset collects mainstream food ingredients such as chicken wings, fish, potatoes, and pastries. It includes real-life photos of steaming and baking cavities in various states, such as multiple portions, stacked arrangements, and half-cooked / fully cooked / burnt. It also includes differentiated samples with different cavity sizes, different linings, and different light reflections to ensure the diversity of sample scenarios.

[0042] Furthermore, all images in the dataset are randomly divided into training, validation, and test sets in an 8:1:1 ratio.

[0043] S302. Call the SAM3 image segmentation model to standardize and pre-label all image data in the cooking scene-specific dataset, and generate initial target boxes and corresponding labeling. In an optional implementation of this embodiment, the local deployment and adaptation parameter tuning of the SAM3 image segmentation model are completed, the pre-trained weight file adapted to the target kitchenware in this embodiment is loaded, and the image input resolution is set to match the dataset image size. Import the original images of the two datasets, namely cooking fumes and food ingredients, in batches and perform pixel-level instance segmentation: automatically extract the complete edge contours of the targets in the images and output a high-precision pixel mask without jagged edges; automatically generate rectangular target detection boxes based on the mask, and match them with a preset general target label library to automatically bind basic classification labels to each target box, thus completing the automated pre-labeling of all images in the dataset in batches.

[0044] Specifically, the complete edge contours of the image are extracted, including but not limited to the areas of oil fume dispersion, the outline of the cookware, various food ingredients, scorched spots, overflow liquid, and open flame, and a pixel mask representing the contours is formed.

[0045] Furthermore, based on the complete pixel mask, the coordinates of the minimum bounding rectangle of the target are automatically calculated to generate a standard rectangular target detection box that is adapted to the YOLO-V8-nano training format, and the normalization format of the box coordinates is unified.

[0046] Furthermore, the entire mask image is traversed row by row and column by column to filter out all foreground pixels with a mask gray value of 255. The x-coordinate and y-coordinate of each valid foreground pixel are recorded to form a set of pixel coordinates specific to the target. Numerical comparisons were performed on the coordinates of all foreground pixels of the current target, and four key extrema were extracted: x min : The minimum x-coordinate of all foreground pixels; max : The maximum x-coordinate of all foreground pixels; y min : The minimum value of the y-coordinate of all foreground pixels; max : The maximum value of the ordinate of all foreground pixels; It should be noted that the above four extreme coordinate points constitute the smallest rectangular boundary that can completely enclose the target mask area.

[0047] Furthermore, the rectangle's width w, height h, and center point coordinates are calculated based on the four extreme values: Rectangle width w = x max - x min The rectangle's height h = y max - y min The central x-coordinate cx = (x min + x max ) / 2; Central ordinate cy = y min + y max ) / 2.

[0048] Furthermore, the rectangle width w, height h, and center point coordinates are overnormalized to obtain the annotation format coordinates.

[0049] Specifically, the normalized coordinates, including the center x-coordinate cx, the center y-coordinate cy, the rectangle width w, and the rectangle height h, are combined into a four-element coordinate array, which serves as the detection parameters for a single target bounding box, and is then assigned a label.

[0050] S303. Review the standardized pre-annotation results and correct any missing, incorrect, or offset target boxes. In one optional implementation of this embodiment, if necessary, manual review of the standardized pre-annotation results may be arranged, and non-compliant requests may be corrected.

[0051] Specifically, the revisions include: Complete the missing marker boxes for small targets such as thin oil fumes, small pieces of food, and minor scorched spots; Corrected misclassification labels for food categories, oil fume levels, and hazardous scenarios; adjusted target bounding boxes with excessive positional and size deviations; For complex images that SAM3 cannot automatically recognize, such as those with severe oil stains, highly stacked food, or weak open flames, the target boxes are manually drawn and the corresponding labels are manually entered to ensure that all image annotations are complete and without omissions.

[0052] S304. Further set differentiated labeling based on the type of the cooking scenario-specific dataset.

[0053] In one optional implementation of this embodiment, a dedicated hierarchical labeling system is defined for the two types of datasets.

[0054] Specifically, the oil fume recognition dataset is tagged with three main categories: oil fume concentration level, actions that generate smoke such as adding food / stirring / lifting the lid, and dangerous scenarios such as overflowing / dry burning / fire.

[0055] Furthermore, the food identification dataset sets up four main categories of labels: food type, food region, charred area, and food maturity status.

[0056] It should be noted that all labels must use a standardized naming format.

[0057] Through the above technical solutions, this application effectively addresses the problems of low annotation efficiency and poor accuracy caused by the diversity and complexity of targets in cooking scenarios. By constructing separate datasets for oil fume recognition and food ingredient recognition, data classification management and targeted annotation are achieved, improving the professionalism of the annotation. Standardized pre-annotation using the SAM3 image segmentation model significantly improves the efficiency and consistency of initial annotation. Reviewing and correcting the pre-annotation results ensures the quality and accuracy of the datasets, avoiding misidentification caused by low-quality data. Further differentiated labeling allows the datasets to capture key information such as oil fume concentration, food type, and cooking state more precisely, providing high-quality, high-precision training data for subsequent deep learning models. This significantly improves the recognition accuracy and intelligent control command level of the kitchen appliance intelligent control method in complex cooking scenarios.

[0058] In one optional implementation of this embodiment, after the annotation process is completed, the subsequent iterative optimization process is performed.

[0059] Specifically, such as Figure 4 As shown, Figure 4 The flowchart illustrating the iteration of a cooking scenario-specific dataset according to Embodiment 1 of the present invention is shown, including the following steps: S401. Build a training framework based on the YOLO-V8-nano lightweight network, generate a dataset specifically for cooking scenarios, and iteratively optimize the processing model. In one optional implementation of this embodiment, the YOLO-V8-nano lightweight basic network is selected as the backbone inference structure. This network has a small number of parameters, fast inference speed, and is suitable for local deployment requirements of embedded SOC chips.

[0060] Furthermore, a complete training framework is built by making targeted modifications based on the original network architecture.

[0061] Specifically, the network is initialized by loading publicly available pre-trained weights, adapted to the dual-target detection task of oil fumes and food, and the number of network detection heads and classification channels is adjusted to match all labeled classifications such as oil fumes, food, and dangerous anomalies.

[0062] Furthermore, the configuration training data reading module supports batch loading of previously created labeled images, pixel masks, normalized target boxes, and classification labels. The framework has built-in gradient backpropagation, weight saving, and real-time calculation of validation set accuracy modules to complete the construction of a dedicated iterative training model for cooking scenarios.

[0063] S402. Perform data augmentation processing on the image data to be iterated in the dataset; In one optional implementation of this embodiment, before reading samples in each iteration, multi-dimensional random data augmentation is performed on the input images in batches to expand sample diversity and improve the model's generalization ability.

[0064] Specifically, the data augmentation processing includes rotation, mirroring, scaling, multi-scale, mosaic, mixup, etc.

[0065] S403. Train the image data after data augmentation processing iteratively, and output the optimal weight file after a preset number of training iterations.

[0066] In one optional implementation of this embodiment, the enhanced image and corresponding labeled data are input into the constructed YOLO-V8-nano training model, multiple rounds of iterative training are initiated, and the optimal weight file is output after the iterative training is completed.

[0067] Specifically, several optimization mechanisms need to be configured in iterative training to improve the optimization effect of iterative training.

[0068] In one optional implementation of this embodiment, a dynamic loss weighting mechanism is configured during training iterations of image data, including: The distribution of sample numbers for various labeled targets in the training set is statistically analyzed to obtain real-time loss feedback. Based on the real-time loss feedback, the loss weight coefficients of the corresponding target boxes are configured.

[0069] Specifically, after each round of forward inference in the YOLO-V8-nano model, the sample quantity distribution of different labeled targets such as oil fumes, various ingredients, scorched areas, overflowing pans, and open flames in the training set is statistically analyzed. The real-time loss feedback values ​​corresponding to each type of target are also statistically analyzed. For small sample targets such as niche ingredients with few samples, thin oil fumes, and tiny scorched spots, higher loss weight coefficients are automatically configured, while for large categories of targets such as regular ingredients with sufficient samples and high-concentration oil fumes, lower loss weights are matched.

[0070] Furthermore, during the backpropagation phase of the model, the total loss is obtained by weighting and summing the losses of various objectives according to the assigned differentiated weights, and the network weight parameters are updated based on the weighted total loss.

[0071] By employing a dynamic loss weighting mechanism, the model can adaptively balance the sample imbalance problem of various recognition targets in cooking scenarios, improve the convergence speed of small sample targets such as thin oil fumes, minor charred spots, and niche ingredients, reduce the model's learning bias towards large sample targets, and significantly improve the model's ability to recognize various weak feature small targets.

[0072] In one optional implementation of this embodiment, the learning rate value of image data during the training iteration process is dynamically presented as a cosine function change curve, thereby generating a learning rate training gradient change curve. The system monitors the changes in the learning rate training gradient curve in real time. When it is determined that the learning rate training gradient curve is oscillating continuously, the optimizer parameters are adaptively switched to fit the current learning rate range.

[0073] Specifically, during the model training initialization phase, an initial maximum learning rate and the total number of training iterations are set. Throughout the training cycle, the learning rate dynamically decays according to a cosine function curve. After each iteration, the cosine decay coefficient is calculated based on the current number of completed iterations and the total number of iterations, and the learning rate used for the next round of network parameter updates is updated in real time.

[0074] Furthermore, after using the cosine function curve to dynamically represent the changes in the learning rate training gradient, the results are as follows: the learning rate is relatively large in the early stage of training, which quickly updates the network weights and significantly reduces the model's basic loss; in the middle and later stages of training, the learning rate decreases smoothly and slowly with the cosine curve, allowing the model gradient descent process to converge smoothly and avoiding training oscillations caused by a sudden drop in the learning rate and the model getting stuck in a local optimum.

[0075] It's important to note that the learning rate is a key hyperparameter in deep learning optimization algorithms, determining the step size at which the model updates weights in each iteration. Traditional fixed or linearly decaying learning rates may not adequately adapt to the complex changes in the loss function surface during training. Using a cosine function curve to dynamically adjust the learning rate means that the learning rate decays or changes periodically according to a cosine fluctuation pattern. For example, the learning rate can start from a high initial value, smoothly decrease to a low minimum, and then potentially increase again after a cycle, creating a "warm start" effect. This dynamic adjustment helps the model converge quickly in the early stages of training and finely adjust the weights in the later stages, avoiding getting trapped in local optima.

[0076] Furthermore, after visualizing and monitoring the learning rate training gradient change curve, the trajectory of the learning rate over time or iterations is recorded and visualized by monitoring the changes in the curve. Even with dynamic learning rate scheduling, in some complex or unstable training scenarios, the model may still experience training instability, manifested as continuous, irregular, and drastic fluctuations in the learning rate training gradient change curve (or the resulting loss value, gradient norm, etc.), i.e., "continuous oscillation." Real-time monitoring of this change means that the system continuously evaluates the effectiveness of the learning rate and its impact on model updates. Once continuous oscillation is detected, it indicates that the current learning rate or optimizer configuration may not be suitable for continued stable training. At this point, the system will "adaptively switch optimizer parameters." This can include adjusting hyperparameters within the optimizer, such as momentum terms and decay rates, or directly adjusting the current learning rate to a smaller, more stable "learning rate range" to reduce the step size of each update, thereby suppressing oscillations and helping the model return to a stable convergence path.

[0077] In an optional implementation of this embodiment, after each iteration, the current model weights are saved, the real-time recognition accuracy and loss value are read, and the real-time recognition accuracy and loss value are compared with a preset threshold. Specifically, before starting the model training task, two core threshold parameters are pre-set: one is the "patience" threshold, which represents how many consecutive rounds of validation metrics fail to optimize before stopping; the other is the minimum retention baseline for validation loss, used to record the optimal model loss throughout the entire process. Once these parameters are finalized, they are written into the training framework configuration file and remain fixed throughout the entire training process, without dynamic changes with each iteration. After the model completes one round of forward inference and backward weight updates for augmented data samples, the training process is paused, and independent, non-overlapping validation set images and labeled data are loaded for batch inference. During the inference process, random data augmentation and gradient updates are disabled; only forward prediction is performed to accurately reproduce the model's true recognition performance on unfamiliar samples.

[0078] After all inference on the validation set is completed, the framework automatically calculates two types of core evaluation metrics: the first is the average loss value of the validation set, which is the weighted sum of losses for all categories of oil fumes, food ingredients, and dangerous targets; the second is the model recognition accuracy, which includes two key data points: average precision (mAP) for various targets and recall rate for small targets. The two sets of metrics are then stored in the log file in correspondence with the current iteration round.

[0079] The framework incorporates a built-in metric comparison logic. After each round of verification, the loss and accuracy of this round are compared with the historical best metrics. If the loss of this round is less than the historical minimum loss and the recognition accuracy is higher than the historical highest accuracy, the consecutive invalid round counter is reset, and the global optimal weight backup is updated. If the metrics are not optimized in this round, the consecutive invalid round counter is incremented by 1 to continuously track the decline or stagnation of the metrics.

[0080] Furthermore, after multiple iterations, if an overfitting trend occurs where the recognition accuracy of the validation set does not improve, the training loss value decreases, and the validation loss value increases, an early stopping decision is triggered. Specifically, when the cumulative value of the consecutive invalid round counter equals the preset patience threshold, the system determines that the model has overfitted: the model has excessively memorized unique noise features such as oil stains and reflections in the training set, and its ability to recognize samples from entirely new scenes no longer improves. At this point, the training framework automatically issues an early stop termination command, directly interrupting the remaining unexecuted iterative training process.

[0081] After triggering the early stopping decision, the current iteration process is terminated, and the model weights with the best recognition accuracy on the validation set before the early stopping are selected as the candidate weight file.

[0082] Specifically, after the early stopping mechanism is triggered, the training program reads the entire log record, selects the weight file with the highest recognition accuracy and lowest validation loss in all iteration rounds, saves it separately, and marks it as the optimal model weight; it automatically discards the weight files generated in the stagnation of indicators and the overfitting stage in the consecutive rounds before the early stopping is triggered, and only outputs the weights with the best generalization ability for subsequent equipment deployment.

[0083] Through the above technical solution, this application can effectively solve the overfitting problem that may occur during model training iterations. By monitoring the performance trend of the model on the validation set in real time and combining it with the changes in the training loss value, it is possible to accurately determine when the model begins to overfit the training data. Once an overfitting trend is detected, the system can terminate training in a timely manner, avoiding unnecessary consumption of computational resources and wasted training time. More importantly, this solution ensures that the final selected model weights are optimal in terms of generalization ability, thereby significantly improving the robustness of the model in iterative optimization processing of the cooking scenario-specific dataset and its recognition accuracy in real cooking scenarios, providing a high-quality foundation for subsequent real-time deep learning inference.

[0084] In an optional implementation of this embodiment, an auxiliary loss constraint branch is set in the P4 output layer of the backbone network in the training framework based on the YOLO-V8-nano lightweight network, and the dynamic weight coefficients of the auxiliary loss constraint branch are configured independently. The auxiliary loss constraint branch is used to apply structured similarity constraints to the image features extracted by the model.

[0085] Specifically, an auxiliary branch is added to the back end of the YOLO-V8-nano backbone feature extraction network. This auxiliary branch does not participate in the final detection box and classification result output, but is only used to constrain the feature extraction effect of shallow, medium and deep layers.

[0086] Furthermore, during the training phase, the pixel mask output by SAM3 is synchronously input into the auxiliary branch. The auxiliary branch calculates the pixel-level segmentation loss based on the mask as an auxiliary loss term. After a single forward inference of the model, the total loss is obtained by weighted summation of the main detection loss and the auxiliary segmentation loss. During backpropagation, the auxiliary loss is synchronously propagated back to the backbone network.

[0087] Specifically, the auxiliary loss function is defined as follows:

[0088] In the formula, This represents the total loss value of a single iteration of the model. The main detection task loss item, The weight hyperparameters for the structured similarity auxiliary loss of the P4 layer features are... This is the structured similarity auxiliary loss term for the P4 feature layer of the backbone network. The weight hyperparameters for the pixel segmentation auxiliary loss. This is an auxiliary loss term for pixel-level segmentation.

[0089] Through the aforementioned technical solutions, the model can more effectively learn image features with inherent structural consistency during training, significantly enhancing its ability to understand complex visual information in cooking scenarios. This enables the model to more accurately identify key information such as the concentration level of cooking fumes, subtle changes in cooking actions, and the cooking state of ingredients, thereby improving the precision and reliability of intelligent kitchen appliance control. Simultaneously, dynamically adjusting the weights of the auxiliary loss ensures the stability and efficiency of the training process, avoiding local optima problems that may arise from a single loss function, allowing the model to converge to a state with optimal performance.

[0090] Furthermore, in this embodiment, the visual recognition of kitchen utensils is applied to complex interferences in kitchen scenes, such as oil stains, high temperature reflections, food stacking, small target oil fumes / burnt spots, and uneven sample numbers. Directly using the basic YOLO-V8-nano network for iterative training is prone to defects such as slow convergence of small target samples, the model being prone to getting stuck in local optima, overfitting after multiple iterations, and insufficient feature extraction discrimination under complex conditions. Relying solely on basic data augmentation cannot completely solve the above problems.

[0091] Therefore, in this embodiment, multiple optimization mechanisms are added to the iterative training process, including dynamic loss weighting, cosine annealing learning rate, early stopping judgment, and backbone network auxiliary loss constraints. These mechanisms work together to optimize the training process from multiple dimensions, including loss calculation, learning rate control, training termination strategy, and feature extraction constraints. This balances the convergence speed of various samples, stabilizes the gradient descent process, suppresses overfitting, and enhances the target feature extraction capability under complex interference, thereby improving the model's final recognition accuracy and generalization performance.

[0092] S104. Load the processed cooking scene-specific dataset, perform real-time deep learning inference on the standardized image data, and output structured recognition results. In one optional implementation of this embodiment, such as Figure 5 As shown, Figure 5 The flowchart of real-time deep learning inference in Embodiment 1 of the present invention is shown, including the following steps: S501. Read the kitchenware type representation bound to the standardized image data and automatically switch to the oil fume recognition reasoning mode or the food ingredient recognition reasoning mode. In an optional implementation of this embodiment, the kitchen utensil type identifier field encapsulated and transmitted along with the image data stream is read. This identifier is pre-written in the image acquisition stage according to the kitchen utensil category matched by the hardware, and is divided into two categories: range hood identifier and steam oven identifier. If the corresponding identifier of the range hood is obtained through parsing, the dedicated inference branch for fume recognition will be automatically loaded and the fume recognition inference mode will be switched. If the identifiers of kitchen appliances such as air fryers, steam ovens, and microwave ovens are obtained through parsing, then the food identification reasoning branch is loaded and the food identification reasoning mode is switched; Both modes share the same lightweight YOLO-V8-nano backbone network, only switching the detection head classification channel and post-processing parsing rules, without having to repeatedly load the complete model. The mode switching time is controlled in milliseconds, enabling adaptive switching of recognition tasks.

[0093] S502. In the oil fume recognition and reasoning mode, after simultaneously recognizing the oil fume concentration level, smoke-generating actions and dangerous and abnormal scenarios, the oil fume dimension structured recognition data is output. In one optional implementation of this embodiment, when the system runs the oil fume recognition inference mode, the model performs three types of target detection inference in parallel: First, it distinguishes between low / medium / high oil fume concentration areas and calculates the percentage of pixels covered by oil fume to determine the real-time oil fume concentration level; second, it identifies target boxes of typical smoke-generating actions such as adding vegetables, stir-frying, and lifting the lid, and marks the currently executed cooking action; third, it simultaneously detects dangerous and abnormal targets such as overflowing, dry burning, and open flame fire and outputs the abnormal confidence level; after the inference is completed, the oil fume concentration value, smoke-generating action type, dangerous and abnormal marker, confidence level of each target, and target box coordinates are uniformly encapsulated into a fixed-format oil fume dimension structured array. Each field in the array is divided into independent data bits to form standardized oil fume dimension structured recognition data for output.

[0094] S503. In the food ingredient recognition and reasoning mode, the food ingredient category, quantity and maturity status are simultaneously identified, and the food ingredient dimension structured recognition data is output. In one optional implementation of this embodiment, when the system runs the food ingredient recognition inference mode, the model simultaneously completes multi-dimensional food ingredient feature inference: identifying various food ingredient entities in the image and matching corresponding food ingredient category labels; calculating the total pixel area of ​​the food ingredient based on the SAM mask, and calculating the actual volume of the food ingredient by combining the cavity calibration size, and quantifying the food ingredient weight level; accurately identifying charred patch areas and internal texture features of the food ingredient, and distinguishing between three ripeness states: uncooked, semi-cooked, and fully ripe; integrating and encapsulating the identified food ingredient type, weight quantification value, ripeness state mark, charred area range, and corresponding confidence level to generate well-structured food ingredient dimension recognition data that can be directly parsed.

[0095] S504. Combining the structured recognition data of the oil fume dimension and the structured recognition data of the food ingredient dimension, the structured recognition result is output.

[0096] In one optional implementation of this embodiment, the single-dimensional structured data output by the corresponding inference mode is integrated and encapsulated according to the current operating mode: the oil fume mode retains only the oil fume dimension data and leaves the food-related fields empty, and the food ingredient mode retains only the food ingredient dimension data and leaves the oil fume-related fields empty.

[0097] Furthermore, additional fields such as frame sequence number, device status check code, and data check bit are added to the integrated data to assemble a complete structured identification message that can be recognized by the whole machine. The message is encoded according to the preset serial port / SPI / USB communication protocol, redundant and invalid fields are removed, and the data transmission length is compressed. Finally, the integrated standardized structured identification result is obtained.

[0098] Through the above technical solution, this application can dynamically adjust the focus of deep learning inference according to the actual needs of the cooking scenario, thereby significantly improving the targeting and efficiency of deep learning inference. This ensures comprehensive and accurate acquisition of key information from different cooking dimensions (such as the state of oil fumes and ingredient information), avoiding unnecessary consumption of computing resources. Compared with the basic solution, this adaptive inference mode switching mechanism makes the intelligent control of kitchen utensils more precise and responsive, providing users with a more refined and personalized cooking assistance experience. It effectively solves the problem of insufficient recognition efficiency and accuracy under a single inference mode, thereby improving the practicality and intelligence level of the entire kitchen utensils intelligent control system.

[0099] S105. Analyze the structured recognition results, generate control commands, and send the control commands to the kitchenware control motherboard.

[0100] In an optional implementation of this embodiment, the structured recognition result obtained in step S104 is analyzed to generate corresponding control commands, which are then sent to the kitchenware control motherboard. The kitchenware control motherboard executes the corresponding control operation according to the corresponding control commands.

[0101] In summary, Embodiment 1 of this invention provides a visual recognition-based intelligent control method for kitchen utensils. It employs a model compatible with both oil fume and food ingredient recognition and adaptable to various types of kitchen utensils, effectively reducing software and hardware development and production adaptation costs. During the training phase, it integrates optimization techniques such as dynamic loss weighting, cosine annealing learning rate, early stopping to prevent overfitting, backbone network-assisted loss constraints, and multiple data augmentation methods. Combined with category-specific lens protection and adaptive adjustment of light transmission weights, it significantly improves recognition accuracy and generalization ability in complex kitchen scenarios involving oil stains and high temperatures. The device enables local offline low-latency inference, automatically switching recognition modes and outputting structured data. It can automatically adjust the fan speed of the range hood and match cooking parameters with steaming and baking equipment, simultaneously providing safety warnings for overflow, fire, and burning without manual intervention. The overall automation, reliability, and safety of the machine are significantly superior to traditional control schemes, achieving fully automatic and precise offline control of kitchen utensils.

[0102] Example 2 Embodiment 2 of the present invention provides a smart kitchenware control system based on visual recognition. The smart kitchenware control system based on visual recognition is used to implement the smart kitchenware control method based on visual recognition described in Embodiment 1. The system includes a data acquisition module, a data preprocessing module, a recognition model building module, a structured recognition module, and a control command generation module.

[0103] In one optional implementation of this embodiment, such as Figure 6 As shown, Figure 6 The diagram shows the architecture of a vision-based intelligent kitchenware control system according to Embodiment 2 of the present invention, which includes the following modules: Data acquisition module 10, the data acquisition module 10 is used to acquire cooking scene image data in real time based on an integrated visual recognition device integrated on the kitchenware; In an optional implementation of this embodiment, the real-time acquisition of cooking scene image data based on the integrated visual recognition device integrated on the kitchenware includes: The acquisition mode of the integrated visual recognition device is matched according to the category of kitchenware, and the imaging verification is completed; The camera is controlled to cover the shooting area based on a matching acquisition mode and continuously acquire image data at a fixed frame rate; The light transmittance of the camera lens is detected in real time, and the recognition weight of the image data is generated synchronously based on the light transmittance.

[0104] The data preprocessing module 20 is used to perform correction preprocessing on the acquired cooking scene image data and output standardized image data. The recognition model building module 30 is used to build a cooking scene-specific dataset and to annotate, iterate, and optimize the cooking scene-specific dataset. In an optional implementation of this embodiment, the annotation, iteration, and optimization processing of the cooking scenario-specific dataset includes: Separate datasets were built for oil fume recognition and food ingredient recognition. The SAM3 image segmentation model is called to standardize and pre-label all image data in the cooking scene-specific dataset, generating initial bounding boxes and corresponding annotation labels. Review the standardized pre-annotation results and correct any missing, incorrect, or offset target boxes. Further differentiated labeling is set based on the type of the cooking scenario-specific dataset.

[0105] In an optional implementation of this embodiment, the annotation, iteration, and optimization processing of the cooking scenario-specific dataset further includes: A training framework based on the YOLO-V8-nano lightweight network was built to generate a dataset specifically for cooking scenarios and iteratively optimize the processing model. Perform data augmentation on the image data to be iterated in the dataset; The image data after data augmentation is trained iteratively, and the optimal weight file is output after a preset number of training iterations.

[0106] In an optional implementation of this embodiment, the step of training iterations on the image data after data augmentation includes: Configure a dynamic loss weighting mechanism during training iterations on image data, including: The distribution of sample numbers for various labeled targets in the training set is statistically analyzed to obtain real-time loss feedback. Based on the real-time loss feedback, the loss weight coefficients of the corresponding target boxes are configured.

[0107] In an optional implementation of this embodiment, the step of training iterations on the data-augmented image data further includes: The learning rate value of image data during the training iteration process is dynamically presented using the cosine function change curve, generating the learning rate training gradient change curve. The system monitors the changes in the learning rate training gradient curve in real time. When it is determined that the learning rate training gradient curve is oscillating continuously, the optimizer parameters are adaptively switched to fit the current learning rate range.

[0108] In an optional implementation of this embodiment, the step of training iterations on the data-augmented image data further includes: After each iteration, the current model weights are saved, the real-time recognition accuracy and loss value are read, and the real-time recognition accuracy and loss value are compared with a preset threshold. When an overfitting trend occurs after multiple iterations, where the recognition accuracy on the validation set does not improve, the training loss value decreases, and the validation loss value increases, an early stopping decision is triggered. After triggering the early stopping decision, the current iteration process is terminated, and the model weights with the best recognition accuracy on the validation set before the early stopping are selected as the candidate weight file.

[0109] In an optional implementation of this embodiment, the step of training iterations on the data-augmented image data further includes: An auxiliary loss constraint branch is set in the P4 output layer of the backbone network in the training framework based on the YOLO-V8-nano lightweight network, and the dynamic weight coefficients of the auxiliary loss constraint branch are configured independently. The auxiliary loss constraint branch is used to apply structured similarity constraints to the image features extracted by the model.

[0110] The structured recognition module 40 is used to load the processed cooking scene-specific dataset, perform real-time deep learning inference on the standardized image data, and output the structured recognition result. In one optional implementation of this embodiment, the loaded and processed cooking scene-specific dataset is used to perform real-time deep learning inference on the standardized image data, and the output structured recognition results include: Read the kitchenware type representation bound to the standardized image data and automatically switch to either the oil fume recognition reasoning mode or the food ingredient recognition reasoning mode. In the oil fume recognition reasoning mode, after simultaneously recognizing the oil fume concentration level, smoke-generating actions and dangerous and abnormal scenarios, the oil fume dimension structured recognition data is output. In the food ingredient recognition and reasoning mode, the food ingredient category, quantity and maturity status are identified simultaneously, and the structured recognition data of the food ingredient dimension is output. By combining the structured recognition data of oil fume dimension and the structured recognition data of food ingredient dimension, the structured recognition result is output.

[0111] The control command generation module 50 is used to analyze the structured recognition results, generate control commands, and send the control commands to the kitchenware control motherboard.

[0112] In summary, Embodiment 2 of this invention provides a vision-based intelligent control system for kitchen utensils, used to implement the vision-based intelligent control method for kitchen utensils described in Embodiment 1. It employs a model compatible with both oil fume and food ingredient recognition and adaptable to multiple types of kitchen utensils, effectively reducing software and hardware development and production adaptation costs. During the training phase, it integrates optimization techniques such as dynamic loss weighting, cosine annealing learning rate, early stopping to prevent overfitting, backbone network-assisted loss constraints, and multiple data augmentation methods. Combined with category-specific lens protection and adaptive adjustment of light transmission weights, it significantly improves recognition accuracy and generalization ability in complex kitchen scenarios involving oil stains and high temperatures. The device enables local offline low-latency inference, automatically switching recognition modes and outputting structured data. It can automatically adjust the fan speed of the range hood and match cooking parameters with steaming and baking equipment, simultaneously providing safety warnings for overflow, fire, and burning without manual intervention. The overall automation, reliability, and safety of the system are significantly superior to traditional control schemes, achieving fully automatic and precise offline control of kitchen utensils.

[0113] The above provides a detailed description of a visual recognition-based intelligent control method and system for kitchen utensils provided by the present invention. Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0114] Furthermore, the embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for intelligent control of kitchen utensils based on visual recognition, characterized in that, The method includes: Real-time acquisition of cooking scene image data based on an integrated visual recognition device on kitchen utensils; The collected cooking scene image data is corrected and preprocessed to output standardized image data; A dataset specifically designed for cooking scenarios is constructed, and the dataset is then labeled, iterated, and optimized. Load the processed cooking scene-specific dataset, perform real-time deep learning inference on the standardized image data, and output structured recognition results; The structured recognition results are analyzed to generate control commands, which are then sent to the kitchen appliance control motherboard.

2. The intelligent kitchen appliance control method based on visual recognition as described in claim 1, characterized in that, The real-time acquisition of cooking scene image data based on the integrated visual recognition device integrated on kitchen utensils includes: The acquisition mode of the integrated visual recognition device is matched according to the category of kitchenware, and the imaging verification is completed; The camera is controlled to cover the shooting area based on a matching acquisition mode and continuously acquire image data at a fixed frame rate; The light transmittance of the camera lens is detected in real time, and the recognition weight of the image data is generated synchronously based on the light transmittance.

3. The intelligent kitchen appliance control method based on visual recognition as described in claim 1, characterized in that, The annotation, iteration, and optimization processing of the cooking scenario-specific dataset includes: Separate datasets were built for oil fume recognition and food ingredient recognition. The SAM3 image segmentation model is called to standardize and pre-label all image data in the cooking scene-specific dataset, generating initial bounding boxes and corresponding annotation labels. Review the standardized pre-annotation results and correct any missing, incorrect, or offset bounding boxes. Further differentiated labeling is set based on the type of the cooking scenario-specific dataset.

4. The intelligent kitchen utensil control method based on visual recognition as described in claim 1, characterized in that, The annotation, iteration, and optimization processing of the cooking scenario-specific dataset also includes: A training framework based on the YOLO-V8-nano lightweight network was built to generate a dataset specifically for cooking scenarios and iteratively optimize the processing model. Perform data augmentation on the image data to be iterated in the dataset; The image data after data augmentation is trained iteratively, and the optimal weight file is output after a preset number of training iterations.

5. The intelligent kitchen appliance control method based on visual recognition as described in claim 4, characterized in that, The training iteration of the image data after data augmentation includes: Configure a dynamic loss weighting mechanism during training iterations on image data, including: The distribution of sample numbers for various labeled targets in the training set is statistically analyzed to obtain real-time loss feedback. Based on the real-time loss feedback, the loss weight coefficients of the corresponding target boxes are configured.

6. The intelligent kitchen utensil control method based on visual recognition as described in claim 4, characterized in that, The training iteration of the image data after data augmentation also includes: The learning rate value of image data during the training iteration process is dynamically presented using the cosine function change curve, generating the learning rate training gradient change curve. The system monitors the changes in the learning rate training gradient curve in real time. When it is determined that the learning rate training gradient curve is oscillating continuously, the optimizer parameters are adaptively switched to fit the current learning rate range.

7. The intelligent kitchen utensil control method based on visual recognition as described in claim 4, characterized in that, The training iteration of the image data after data augmentation also includes: After each iteration, the current model weights are saved, the real-time recognition accuracy and loss value are read, and the real-time recognition accuracy and loss value are compared with a preset threshold. When an overfitting trend occurs after multiple iterations, where the recognition accuracy on the validation set does not improve, the training loss value decreases, and the validation loss value increases, an early stopping decision is triggered. After triggering the early stopping decision, the current iteration process is terminated, and the model weights with the best recognition accuracy on the validation set before the early stopping are selected as the candidate weight file.

8. The intelligent kitchen appliance control method based on visual recognition as described in claim 4, characterized in that, The training iteration of the image data after data augmentation also includes: An auxiliary loss constraint branch is set in the P4 output layer of the backbone network in the training framework based on the YOLO-V8-nano lightweight network, and the dynamic weight coefficients of the auxiliary loss constraint branch are configured independently. The auxiliary loss constraint branch is used to apply structured similarity constraints to the image features extracted by the model.

9. The intelligent kitchen appliance control method based on visual recognition as described in claim 1, characterized in that, The loaded and processed cooking scene-specific dataset is used to perform real-time deep learning inference on the standardized image data, and the output structured recognition results include: Read the kitchenware type representation bound to the standardized image data and automatically switch to either the oil fume recognition reasoning mode or the food ingredient recognition reasoning mode. In the oil fume recognition reasoning mode, after simultaneously recognizing the oil fume concentration level, smoke-generating actions and dangerous and abnormal scenarios, the oil fume dimension structured recognition data is output. In the food ingredient recognition and reasoning mode, the food ingredient category, quantity and maturity status are identified simultaneously, and the structured recognition data of the food ingredient dimension is output. By combining the structured recognition data of oil fume dimension and the structured recognition data of food ingredient dimension, the structured recognition results are output.

10. A vision-based intelligent control system for kitchen utensils, characterized in that, The vision-recognition-based intelligent kitchenware control system is used to implement the vision-recognition-based intelligent kitchenware control method according to any one of claims 1-9, the system comprising: The data acquisition module is used to acquire cooking scene image data in real time based on an integrated visual recognition device integrated on the kitchenware; The data preprocessing module is used to perform correction preprocessing on the acquired cooking scene image data and output standardized image data. A recognition model building module is used to build a cooking scenario-specific dataset and to annotate, iterate, and optimize the cooking scenario-specific dataset. The structured recognition module is used to load the processed cooking scene-specific dataset, perform real-time deep learning inference on the standardized image data, and output the structured recognition result. A control command generation module is used to analyze the structured recognition results, generate control commands, and send the control commands to the kitchen appliance control motherboard.

Images