A cooking scene picture generation method and device, a cooker, and a cooker control method
By generating cooking scene images through conditional adversarial generative networks, the problem of insufficient sample size is solved, enabling the construction of a low-cost intelligent cooking dataset and improving the accuracy of recognition and safety warnings.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FOSHAN SHUNDE MIDEA WASHING APPLIANCES MANUFACTURING CO LTD
- Filing Date
- 2021-06-08
- Publication Date
- 2026-07-31
AI Technical Summary
The current technology suffers from insufficient sample size of cooking scene images and high acquisition costs, hindering the breakthrough development of intelligent cooking.
We use a conditional generative adversarial network (CGAN) to generate cooking scene images. We construct a conditional image sample dataset by crawling images of ingredients, pots and pans, and combine deep learning and image recognition technologies to generate a large number of cooking scene images.
It reduces the difficulty and cost of acquiring cooking scene images, provides a large amount of sample data to support the development of smart cooking, and improves the accuracy of cooking-related action recognition and safety warnings.
Smart Images
Figure CN115457289B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent cooking equipment technology. More specifically, this invention can provide a method, apparatus, cooking equipment, and cooking equipment control method for generating cooking scene images. Background Technology
[0002] With the continuous development of society and the economy and the continuous improvement of people's living standards, more and more smart furniture and smart home appliances are appearing in people's lives. At the same time, users' requirements for the quality of home appliances are also increasing, and their demand for the intelligence and comfort of home appliances is becoming stronger. Among them, the smart kitchen is an important link in reflecting the level of intelligence of home appliances, and smart cooking is the core of the smart kitchen. However, due to the complexity of the entire cooking process, the diversity of cooking elements, the diversity of personal habits, and the complexity of the environment, obtaining the original dataset samples required for smart cooking has become extremely difficult, and the cost of obtaining samples is very high, which has prevented smart cooking from making a breakthrough. Summary of the Invention
[0003] The main objective of this invention is to provide a method, apparatus, cookware, and cookware control method for generating cooking scene images, so as to solve the problem of insufficient cooking image samples in the prior art, and achieve the technical objectives of increasing the number of cooking scene image samples and reducing the construction cost of cooking scene image sample sets.
[0004] To address the aforementioned technical problems, this invention provides a method for generating cooking scene images. This method includes, but is not limited to, at least one of the following steps: acquiring a cooking background image and a cooking object image, both of which are images related to a cooking scene; performing feature extraction processing on the cooking background image and the cooking object image respectively, to extract a first feature vector from the cooking background image and a second feature vector from the cooking object image; concatenating the first and second feature vectors to obtain a target feature vector; and finally generating a cooking scene image based on the target feature vector and parameters of a real cooking image, wherein the parameters of the cooking scene image are the same as those of the real cooking image.
[0005] To address the aforementioned technical problems, the present invention also provides a cooking scene image generation device, which may include, but is not limited to, an image acquisition module, a feature extraction module, a vector stitching module, and an image generation module.
[0006] The image acquisition module is used to acquire cooking background images and cooking object images, both of which are related to the cooking scene.
[0007] The feature extraction module is used to perform feature extraction processing on the cooking background image and the cooking object image respectively, so as to extract the first feature vector from the cooking background image and the second feature vector from the cooking object image.
[0008] The vector concatenation module is used to concatenate the first feature vector and the second feature vector to obtain the target feature vector.
[0009] The image generation module is used to generate cooking scene images based on the target feature vector and parameters of real cooking images. The parameters of the cooking scene images are the same as those of the real cooking images, including but not limited to image height, image width, and image channel parameters.
[0010] To address the aforementioned technical problems, the present invention also provides an intelligent cooking appliance, which may include, but is not limited to, the cooking scene image generation device in any embodiment of the present invention.
[0011] To address the aforementioned technical problems, the present invention may also provide a cookware control method, which may include, but is not limited to, the cooking scene image generation method in any embodiment of the present invention, for intelligent control of the cookware based on the generated cooking scene image.
[0012] To address the aforementioned technical problems, the present invention also provides a computer storage medium storing a cooking scene image generation program. When the cooking scene image generation program is executed by a processor, the cooking scene image generation method of any embodiment of the present invention is implemented.
[0013] The beneficial effects of this invention are as follows: This invention greatly reduces the difficulty of acquiring cooking scene images, making it possible to obtain a large number of original dataset samples required for intelligent cooking, and at a very low cost. It is evident that this invention can construct a cooking scene image dataset, providing sample data support for the breakthrough development of intelligent cooking. Therefore, the image generation method of this invention can completely eliminate the pain point of difficult sampling of cooking scene images, and the entire process requires no additional parameters or human judgment factors, making it highly applicable.
[0014] Based on the large number of cooking scene images generated by this invention, the invention can provide users with more accurate recognition of cooking-related actions, various safety warning prompts, and safer and healthier dietary and cooking suggestions, resulting in an excellent user experience. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.
[0016] Figure 1 A flowchart illustrating the method for generating cooking scene images in an embodiment of the present invention is shown.
[0017] Figure 2 A schematic diagram illustrating the working principle of the conditional adversarial generative network in an embodiment of the present invention is shown.
[0018] Figure 3 This illustration shows a schematic diagram of acquiring real cooking images using a range hood with a camera, according to an embodiment of the present invention.
[0019] Figure 4 This illustration shows a schematic diagram of obtaining a cooking scene image through downsampling convolution and upsampling convolution operations in an embodiment of the present invention.
[0020] Figure 5 This diagram illustrates how, in an embodiment of the present invention, features are fused by concatenating the outputs of downsampling convolution (encode) and upsampling convolution (decode).
[0021] Figure 6 The diagram illustrates the generation method of cooking scene images and related applications in an embodiment of the present invention.
[0022] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0024] In view of the limited number of existing cooking scene image samples and the high cost of collecting images of different cooking scenes, this invention provides a cooking scene image generation method, apparatus, cookware, and cookware control method to solve one or more problems existing in the prior art.
[0025] This invention, based on deep learning, big data, and IoT technologies, and utilizing intelligent image detection and recognition methods, enables intelligent cooking and provides a novel interactive experience for smart kitchens. The invention employs a Conditional Generative Adversarial Network (CGAN) to implement an algorithm for generating cooking scene images. Figure 2 The illustrated conditional adversarial generative network may include a generator and a discriminator. The generator includes... Figure 4 The diagram shows a series of networks with downsampling (encode) convolutions and upsampling (decode) convolutions. The discriminator includes a series of fully connected networks and a binary classifier (similar to a 0-1 classifier); in this embodiment, the binary classifier is implemented using the sigmoid (S-shaped growth curve) function.
[0026] This invention utilizes a conditional adversarial generative network to generate a large number of cooking scene images, thereby enriching the cooking scene image dataset. The specific implementation of this invention includes the training and application processes of the conditional adversarial generative network.
[0027] (I) Training of Conditional Adversarial Generative Networks
[0028] (1) Constructing a conditional image sample dataset: In this embodiment of the invention, a large number of food ingredient images, cookware images, and stove images are crawled from the Internet using web crawler software. Frontal images are selected from the cookware and stove images to form a conditional image sample dataset using frontal cookware images, frontal stove images, and food ingredient images. In this invention, cookware and stove images serve together as cooking background images, and food ingredient images are images of the objects being cooked.
[0029] (2) Construct a dataset of real cooking images: such as Figure 3 The camera mounted on the range hood shown captures the entire cooking process. The captured images are saved, and their names can be in the format of "ingredients + pot + stove + number," allowing users to quickly identify the content of the image through its name. In this embodiment, the captured image size is W×H×3, where W refers to the width of the image, H refers to the height of the image, and 3 refers to the RGB channels of the color image. Therefore, this invention can capture and save images of real cooking scenes.
[0030] (3) Randomly select one food image, one pot image, and one stove image from the conditional image sample dataset, and randomly select one image from the real cooking image sample dataset as the real input sample.
[0031] (4) Input the food images, cookware images and stove images into their respective classification networks, extract feature vectors as condition vectors, and obtain three condition vectors (y1, y2, y3) by extracting features from cooking background images and cooking object images.
[0032] This invention provides a food classifier for distinguishing different ingredients, a cookware classifier for distinguishing different cookware, and a stove classifier for distinguishing different stoves. Specifically, this invention applies a pre-trained deep convolutional neural network based on an ImageNet image dataset and uses fine-tuning to classify the targets (ingredients, cookware, and stoves) to obtain trained food classifiers, cookware classifiers, and stove classifiers.
[0033] Based on ImageNet and FineTune, the food images are classified using a trained food classifier to extract the food classification feature vector y1 from the food classifier.
[0034] Based on ImageNet and FineTune, a pre-trained cookware classifier is used to classify cookware images by feature, so as to extract the cookware classification feature vector y2 from the cookware classifier.
[0035] Based on ImageNet and FineTune, a pre-trained stove classifier is used to classify the stove images by feature, so as to extract the stove classification feature vector y3 from the stove classifier.
[0036] (5) A noise vector z is randomly generated, and this noise vector, along with the three conditional vectors y1, y2, and y3 mentioned above, is input into the generator to produce a synthesized cooking scene image. Specifically, this invention performs vector concatenation processing on the food classification feature vector, cookware classification feature vector, stove classification feature vector, and Gaussian noise signal according to a certain dimension; wherein, the Gaussian noise signal is a random noise signal used by the conditional adversarial generative network, and the generator in the conditional adversarial generative network is used to generate a cooking scene image based on the target feature vector and the parameters of the real cooking image.
[0037] For the downsampling convolution of the generator, vectors z, y1, y2, and y3 are concatenated along their dimensions to form the input layers. This invention performs downsampling convolution on the target feature vector to reduce the feature size and obtain intermediate results; this process may include fully connected convolution operations, matrix transformation (reshape) operations, etc. More specifically, the image parameters obtained after matrix transformation in this embodiment can be... This indicates the width parameter of the currently output image. This represents the height parameter of the currently output image. This represents the current output image channel parameters. In downsampling convolution operations, features are continuously extracted through convolution, and this may also include Batch Normalization operations, ReLU (Remote Activation Function) calculations, etc. The image parameters output after the downsampling convolution operation can be... This represents the width parameter of the downsampled output image. This represents the height parameter of the downsampled output image. This represents the image channel parameters of the downsampled output. In this embodiment, the convolution operation in downsampling can use a 3×3 kernel, and the stride of the convolution operation can be, for example, 2.
[0038] Based on parameters from real cooking images, for those with The intermediate calculation results of the parameters are subjected to upsampling convolution operations to expand the feature size. Each time the feature size is doubled, the number of channels is halved. Finally, the upsampling convolution results are processed by tanh (a non-linear function) to obtain a cooking scene image of size W×H×3 (the same as a real cooking image). In this embodiment, the upsampling convolution operation may include batch regularization, non-linear activation function calculation, etc. A 3×3 convolution kernel can be used in the upsampling convolution operation.
[0039] Optionally, some embodiments of the present invention perform upsampling convolution operations on intermediate computation results by performing bilinear interpolation and convolution operations on the intermediate computation results. Bilinear interpolation and convolution operations can preserve better feature map details and avoid potential edge jaggedness or stacking effects.
[0040] Optionally, in other embodiments of the present invention, upsampling convolution operations on intermediate computation results include: performing deconvolution operations on the intermediate computation results to obtain deconvolution results. For example... Figure 5 As shown, based on the U-Net (semantic segmentation network) architecture, the deconvolution result is fused with the intermediate result. This method enhances semantic information, resulting in higher accuracy of the generated cooking scene images.
[0041] In the process of generating cooking scene images, the present invention achieves this by continuously changing the background of the cooking process through the generator, that is, by continuously changing the pots and stoves, thereby quickly generating cooking images with constantly changing backgrounds and expanding the dataset of intelligent cooking training samples.
[0042] (6) The synthesized cooking images are used as fake samples, and the real cooking images are used as real samples, which are then input into the discriminator of the conditional adversarial generative network. For example... Figure 2As shown, and can be combined Figure 6 This invention uses a discriminator to distinguish generated cooking scene images from images in the current sample image dataset. For example, it compares the generated cooking scene images with real-world samples to determine their authenticity, thereby training the discriminator.
[0043] like Figure 2 As shown, this embodiment uses a binary classifier in the discriminator to determine whether the input sample is real or fake. In specific implementation, based on randomly selected food images, pot images, and stove images, the condition vector (y1, y2, y3) and noise vector z are input into the generator to obtain an image of size W×H×3, denoted as x1. A corresponding image is randomly selected from the current sample image dataset (i.e., the real sample dataset) as the real cooking scene image, denoted as x. x1 and x are respectively input into a pre-trained network, such as a convolutional neural network based on ImageNet. For x1, a feature vector composed of one or more layers of feature fusion is extracted from it. This feature vector can be a one-dimensional vector. This one-dimensional vector is then concatenated with the condition vector (y1, y2, y3) to obtain a one-dimensional vector F with a higher dimension. For x, feature vectors formed by fusing one or more layers of features are extracted, for example, one-dimensional vectors, and concatenated with conditional vectors (y1, y2, y3) to obtain a higher-dimensional one-dimensional vector T. Multiple fully connected nonlinear transformations are then performed on the higher-dimensional one-dimensional vectors F and T. In this embodiment, Leaky ReLU (a nonlinear activation function) is used to implement the nonlinear transformation. Finally, the sigmoid function outputs a value between (0,1), which is used to determine the truth value of x1 and x. In this embodiment, a threshold of 0.5 is set. If the output value is greater than 0.5, the image is considered true; if the output value is less than 0.5, the image is considered false. The purpose of this training process is to enable the discriminator to determine whether x1 is false and x is true.
[0044] (7) The generator and discriminator are trained alternately to bring the entire conditional adversarial generative network to a Nash equilibrium point, ultimately making the cooking scene images generated by the generator almost identical to real cooking images. During the training of the conditional adversarial generative network, the generator and discriminator are trained alternately. The generator is used to replace the cooking images of pots and pans with the background of the stove, and the discriminator is used to judge the similarity between the generated cooking images and real cooking images, guiding the generator to make the generated images increasingly closer to real images.
[0045] The loss function of the conditional adversarial network provided in this embodiment of the invention is as follows:
[0046]
[0047] The optimization objective of this invention is V(D,G), which aims to minimize the probability D(G(z)) that the cooking image generated by G(z) is judged as real, i.e., maximize 1-D(G(z)), and maximize the probability D(x) that the real data x is judged as real. Here, G represents the generator, D represents the discriminator, x represents the real data (real-world collected cooking image data), z represents random noise, G(z) represents the cooking image synthesized by the generator, and y represents the condition vector composed of y1, y2, and y3 (representing the ingredient classification feature vector, cookware classification feature vector, and stove classification feature vector, respectively). It expresses expectation.
[0048] (II) Applications of Conditional Adversarial Generative Networks
[0049] like Figure 6 As shown, the generator of the trained conditional adversarial generative network is extracted and deployed to a device; for example, it is deployed to a home appliance, which may include, but is not limited to, a range hood.
[0050] like Figure 1 As shown, and able to combine Figure 2 , 4 ~6. Embodiments of the present invention provide a method for generating cooking scene images. This method is a cooking scene image generation method based on conditional GAN, and may include, but is not limited to, one or more of the following steps, as detailed below.
[0051] First, the cooking background image and cooking object image are acquired as input to the trained generator. Both the cooking background image and the cooking object image are images related to the cooking scene, such as an image of a wok with cooking oil, an image of a gas stove turned on to "medium heat," and an image of tomatoes cut into chunks. Specifically, acquiring the cooking background image and cooking object image may include: acquiring images of a cookware and a stove, and using these images as the cooking background image; specifically, the invention also involves acquiring images of ingredients and using these images as the cooking object image.
[0052] Secondly, feature extraction processing is performed on the acquired cooking background image and cooking object image respectively, so as to extract the first feature vector from the cooking background image and the second feature vector from the cooking object image.
[0053] Optionally, feature extraction processing is performed on the acquired cooking background images and cooking object images, including: forming cookware classifiers, stove classifiers, and food classifiers based on ImageNet and fine-tuning; classifying the cookware images using the trained cookware classifier to extract cookware classification feature vectors; classifying the stove images using the trained stove classifier to extract stove classifier feature vectors; the first feature vector includes cookware classification feature vectors and stove classification feature vectors; and classifying the food images using the trained food classifier to extract food classification feature vectors; the second feature vector is the food classification feature vector.
[0054] Next, the first feature vector and the second feature vector are concatenated to obtain the target feature vector. Specifically, this concatenation process in this embodiment may include: sequentially concatenating the food category feature vector, the cookware category feature vector, the stove category feature vector, and the Gaussian noise signal; wherein the Gaussian noise signal is the random noise signal used by the trained conditional adversarial generative network, which generates a cooking scene image based on the target feature vector and parameters of the real cooking image.
[0055] Finally, a cooking scene image is generated based on the target feature vector and the parameters of the real cooking image. This cooking scene image is then placed into a real cooking image sample dataset to enrich the real sample dataset. The image parameters in this embodiment include, but are not limited to, image height, image width, and image channel parameters. Specifically, this invention can transform the pots and stoves in the input cooking scene image to generate a large number of cooking scene images with constantly changing backgrounds. Generating a cooking scene image based on the target feature vector and the parameters of the real cooking image includes: performing a downsampling convolution operation on the target feature vector to obtain an intermediate operation result. The downsampling convolution operation process in this embodiment may include, but is not limited to, fully connected convolution operations, matrix transformation operations, batch regularization operations, and nonlinear activation function operations. Based on the parameters of the real cooking image, an upsampling convolution operation is performed on the intermediate operation result to generate the cooking scene image.
[0056] Optionally, upsampling convolution operations on intermediate computation results include performing bilinear interpolation and convolution operations on intermediate computation results to maintain better feature map details and avoid problems such as edge jaggedness or stacking effects.
[0057] Optionally, performing upsampling convolution on the intermediate computation results includes: performing deconvolution on the intermediate computation results to obtain a deconvolution result. Then, the deconvolution result is fused with the intermediate computation results. This method can improve the accuracy of the generated cooking scene images.
[0058] In summary, the cooking scene image generation method provided by this invention only requires inputting specified images of ingredients, cookware, and stove into a pre-trained generator. No additional parameters or human intervention are needed. This allows the conditional generative network with the generator to automatically generate a large number of cooking images with the cookware and stove background, significantly increasing the number of images. This invention effectively avoids repetitive cooking scenes and the high cost of reproducing cooking, providing image input data for deep learning in smart cooking, improving the recognition accuracy of model algorithms, and ultimately providing users with a better interactive experience.
[0059] Based on the same inventive concept as the cooking scene generation method, one or more embodiments of the present invention can also provide a cooking scene image generation device, which includes, but is not limited to, an image acquisition module, a feature extraction module, a vector stitching module, and an image generation module.
[0060] The image acquisition module is used to acquire cooking background images and cooking object images, both of which are related to the cooking scene.
[0061] Optionally, the image acquisition module is specifically used to acquire images of the cookware and the stove, and to use these images as cooking background images. The image acquisition module is also specifically used to acquire images of the ingredients, and to use these images as cooking object images.
[0062] The feature extraction module is used to perform feature extraction processing on the cooking background image and the cooking object image respectively, so as to extract the first feature vector from the cooking background image and the second feature vector from the cooking object image.
[0063] Optionally, the feature extraction module is specifically used to perform feature classification on cookware images using a trained cookware classifier, so as to extract cookware classification feature vectors from the cookware classifier. The feature extraction module is specifically used to perform feature classification on stovetop images using a trained stovetop classifier, so as to extract stovetop classification feature vectors from the stovetop classifier; the first feature vector includes a cookware classification feature vector and a stovetop classification feature vector. The feature extraction module is specifically used to perform feature classification on food images using a trained food classifier, so as to extract food category feature vectors from the food classifier; the second feature vector is the food category feature vector.
[0064] The vector concatenation module is used to concatenate the first feature vector and the second feature vector to obtain the target feature vector.
[0065] Optionally, the vector concatenation module is specifically used to concatenate the food classification feature vector, cookware classification feature vector, stove classification feature vector, and Gaussian noise signal. The Gaussian noise signal is a random noise signal used by the conditional adversarial generative network (GAN), which generates a cooking scene image based on the target feature vector and parameters from the real cooking image.
[0066] The image generation module is used to generate cooking scene images based on the target feature vector and parameters of real cooking images.
[0067] Optionally, the image generation module is specifically used to perform downsampling convolution operations on the target feature vector to obtain intermediate operation results; and to perform upsampling convolution operations on the intermediate operation results based on the parameters of the real cooking image to generate a cooking scene image. The parameters of the cooking scene image are the same as those of the real cooking image.
[0068] Optionally, the image generation module is specifically used to perform bilinear interpolation and convolution operations on the intermediate calculation results, so as to realize upsampling convolution operations on the intermediate calculation results.
[0069] Optionally, the image generation module is specifically used to perform deconvolution operation on the intermediate operation results to obtain the deconvolution operation result; and to perform feature fusion between the deconvolution operation result and the intermediate operation results by concatenation to realize upsampling convolution operation on the intermediate operation results.
[0070] like Figure 3 As shown, based on the same inventive concept, one or more embodiments of the present invention can provide a smart cooker. This smart cooker may include, but is not limited to, the cooking scene image generation device in any embodiment of the present invention; optionally, the smart cooker is a smart range hood.
[0071] It is worth noting that this invention provides a deep learning-based intelligent range hood with a cooking scene image generation function. The intelligent range hood of this invention is equipped with a camera to acquire real cooking scene images, including stovetop images. Based on the large number of cooking scene images generated by this invention, it can identify and judge various information such as different stages of the cooking process, various ingredients, and human postures and movements. Furthermore, by combining big data statistics, it can provide users with accurate cooking action recognition, various warning prompts, and safer and healthier dietary cooking suggestions, resulting in a better user experience.
[0072] It is understood that the smart range hood of the present invention may be equipped with a communication module, which can be connected to a communication network to realize Internet of Things functions, or communicate with devices such as mobile terminals.
[0073] like Figure 6As shown, based on the same inventive concept, one or more embodiments of the present invention can provide a cooker control method. When the cooker is a smart range hood, the present invention can provide a control method for a smart range hood. This cooker control method may include, but is not limited to, the cooking scene image generation method in any embodiment of the present invention, to control the cooker according to the generated cooking scene image, so as to realize functions such as smart cooking, anomaly detection, ingredient detection, recipe recommendation, ingredient processing guidance, and AI smart kitchen. The cooking scene image generation method is as described above and will not be repeated here.
[0074] It should be understood that the present invention can also provide a computer storage medium storing a cooking scene image generation program. When this cooking scene image generation program is executed by a processor, it implements the cooking scene image generation method in any embodiment of the present invention, which is as described above and will not be repeated here. Therefore, the intelligent range hood provided by the present invention can be equipped with a processor and a memory, the memory storing the cooking scene image generation program, and the processor being able to execute the cooking scene image generation program.
[0075] In the description of this specification, the references to terms such as "this embodiment," "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0076] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0077] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0078] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0079] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections (electronic devices) having one or more wires, portable computer disks (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, or flash memory), fiber optic devices, and compact disc read-only memory (CDROM). Furthermore, computer-readable storage media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
[0080] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0081] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made under the concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.
Claims
1. A method for generating cooking scene images, characterized in that, include: Acquire images of cookware and stovetop, and use these images as cooking background images; Acquire food ingredient images and use these images as cooking object images, where both the cooking background image and the cooking object image are images related to the cooking scene; The cookware image is categorized using a trained cookware classifier to extract a cookware classification feature vector. The stove image is categorized using a trained stove classifier to extract a stove classification feature vector. The food image is categorized using a trained food classifier to extract a food classification feature vector. A first feature vector is extracted from the cooking background image, and a second feature vector is extracted from the cooking object image. The first feature vector includes the cookware classification feature vector and the stove classification feature vector, and the second feature vector is the food classification feature vector. The first feature vector and the second feature vector are concatenated to obtain the target feature vector; The target feature vector is used as the input to the generator in the conditional adversarial generative network; The generator outputs a cooking scene image to combine the cooking object image and the cooking background image into a cooking scene image; wherein, the parameters of the cooking scene image are the same as those of the real cooking image; the parameters of the image include the image's height, width, and channel parameters.
2. The method for generating cooking scene images according to claim 1, characterized in that, The step of generating a cooking scene image based on the target feature vector and real cooking image parameters includes: Perform downsampling convolution on the target feature vector to obtain intermediate calculation results; Based on the parameters of the real cooking image, an upsampling convolution operation is performed on the intermediate calculation results to generate the cooking scene image.
3. The method for generating cooking scene images according to claim 2, characterized in that, The upsampling convolution operation on the intermediate calculation result includes: Bilinear interpolation and convolution operations are performed on the intermediate calculation results.
4. The method for generating cooking scene images according to claim 2, characterized in that, The upsampling convolution operation on the intermediate calculation result includes: Perform a deconvolution operation on the intermediate calculation results to obtain the deconvolution result; The deconvolution result is fused with the intermediate result to perform feature fusion.
5. The method for generating cooking scene images according to claim 1, characterized in that, The process of concatenating the first feature vector and the second feature vector includes: The food ingredient classification feature vector, the cookware classification feature vector, the stove classification feature vector, and the random noise signal are spliced together.
6. A cooking scene image generation device, characterized in that, include: The image acquisition module is used to acquire images of cookware and stove, and use the cookware and stove images as cooking background images; acquire images of ingredients, and use the ingredients images as cooking object images; both the cooking background images and the cooking object images are images related to the cooking scene. The feature extraction module is used to classify cookware images using a trained cookware classifier to extract cookware classification feature vectors from the cookware classifier; to classify stove images using a trained stove classifier to extract stove classifier feature vectors from the stove classifier; and to classify food images using a trained food classifier to extract food classifier feature vectors from the food classifier. These features are used to extract a first feature vector from the cooking background image and a second feature vector from the cooking object image. The first feature vector includes the cookware classification feature vector and the stove classification feature vector, and the second feature vector is the food classification feature vector. The vector concatenation module is used to concatenate the first feature vector and the second feature vector to obtain the target feature vector; The image generation module is used to take the target feature vector as input to the generator in the conditional adversarial generative network, and output a cooking scene image through the generator to synthesize the cooking object image and the cooking background image into a cooking scene image; wherein, the parameters of the cooking scene image are the same as the parameters of the real cooking image; the parameters of the image include the image height parameter, width parameter, and channel parameter.
7. A cooking utensil, characterized in that, Includes the cooking scene image generation device as described in claim 6.
8. The cookware according to claim 7, characterized in that, The cooking appliance is a range hood.
9. A cooking utensil control method, characterized in that, The method for generating cooking scene images according to any one of claims 1 to 5 is used to control cooking utensils based on the generated cooking scene images.