Method, processor, household appliance and storage medium for processing images

Abstract

The present invention relates to the field of household appliances, and discloses a method for processing images, a processor, a household appliance, and a storage medium. The method is applied to household appliances, and comprises: when the household appliance starts to work, obtaining an image associated with the working scene of the household appliance; determining whether there is an interference component that needs to be removed in the image; when it is determined that there is an interference component that needs to be removed in the image, inputting the image into a generator of a generative adversarial network to generate an image with the interference component removed; wherein the generative adversarial network also includes a discriminator, and the generator and the discriminator are trained alternately. The above technical solution can be used to remove oil smoke in an image when the household appliance is a range hood; and can be used to remove watermarks in an image when the household appliance is a dishwasher. The present invention improves the image imaging quality of the working scene of the household appliance, improves the image processing effect of the smart household appliances, and improves the quality of the household appliances.

Description

Technical Field

[0001] The present invention relates to the field of household appliances, and in particular to a method for processing an image, a processor, a household appliance, a machine-readable storage medium and a computer program product. Background Art

[0002] With the development of the information society, smart home appliances are increasingly appearing in people's lives. The development of social and economic levels has led to higher and higher pursuit of the quality of home appliances, and the demand for home comfort and intelligence has become stronger and stronger. Smart kitchen is an important part of the entire smart home, and smart cooking and smart dishwashing are the top priorities of smart kitchen.

[0003] At present, when a range hood acquires an image of a stove, the image analysis is affected by the influence of oil smoke on the image. When a dishwasher acquires an image of tableware, the image analysis is affected by the influence of watermarks on the image. This results in poor image processing effects for smart home appliances. Summary of the invention

[0004] In order to overcome the deficiencies in the prior art, embodiments of the present invention provide a method, a processor, a household appliance, and a storage medium for processing an image.

[0005] In order to achieve the above object, the present invention provides a method for processing an image in a first aspect, which is applied to a household appliance. The method comprises:

[0006] When the household appliance starts to work, acquiring an image associated with a working scene of the household appliance;

[0007] Determine whether there is an interference component in the image that needs to be removed;

[0008] When it is determined that there is an interference component that needs to be removed in the image, the image is input into a generator of a generative adversarial network (GAN) to generate an image with the interference component removed;

[0009] The generative adversarial network also includes a discriminator, and the generator and discriminator are trained alternately.

[0010] In an embodiment of the present invention, when the household appliance starts working, a first frame image associated with the working scene of the household appliance is acquired and saved, and the first frame image and the generated image with interference components removed are used as sample pairs for training a generative adversarial network.

[0011] In an embodiment of the present invention, an image training sample set is obtained, the image training sample set includes a plurality of image pairs, each image pair includes a non-interference component image and an interference component image corresponding to the non-interference component image;

[0012] Inputting at least an image having an interference component into a generator to generate an image with the interference component removed;

[0013] Input the image without interference component and the image with interference component removed into the discriminator to output the result;

[0014] Calculate the loss function of the generative adversarial network, which is associated with the generator and the discriminator;

[0015] When the loss function satisfies the expected value, the training of the generator and the discriminator is terminated.

[0016] In the embodiment of the present invention, the interference component includes oil smoke or watermark.

[0017] In an embodiment of the present invention, inputting an image having at least an interference component into a generator to generate an image with the interference component removed includes:

[0018] Inputting the interference component image and the non-interference component image into the generator;

[0019] Down-sampling the input interference component image and the non-interference component image to extract a feature map;

[0020] The feature map is upsampled to generate an image with the interference components removed.

[0021] In an embodiment of the present invention, downsampling the input interference component image and the non-interference component image to extract a feature map includes:

[0022] splicing the interference component image and the non-interference component image;

[0023] The stitched image is downsampled to extract feature maps.

[0024] In an embodiment of the present invention, downsampling the input interference component image and the non-interference component image to extract a feature map includes:

[0025] Downsampling the image with interference component and the image without interference component respectively to extract the first feature map and the second feature map respectively;

[0026] The first feature map and the second feature map are concatenated to generate a feature map.

[0027] In an embodiment of the present invention, inputting at least an image having an interference component into a generator to generate an image with the interference component removed also includes:

[0028] Perform convolution operations on the feature map to perform nonlinear transformation without changing the feature size;

[0029] Upsampling the feature map to generate an image with interference components removed includes: upsampling the feature map after nonlinear transformation to generate an image with interference components removed.

[0030] In an embodiment of the present invention, downsampling includes multi-layer convolution operations.

[0031] In this embodiment of the present invention, the step size of the convolution operation is 2.

[0032] In an embodiment of the present invention, upsampling includes a combination of multi-layer bilinear interpolation and convolution operations.

[0033] In an embodiment of the present invention, the generator adopts a U-Net structure model.

[0034] A second aspect of the present invention provides a processor configured to execute the above-mentioned method for processing an image.

[0035] A third aspect of the present invention provides a household appliance, comprising the above-mentioned processor.

[0036] In an embodiment of the present invention, the household appliance includes a range hood or a dishwasher.

[0037] A fourth aspect of the present invention provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the above-mentioned method for processing an image.

[0038] The above technical solution obtains images related to the working scenes of household appliances, processes the images based on a generative adversarial network, and removes interference components in the images. For example, when the household appliance is a range hood, the above technical solution can be used to remove oil smoke in the image; when the household appliance is a dishwasher, the above technical solution can be used to remove watermarks in the image. In the above technical solution, the generator of the generative adversarial network is used to remove the interference components in the image, and the discriminator of the generative adversarial network is used to judge and identify the image with the interference components removed, to ensure that the interference components of the image are removed and meet the expected effect. At the same time, the generator and the discriminator are trained alternately. In this way, the imaging quality of the pictures of the working scenes of household appliances is improved, the image processing effect of smart appliances is improved, the quality of appliances is improved, and the intelligence level of appliances is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0040] Figure 1 A flowchart of a method for processing an image according to an embodiment of the present invention is schematically shown;

[0041] Figure 2 A diagram schematically shows an application scenario of a range hood according to an embodiment of the present invention;

[0042] Figure 3 A diagram schematically shows an application scenario of a dishwasher according to an embodiment of the present invention;

[0043] Figure 4 The overall design diagram of a generative adversarial network according to an embodiment of the present invention is schematically shown;

[0044] Figure 5 Schematically shows a workflow diagram of a generator according to an embodiment of the present invention;

[0045] Figure 6 A diagram schematically shows an input method diagram of a generator according to an embodiment of the present invention;

[0046] Figure 7 A schematic diagram of a U-Net structure model according to an embodiment of the present invention is shown;

[0047] Figure 8 A schematic diagram of a working flow of a discriminator according to an embodiment of the present invention is shown;

[0048] Fig. 9 A control flow chart of a method for processing an image according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0049] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0050] It should be noted that if the implementation methods of the present application involve directional indications (such as up, down, left, right, front, back...), such directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0051] In addition, if there are descriptions involving "first", "second", etc. in the implementation methods of this application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various implementation methods can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0052] Figure 1 The flowchart of the method for processing an image according to an embodiment of the present invention is schematically shown. Figure 1 As shown, in one embodiment of the present invention, a method for processing an image is provided, comprising the following steps:

[0053] Step 101, when the household appliance starts to work, acquiring an image associated with a working scene of the household appliance;

[0054] Step 102, determining whether there is an interference component that needs to be removed in the image;

[0055] Step 103: When it is determined that there is an interference component in the image that needs to be removed, the image is input into a generator of a generative adversarial network to generate an image with the interference component removed; wherein the generative adversarial network also includes a discriminator, and the generator and the discriminator are trained alternately.

[0056] With the development of the information society, smart home appliances are increasingly appearing in people's lives. The development of social and economic levels has led to higher and higher pursuit of the quality of home appliances, and the demand for home comfort and intelligence has become stronger and stronger. Smart kitchen is an important part of the entire smart home, and smart cooking and smart dishwashing are the top priorities of smart kitchens. Deploy a camera on the range hood to obtain image information of the stove, or deploy a camera on the dishwasher to obtain image information of the tableware, and then use artificial intelligence technology to analyze the image. The image quality of the picture is crucial for the subsequent image analysis of home appliances.

[0057] The method for processing an image in an embodiment of the present invention is applied to household appliances. Specifically, the household appliance may be a range hood, a dishwasher, or other types of appliances, which are not limited in the embodiment of the present invention. The interference component may be oil smoke or a watermark, or other types of interference components, which are not limited in the embodiment of the present invention. Here, oil smoke as an interference component refers to the interference of oil smoke that affects the clarity of the image captured by the camera. Watermark as an interference component refers to the image captured by the camera containing water droplets or water mist attached to the lens caused by water splashing onto the lens of the camera, also known as a "watermark".

[0058] In one embodiment, if Figure 2 As shown, Figure 2 The application scenario diagram of a range hood according to an embodiment of the present invention is schematically shown. When the household appliance is a range hood, when the range hood 1 starts to work, the range hood 1 uses a camera 2 to capture images associated with the working scene of the range hood 1, for example, to obtain image information of a stove 3, and then uses the method provided in the embodiment of the present invention to remove the oil smoke interference in the image.

[0059] It is inevitable that a large amount of oil smoke will be generated during the cooking process. Even with the use of a range hood, it is impossible to completely eliminate the impact of oil smoke on imaging. The quality of imaging is poor, which brings great trouble to the accuracy of the subsequent image processing algorithm of the home appliance. The method for processing images provided by the embodiment of the present invention can restore images with oil smoke and eliminate the interference of oil smoke. High-quality imaging quality determines the accuracy and precision of the artificial intelligence algorithm. Therefore, the embodiment of the present invention improves the quality of home appliances and the user's interactive experience.

[0060] In one embodiment, if Figure 3 As shown, Figure 3 The application scenario diagram of a dishwasher according to an embodiment of the present invention is schematically shown. When the household appliance is a dishwasher 4, when the dishwasher 4 starts to work, the dishwasher 4 uses a camera to capture images associated with the working scene of the dishwasher 4, such as obtaining image information of tableware 5, and then uses the method provided by the embodiment of the present invention to remove watermark interference in the image.

[0061] Similarly, when a dishwasher washes dishes, the water flow impacting the dishes will inevitably cause water splashes to adhere to the camera of the dishwasher. The watermark will cause image distortion, which will cause great trouble to the subsequent recognition and detection tasks of the dishwasher. The method for processing images provided by the present invention can restore images with watermarks and eliminate the interference of watermarks.

[0062] The embodiment of the present invention provides a method for removing interference components in an image based on a generative adversarial network. The method is an end-to-end algorithm process. It only requires the input of a picture, and does not require the assistance of additional parameters or human factors. The network can automatically generate pictures without interference components, thereby improving the imaging quality of home appliances, providing high-quality image input guarantees for home appliances, improving the recognition accuracy of image processing algorithms, and providing users with a better interactive experience.

[0063] The above technical solution obtains images associated with the working scenes of household appliances, processes the images based on a generative adversarial network, and removes interference components in the images. For example, when the household appliance is a range hood, the above technical solution can be used to remove oil smoke in the image; when the household appliance is a dishwasher, the above technical solution can be used to remove watermarks in the image. In the above technical solution, the generator of the generative adversarial network is used to remove the interference components in the image, and the discriminator of the generative adversarial network is used to judge and identify the image with the interference components removed, to ensure that the interference components of the image are removed and meet the expected effect. At the same time, the generator and the discriminator are trained alternately. In this way, the imaging quality of the pictures of the working scenes of household appliances is improved, the image processing effect of smart appliances is improved, the quality of appliances is improved, and the intelligence level of appliances is improved.

[0064] In one embodiment, the method for processing an image further comprises:

[0065] When the household appliance starts to work, a first frame image associated with a working scene of the household appliance is acquired and saved, and the first frame image and the generated image with interference components removed are used as sample pairs for training a generative adversarial network.

[0066] In one embodiment, the method for processing an image further comprises:

[0067] Acquire an image training sample set, the image training sample set includes a plurality of image pairs, each image pair includes a non-interference component image and an interference component image corresponding to the non-interference component image;

[0068] Inputting at least an image having an interference component into a generator to generate an image with the interference component removed;

[0069] Input the image without interference component and the image with interference component removed into the discriminator to output the result;

[0070] Calculate the loss function of the generative adversarial network, which is associated with the generator and the discriminator;

[0071] When the loss function satisfies the expected value, the training of the generator and the discriminator is terminated.

[0072] In one embodiment, the interference component includes oil smoke or watermark.

[0073] In one embodiment, inputting at least an image having an interference component into a generator to generate an image with the interference component removed includes:

[0074] Inputting the interference component image and the non-interference component image into the generator;

[0075] Down-sampling the input interference component image and the non-interference component image to extract a feature map;

[0076] The feature map is upsampled to generate an image with the interference components removed.

[0077] In one embodiment, downsampling the input interference component image and the non-interference component image to extract a feature map includes:

[0078] splicing the interference component image and the non-interference component image;

[0079] The stitched image is downsampled to extract feature maps.

[0080] In one embodiment, downsampling the input interference component image and the non-interference component image to extract a feature map includes:

[0081] Downsampling the image with interference component and the image without interference component respectively to extract the first feature map and the second feature map respectively;

[0082] The first feature map and the second feature map are concatenated to generate a feature map.

[0083] In one embodiment, the method of inputting at least an image having an interference component into a generator to generate an image with the interference component removed further comprises:

[0084] Perform convolution operations on the feature map to perform nonlinear transformation without changing the feature size;

[0085] Upsampling the feature map to generate an image with interference components removed includes: upsampling the feature map after nonlinear transformation to generate an image with interference components removed.

[0086] In one embodiment, downsampling includes multiple layers of convolution operations.

[0087] In one embodiment, the stride of the convolution operation is 2.

[0088] In one embodiment, upsampling includes a combination of multiple layers of bilinear interpolation and convolution operations.

[0089] In one embodiment, the generator adopts a U-Net structure model.

[0090] The method for processing an image provided by an embodiment of the present invention is described below by taking a specific embodiment as an example.

[0091] Cameras can be flexibly deployed on household appliances according to different application scenarios. For example, if a camera is deployed on a range hood, the camera can continuously record the cooking process on the stove. When the user turns on the ignition, the range hood starts to store the first frame of the image, and then continuously transmits the video or image data to the back-end system for distribution and processing. The camera can also be deployed on a dishwasher. The camera can continuously record the information of the tableware in the dishwasher. When the dishwasher door is closed, the camera is initialized, and the dishwasher starts to store the first frame of the image, and then continuously transmits the video or image data to the back-end system for distribution and processing.

[0092] like Figure 4 As shown, Figure 4 The overall design diagram of the generative adversarial network for removing interference components is schematically shown, including a generator and a discriminator. The generator is used to generate a synthetic image, and the discriminator is used to judge the authenticity of the generated image with interference components removed and the real image without interference. The discriminator is used to guide the image with interference components removed generated by the generator to be closer and closer to the real image without interference components. During the training process, the generator and the discriminator are trained alternately.

[0093] The generator part of the generative adversarial network in an embodiment of the present invention is introduced below.

[0094] The generator is composed of a series of downsampling convolution (encode) and upsampling convolution (decode) networks, which converts the input image with interference components into an image without interference components. Figure 5 As shown, Figure 5 The following is a schematic diagram of a workflow of a generator according to an embodiment of the present invention. The detailed steps are as follows:

[0095] (1) The images captured by the camera can be simultaneously transmitted to the processing chip via USB (Universal Serial Bus). The processing chip integrates the basic image operation software library and multi-threaded processing library to speed up the image processing process.

[0096] (2) In order to train the generative adversarial network, image sample data that can be used for training is first constructed. The construction method is to obtain and save the real non-interference component images associated with the working scene of the household appliance before the household appliance starts working, and then obtain the interference component images after the household appliance starts working. The images can be set to a size of 640*480 pixels, but other pixel sizes are also feasible. In this way, in each working scene of the household appliance, such a set of image pairs with interference components and non-interference components is constructed. The real non-interference component images can also correspond to multiple sets of interference component images. The interference components of these interference component images have different degrees of influence, which constitutes the samples required for training the generative adversarial network.

[0097] (3) Randomly select two images from a group of sample data sets, one of which is an image without interference components, and the other is a corresponding image with interference components. The image with interference components is input into the convolutional neural network. The convolutional neural network consists of two parts: downsampling convolution (encode) and upsampling convolution (decode), and both are deep convolutional neural networks.

[0098] (4) For the downsampling convolution (encode) part, first use the above-mentioned interference component image as the input layer (Input Layers). The image size is 640x480x3 pixels, that is, a color 3-channel image. During the training process, in order to increase the diversity of training samples, data augmentation (Data Augmentation) can be performed on the data samples to make the entire neural network model more robust and adapt to image inputs in more different scenarios. For specific scenarios, the image can be randomly transformed in brightness or saturation in the data augmentation part, and then the interference component images obtained after these random transformations are input into the model as the input layer.

[0099] (5) In order to increase the judgment accuracy of the image, the embodiment of the present invention can provide a variety of input strategies, for example, inputting the interference-free component image of the first frame into the model, so that the generator can better utilize the comparison between images to improve the quality of the generated interference-free component image and generate the interference-free component image more accurately. Figure 6 As shown, Figure 6 The following is a schematic diagram showing an input method of a generator according to an embodiment of the present invention. Figure 6Three input strategies are illustrated. Strategy A is to directly generate an image without interference components from an image with interference components. Strategy B is to concatenate the image without interference components with the image with interference components to generate an image without interference components. Strategy C is to convolve the image without interference components and the image with interference components separately, concatenate the feature maps generated by the two, and then generate an image without interference components. Both strategies B and C can improve the accuracy of the generated network, so that the image generated by the network without interference components is closer to the real image without interference components.

[0100] (6) The downsampling convolution (encode) part is a convolution process that continuously performs downsampling. After the nonlinear transformation of continuous convolution to extract features, the final output image dimension becomes W*H*C. Among them, C refers to the output channel, W refers to the width of the feature, and H refers to the height of the feature. According to different deep learning development frameworks, the arrangement position of the channel may not be the same. Some frameworks may correspond to C*W*H. For the sake of uniformity, the channel can be placed in the last dimension. Among them, each convolution operation is performed through a convolution with a convolution kernel of size 3*3, and then through a batch normalization operation (Batch Normalization), and then through the nonlinear activation function of Relu. It should be noted that the activation function is not necessarily limited to the Relu function. Relu6, Swish or Mish functions can also be used. There is no limitation on this. It depends on the training effect.

[0101] (7) In the downsampling convolution (encode) part, the usual pooling operation is not used, but a convolution conv operation with a stride of 2 is used to halve the size of the feature. After multiple downsampling, the input image is converted from 640*480*3 pixels to a feature map of 20*15*1024 pixels.

[0102] (8) The above feature map of 20*15*1024 pixels undergoes a 3x3 convolution and a nonlinear transformation without changing the feature size to abstract more semantic information. The resulting size is still 20*15*1024 pixels, which will serve as the input for the subsequent upsampling convolution (decode) part.

[0103] (9) Next is the upsampling convolution (decoding) part. Decoding is also a process of continuous convolution and upsampling. Each convolution operation is performed through a convolution with a 3x3 convolution kernel, followed by a batch normalization operation (Batch Normalization), and then a nonlinear activation function.

[0104] In the embodiment of the present invention, there may be multiple up-sampling methods.

[0105] In the first example scheme: direct upsampling, for the upsampled (i.e., enlarged feature size) part, deconvolution (deconv) operation can be used again, or bilinear interpolation upsampling and convolution, to continuously enlarge the feature map, and each time it is enlarged by one time, the number of channels is reduced by one time, and finally a picture of 640x480x3 pixels is obtained. In the present invention, the usual deconv deconvolution operation is not used to enlarge the feature map, but bilinear interpolation upsampling and convolution are used to enlarge the feature map. This method can maintain better feature map details and avoid edge jaggedness or stacking effects caused by deconvolution operations.

[0106] In the second example scheme: In order to enhance more semantic information, the U-Net structural model can be adopted, that is, the feature output of the encoding is spliced ​​to the corresponding decoded output. Figure 7 The structural model diagram of U-Net according to an embodiment of the present invention is schematically shown.

[0107] (10) The obtained image of 640x480x3 pixels is the same size as the original real non-interference component image. Their pixel difference constitutes the final loss function of the entire network, which is used to guide the network training in back propagation, so that the decoded image and the original image tend to be consistent.

[0108] The discriminator part of the generative adversarial network in an embodiment of the present invention is introduced below.

[0109] The discriminator is also composed of a series of convolutional networks, and finally a binary classifier (similar to a 0-1 classifier) ​​is generated through the sigmoid function to determine whether the input image sample is true or false. Figure 8 As shown, Figure 8 The following is a schematic diagram showing the working flow of a discriminator according to an embodiment of the present invention. Detailed steps are as follows.

[0110] (1) The final generated image of 640x480x3 pixels with the interference component removed and the original real image without interference component are input into the network respectively.

[0111] (2) After continuous nonlinear transformation of convolutional feature extraction, the final dimension of the output becomes 1*1*C. Here C refers to the output chanel. Each convolution operation is performed with a convolution kernel of 3x3 size and a stride of 2. In this way, the size of the feature map is halved each time, and the chanel is doubled. Then, it goes through batch normalization and the nonlinear activation function of Leaky Relu.

[0112] (3) Finally, after a 1*1 convolution, the chanel (channel) is changed to 1, and then a sigmoid function is connected to the output. The final result is used as a binary classifier to judge true or false.

[0113] The loss function of the generative adversarial network in the present invention is introduced below.

[0114]

[0115] In the above loss function formula, D represents the discriminator, G represents the generator, x represents the real data, that is, the real panoramic picture collected, z represents the image with interference component synthesized by the generator, and E represents the expectation. The optimization target is V(D,G), the purpose is to make the probability D(G(z)) of judging G(z) as true as small as possible, that is, 1-D(G(z)) as large as possible, and the probability D(x) of judging the real data x as true as large as possible.

[0116] During the entire training process of the generative adversarial network, the generator and the discriminator are trained alternately to make the entire network reach a Nash equilibrium point. Finally, only the generator network is used to deploy it in the system, and the generator is used to generate images without interference components.

[0117] The embodiment of the present invention provides a control method for image processing for removing interference components. The method can utilize a multi-angle camera. When a user uses a household appliance, the camera can capture images of multiple angles in the working scene of the household appliance. After obtaining enough image information of different angles and dimensions, the image information can be recognized more accurately. Fig. 9 The control flow chart of the method for processing an image according to an embodiment of the present invention is schematically shown. The detailed steps are as follows.

[0118] (1) When it is detected that a user starts to use a household appliance, the camera is turned on, the first frame of the image is saved, and images of the working scene of the household appliance are continuously collected and transmitted to the chip for processing.

[0119] (2) Save the first frame of the interference-free component image for subsequent model improvement.

[0120] (3) Determine whether interference components need to be removed from each frame of the image.

[0121] (4) If it is necessary to remove the interference component, the image with the interference component is input into the generative adversarial network.

[0122] (5) The generated image without interference components is sent to the back-end system for subsequent intelligent algorithm analysis and processing.

[0123] (6) If the appliance is not finished, continue reading the image and return to step 3 to continue the cycle.

[0124] (7) If the home appliance stops working, the shooting will end automatically.

[0125] exist Fig. 9 In the case where the household appliance is a range hood and the interference component is oil smoke, manual interaction may include: detecting firepower matching based on food information; detecting firepower matching based on the type of pot; monitoring firepower output to match cooking tasks; and starting a reminder when the stove is accidentally turned off or forgotten to turn off the fire after cooking.

[0126] The embodiment of the present invention provides a solution for removing oil smoke based on a generative adversarial network, which solves the problem of image interference caused by oil smoke during the cooking process. The entire process does not require any parameters or human identification factors. It is an end-to-end architecture, which makes home appliances more applicable. At the same time, an overall control process for oil smoke removal is also designed, which can enable subsequent algorithms to obtain more accurate output results and improve the quality of home appliances.

[0127] The embodiment of the present invention also provides a watermark removal solution based on a generative adversarial network, which solves the problem of image distortion caused by watermarks when washing dishes in a dishwasher. The entire process does not require any parameters or human identification factors, and is an end-to-end architecture that makes home appliances more applicable. At the same time, a control process for watermark removal is also designed, which can enable subsequent algorithms to obtain more accurate output results and improve the quality of home appliances.

[0128] The above technical solution obtains images related to the working scenes of household appliances, processes the images based on a generative adversarial network, and removes interference components in the images. For example, when the household appliance is a range hood, the above technical solution can be used to remove oil smoke in the image; when the household appliance is a dishwasher, the above technical solution can be used to remove watermarks in the image. In the above technical solution, the generator of the generative adversarial network is used to remove the interference components in the image, and the discriminator of the generative adversarial network is used to judge and identify the image with the interference components removed, to ensure that the interference components of the image are removed and meet the expected effect. At the same time, the generator and the discriminator are trained alternately. In this way, the imaging quality of the pictures of the working scenes of household appliances is improved, the image processing effect of smart appliances is improved, the quality of appliances is improved, and the intelligence level of appliances is improved.

[0129] An embodiment of the present invention provides a processor, which is configured to execute any one of the methods for processing an image in the above embodiments.

[0130] Specifically, the processor can be configured to:

[0131] When the household appliance starts to work, acquiring an image associated with a working scene of the household appliance;

[0132] Determine whether there is an interference component in the image that needs to be removed;

[0133] When it is determined that there is an interference component that needs to be removed in the image, the image is input into a generator of a generative adversarial network to generate an image with the interference component removed;

[0134] The generative adversarial network also includes a discriminator, and the generator and discriminator are trained alternately.

[0135] In an embodiment of the present invention, the processor is configured to:

[0136] When the household appliance starts to work, a first frame image associated with a working scene of the household appliance is acquired and saved, and the first frame image and the generated image with interference components removed are used as sample pairs for training a generative adversarial network.

[0137] In an embodiment of the present invention, the processor is configured to:

[0138] Acquire an image training sample set, the image training sample set includes a plurality of image pairs, each image pair includes a non-interference component image and an interference component image corresponding to the non-interference component image;

[0139] Inputting at least an image having an interference component into a generator to generate an image with the interference component removed;

[0140] Input the image without interference component and the image with interference component removed into the discriminator to output the result;

[0141] Calculate the loss function of the generative adversarial network, which is associated with the generator and the discriminator;

[0142] When the loss function satisfies the expected value, the training of the generator and the discriminator is terminated.

[0143] In an embodiment of the present invention, the processor is configured to:

[0144] Interference components include smoke or watermarks.

[0145] In an embodiment of the present invention, the processor is configured to:

[0146] Inputting at least an image with an interference component into a generator to generate an image with the interference component removed, including:

[0147] Inputting the interference component image and the non-interference component image into the generator;

[0148] Down-sampling the input interference component image and the non-interference component image to extract a feature map;

[0149] The feature map is upsampled to generate an image with the interference components removed.

[0150] In an embodiment of the present invention, the processor is configured to:

[0151] The input interference component image and the non-interference component image are downsampled to extract feature maps, including:

[0152] splicing the interference component image and the non-interference component image;

[0153] The stitched image is downsampled to extract feature maps.

[0154] In an embodiment of the present invention, the processor is configured to:

[0155] The input interference component image and the non-interference component image are downsampled to extract feature maps, including:

[0156] Downsampling the image with interference component and the image without interference component respectively to extract the first feature map and the second feature map respectively;

[0157] The first feature map and the second feature map are concatenated to generate a feature map.

[0158] In an embodiment of the present invention, the processor is configured to:

[0159] Inputting at least an image having an interference component into a generator to generate an image with the interference component removed, further comprising:

[0160] Perform convolution operations on the feature map to perform nonlinear transformation without changing the feature size;

[0161] Upsampling the feature map to generate an image with interference components removed includes: upsampling the feature map after nonlinear transformation to generate an image with interference components removed.

[0162] In an embodiment of the present invention, the processor is configured to:

[0163] Downsampling involves multiple layers of convolution operations.

[0164] In an embodiment of the present invention, the processor is configured such that the step size of the convolution operation is 2.

[0165] In an embodiment of the present invention, the processor is configured such that upsampling includes a combination of multi-layer bilinear interpolation and convolution operations.

[0166] In an embodiment of the present invention, the processor is configured such that: the generator adopts a U-Net structure model.

[0167] An embodiment of the present invention provides a home appliance, which includes the above-mentioned processor.

[0168] In an embodiment of the present invention, the household appliance includes a range hood or a dishwasher.

[0169] An embodiment of the present invention provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned method for processing an image.

[0170] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0171] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0172] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0173] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0174] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0175] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0176] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0177] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0178] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for processing an image, applied to a household appliance, characterized in that: The method comprises: When the household appliance starts to work, acquiring an image associated with a working scene of the household appliance; Determining whether there is an interference component that needs to be removed in the image; In the case where it is determined that there is an interference component that needs to be removed in the image, the image is input into a generator of a generative adversarial network to generate an image with the interference component removed; wherein the generative adversarial network also includes a discriminator, and the generator and the discriminator are trained alternately; The method further comprises: Acquire an image training sample set, wherein the image training sample set includes a plurality of image pairs, each image pair includes a non-interference component image and an interference component image corresponding to the non-interference component image; Inputting at least the image with interference components into the generator to generate an image with interference components removed; Inputting the image without interference components and the image with interference components removed into the discriminator to output a result; Calculating a loss function of the generative adversarial network, the loss function being associated with the generator and the discriminator; When the loss function satisfies the expected value, the training of the generator and the discriminator is terminated; The step of inputting at least the image with interference components into the generator to generate an image with the interference components removed comprises: Inputting the interference component image and the non-interference component image into the generator; Down-sampling the input interference component image and the non-interference component image to extract a feature map; Upsampling the feature map to generate an image with interference components removed; The step of downsampling the input interference component image and the non-interference component image to extract the feature map comprises: splicing the interference component image and the non-interference component image; downsampling the spliced ​​image to extract the feature map; or The interference component image and the non-interference component image are downsampled respectively to extract a first feature map and a second feature map respectively; the first feature map and the second feature map are spliced ​​to generate a feature map.

2. The method according to claim 1, characterized in that Also includes: When the household appliance starts to work, a first frame image associated with a working scene of the household appliance is acquired and saved, and the first frame image and the generated image with interference components removed are used as sample pairs for training the generative adversarial network.

3. The method according to claim 1, characterized in that The interference component includes oil smoke or watermark.

4. The method according to claim 1, characterized in that: The step of inputting at least the image with interference component into the generator to generate an image with interference component removed further comprises: Performing a convolution operation on the feature map to perform a nonlinear transformation without changing the feature size; The up-sampling of the feature map to generate an image with interference components removed includes: up-sampling the feature map after nonlinear transformation to generate an image with interference components removed.

5. The method according to claim 1, characterized in that The downsampling includes multiple layers of convolution operations.

6. The method according to claim 5, characterized in that The step size of the convolution operation is 2.

7. The method according to claim 1, characterized in that The upsampling includes a combination of multi-layer bilinear interpolation and convolution operations.

8. The method according to claim 1, characterized in that The generator adopts a U-Net structure model.

9. A processor, characterized in that: The device is configured to perform the method for processing an image according to any one of claims 1 to 8.

10. A household appliance, characterized in that: The processor according to claim 9 is included, wherein the household appliance comprises a range hood or a dishwasher.

Images