Method, processor, control device and household appliance for processing image
By using a generative adversarial network generator to perform deep learning fusion on kitchen images, the complexity and ease of use of the image fusion process are solved, generating realistic panoramic images and improving the accuracy of image recognition and user experience.
Patent Information
- Application Number
- CN202110390977.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-12
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-04-12
AI Technical Summary
Existing technologies suffer from high complexity and low ease of use in image fusion processes, especially when kitchen scenes change, requiring experience-based adjustments, which makes the image fusion process inconvenient.
A generative adversarial network generator is used to stitch and fuse two images with overlapping parts. A panoramic image is generated through deep learning. The generator and discriminator are trained using an image training sample set, and the loss function is optimized to generate a realistic panoramic image.
It simplifies the image fusion process and achieves realistic results, improving the accuracy of image recognition and user experience, and adapting to image analysis needs in different scenarios.
Smart Images

Figure CN113516155B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of household appliances, and in particular to a method for processing an image, a processor, a control device, a household appliance, a machine-readable storage medium and a computer program product. Background Art
[0002] With the development of the information society, smart home appliances are appearing more and more in people's lives. The development of social and economic levels has made people pursue higher and higher quality of home appliances, and the demand for home comfort and intelligence has become increasingly strong. Smart kitchen is an important part of the entire smart home, and smart cooking is the top priority of the smart kitchen.
[0003] At present, range hoods with image recognition can track and shoot the conditions on the stove from multiple angles, then fuse the multi-angle images to construct a fused image of three-dimensional stereograms and object association matching, and then proceed to the next step according to user needs. When performing image fusion in the existing technology, due to many factors such as image size, resolution, image operation calculation complexity, and feature extraction and matching of multiple images, the whole process is divided into several steps. Each step has many hyperparameters, which brings high complexity to the entire fusion process. If you want to change the scene, you still need some experience and skills to make continuous adjustments, which reduces the ease of use of the entire process. Summary of the invention
[0004] The purpose of the embodiments of the present invention is to provide a method for processing an image, a processor, a control device, a household appliance, a machine-readable storage medium, and a computer program product.
[0005] In order to achieve the above object, an embodiment of the present invention provides a method for processing an image, which is applied to a range hood or an integrated stove, comprising:
[0006] Acquire a first image obtained by taking a picture of a first image acquisition area by a first image acquisition device, and a second image obtained by taking a picture of a second image acquisition area by a second image acquisition device, wherein the first image and the second image have an intersecting portion;
[0007] The first image and the second image are input into a generator of a generative adversarial network to generate a panoramic image that fuses the first image and the second image.
[0008] In an embodiment of the present invention, the method further comprises:
[0009] Acquire an image training sample set, the image training sample set comprising a plurality of image triplets, each image triplet comprising a first training image captured by a first image acquisition device, a second training image captured by a second image acquisition device, and a training panoramic image covering the first image acquisition area and the second image acquisition area;
[0010] Inputting the first training image and the second training image in the image triplet into a generator to generate a panoramic image that fuses the first training image and the second training image;
[0011] Input the generated panoramic image and the training panoramic image into the discriminator of the generative adversarial network to output the result;
[0012] Calculate the loss function of the generative adversarial network, which is associated with the generator and the discriminator;
[0013] When the loss function satisfies the expected value, the training of the generator and the discriminator is terminated.
[0014] In an embodiment of the present invention, the first training image and the second training image in the image triplet are input into a generator to generate a panoramic image fused with the first training image and the second training image, including:
[0015] Inputting a first training image and a second training image into a generator;
[0016] Downsampling the input first training image and the second training image to extract a feature map;
[0017] The feature map is upsampled to generate a panoramic image that fuses the first training image and the second training image.
[0018] In an embodiment of the present invention, downsampling the input first training image and the second training image to extract a feature map includes:
[0019] Downsampling the first training image and the second training image respectively to extract a first feature map and a second feature map respectively;
[0020] The first feature map and the second feature map are processed to generate a feature map.
[0021] In an embodiment of the present invention, processing the first feature map and the second feature map to generate a feature map includes:
[0022] concatenating the first feature map and the second feature map to generate a feature map; or
[0023] The first feature map and the second feature map are added to generate a feature map.
[0024] In an embodiment of the present invention, adding the first feature map and the second feature map to generate a feature map includes:
[0025] Adding the first feature map and the second feature map to obtain a third feature map;
[0026] The height and number of channels of the third feature map are varied to generate feature maps.
[0027] In an embodiment of the present invention, the first training image and the second training image in the image triplet are input into a generator to generate a panoramic image fused with the first training image and the second training image, further comprising:
[0028] Perform convolution operations on the feature map to perform nonlinear transformation without changing the feature size;
[0029] Upsampling the feature map to generate a panoramic image that fuses the first training image and the second training image includes: upsampling the feature map that has undergone nonlinear transformation to generate a panoramic image that fuses the first training image and the second training image.
[0030] In an embodiment of the present invention, downsampling includes multiple layers of convolution operations.
[0031] In an embodiment of the present invention, the step size of the convolution operation is 2.
[0032] In an embodiment of the present invention, upsampling includes a combination of multi-layer bilinear interpolation and convolution operations.
[0033] In an embodiment of the present invention, the method further comprises:
[0034] Control the first image acquisition device and the second image acquisition device to shoot synchronously.
[0035] A second aspect of the present invention provides a processor configured to execute the above-mentioned method for processing an image.
[0036] A third aspect of the present invention provides a control device, which is applied to a household appliance. The control device comprises:
[0037] A first image acquisition device, used to take a picture of a first image acquisition area;
[0038] A second image acquisition device, used to take a picture of the second image acquisition area; and
[0039] The processors described above.
[0040] A fourth aspect of the present invention provides a household appliance, which includes the above-mentioned control device, and the household appliance includes at least one of a range hood and an integrated stove.
[0041] A fifth aspect of the present invention provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the above-mentioned method for processing an image.
[0042] A sixth aspect of the present invention provides a computer program product, comprising a computer program, which implements the above-mentioned method for processing an image when executed by a processor.
[0043] Through the above technical solution, by inputting the first image and the second image with the intersecting part into the generator of the generative adversarial network, the two images are spliced and fused using the generative adversarial network based on deep learning to generate a panoramic image that fuses the first image and the second image, thereby overcoming the deviation between image pixels caused by the fusion process of the traditional method, making the panoramic image that fuses the first image and the second image look more realistic, which is conducive to more data analysis to meet more user needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention.
[0045] In the figure:
[0046] Figure 1 is a flow chart of a method for processing an image in an embodiment of the present invention;
[0047] Figure 2 is a flow chart of a training method for a generative adversarial network in an embodiment of the present invention;
[0048] Figure 3 is a control flow chart when the control device in an embodiment of the present invention is applied to a stove;
[0049] Figure 4 is a schematic diagram of the arrangement of the first camera and the second camera in an embodiment of the present invention;
[0050] Figure 5 is a schematic diagram of the arrangement of a third camera in an embodiment of the present invention;
[0051] Figure 6 is a schematic diagram of the composition of a control device in an embodiment of the present invention;
[0052] Figure 7 Schematic diagram of the principle of a generative adversarial network in an embodiment of the present invention;
[0053] Figure 8 is a schematic diagram of the principle of a generator in an embodiment of the present invention;
[0054] Fig. 9 Schematic diagram of the principle of the discriminator in the embodiment of the present invention.
[0055] Description of Reference Numerals
[0056] 1 Range hood 2 Cooktop
[0057] 3 Control device 31 First camera
[0058] 32 Second camera 33 Third camera
[0059] 34 Processor DETAILED DESCRIPTION
[0060] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.
[0061] It should be noted that if the implementation methods of the present application involve directional indications (such as up, down, left, right, front, back, etc.), such directional indications are only used to explain the relative position relationship and movement status of the components under a certain specific posture. If the specific posture changes, the directional indication will also change accordingly.
[0062] In addition, if there are descriptions involving "first", "second", etc. in the implementation methods of this application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various implementation methods can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0063] With the development of information society, smart home appliances are increasingly appearing in people's lives. The development of social and economic levels has made people pursue higher and higher quality of home appliances, and the demand for home comfort and intelligence has become stronger and stronger. Smart kitchen is an important part of the entire smart home, and smart cooking is the top priority of smart kitchen. An important part of realizing smart cooking is to collect images of the kitchen by taking photos from multiple angles, and then stitching the collected multi-angle images to obtain a fused multi-angle image.
[0064] Traditional image stitching technology is to find the rotation-invariant feature points of the two images respectively. These feature points are usually represented by some high-dimensional (such as 128-dimensional) feature vectors. The feature vectors of the two images are then compared point by point to find the matching feature points. The two images are then stitched and fused based on these matching feature points. Due to many factors such as image size, resolution, image operation computational complexity, and feature extraction and matching of multiple images, the entire process of image stitching and fusion is divided into multiple steps, and each step has multiple hyperparameters, which greatly increases the complexity of the image fusion process. In addition, some experience and skills are required to continuously adjust after the kitchen scene is changed, which greatly reduces the ease of use of the entire image fusion process.
[0065] To solve the above problems, an embodiment of the present invention provides a novel method for processing an image, which is applied to a household appliance, wherein the household appliance includes at least one of a range hood or an integrated stove, such as Figure 1 As shown, the method for processing an image comprises the following steps:
[0066] Step S101: obtaining a first image obtained by a first image acquisition device photographing a first image acquisition area, and a second image obtained by a second image acquisition device photographing a second image acquisition area, wherein the first image and the second image have an intersecting portion.
[0067] It can be understood that the household appliance in this embodiment includes a range hood 1. Figure 4 As shown, the range hood 1 is provided with a first image acquisition device and a second image acquisition device, wherein the first image acquisition device is a first camera 31, and the second image acquisition device is a second camera 32. The first camera 31 and the second camera 32 are respectively arranged on the left and right sides of the bottom of the range hood 1. The first camera 31 is used to take pictures of a first image acquisition area. In this embodiment, the first image acquisition area is the left area below the range hood 1, and the image taken by the first camera 31 is the first image; the second camera 32 is used to take pictures of a second image acquisition area. In this embodiment, the second image acquisition area is the right area below the range hood 1, and the image taken by the second camera 32 is the second image. In addition, the image acquisition areas of the first camera 31 and the second camera 32 have overlapping parts, so the first image and the second image have intersecting parts.
[0068] In one embodiment of the present invention, a stove 2 is provided below the range hood 1, and the first camera 31 takes a picture of the left area of the stove 2, and the image taken is a first image; the second camera 32 takes a picture of the right area of the stove 2, and the image taken is a second image, and the intersection of the first image and the second image is the middle area of the stove 2.
[0069] Step S102: input the first image and the second image into a generator of a generative adversarial network to generate a panoramic image that fuses the first image and the second image.
[0070] It can be understood that in one embodiment of the present invention, the range hood 1 is provided with a control device 3, such as Figure 6 As shown, the control device 3 includes a first camera 31, a second camera 32 and a processor 34. The first camera 31 and the second camera 32 are electrically connected to the processor 34. The processor 34 includes a generative adversarial network (GAN, Generative Adversarial Networks, a deep learning model, which produces good output through mutual game learning of two modules, namely, a generator (Generative Model) and a discriminator (Discriminative Model) in the framework.). In this embodiment, when the control device 3 is applied to a mature product, the processor 34 only includes a generator part, which is a trained mature model. After the first image is taken, the first camera 31 transmits the first image to the processor 34; similarly, after the second image is taken, the second camera 32 transmits the second image to the processor 34. The processor 34 then inputs the first image and the second image into the generator of the generative adversarial network with a mature model, and the generator generates a panoramic image that fuses the first image and the second image based on the first image and the second image.
[0071] In one embodiment of the present invention, when the object photographed by the first camera 31 and the second camera 32 is the stove 2 below them, the panoramic image generated by the generator that fuses the first image and the second image includes multiple information on the stove 2, which is conducive to improving the completeness of the information collection of the stove 2, and is conducive to the processor 34 to extract more effective information from it based on the user's usage needs, so as to implement more accurate detection and discrimination, obtain more accurate output results, and greatly improve the accuracy of image recognition. For example, by identifying the size of the flame in the panoramic image that fuses the first image and the second image, and then more accurately determining the degree of doneness of the ingredients being cooked based on the accumulation of flame size and time, the user is assisted in obtaining accurate cooking instructions, thereby improving the user's usage experience.
[0072] The method for processing an image provided in this embodiment inputs a first image and a second image having an intersecting portion into a generator of a generative adversarial network, and uses a generative adversarial network based on deep learning to stitch and fuse the two images to generate a panoramic image that fuses the first image and the second image. The above-mentioned entire process fully implements an end-to-end algorithm, overcomes the deviation between image pixels caused by the fusion process of the traditional method, and makes the panoramic image that fuses the first image and the second image look more realistic. In addition, the panoramic image that fuses the first image and the second image can be used by the back end (such as processor 34) to perform more data analysis to meet more user needs, which is conducive to broadening the applicable scenarios of the product and improving the intelligence and automation of the product.
[0073] In an embodiment of the present invention, the method further comprises the following steps:
[0074] Step 201: Acquire an image training sample set, where the image training sample set includes multiple image triplets, each image triplet includes a first training image taken by a first image acquisition device, a second training image taken by a second image acquisition device, and a training panoramic image covering the first image acquisition area and the second image acquisition area.
[0075] It can be understood that the generator of the generative adversarial network in this embodiment needs to be trained before being applied to products (such as range hood 1 and stove 2) in order to obtain a mature model, and the discriminator is also trained at the same time as the generator. The training method of the generative adversarial network is as follows: Figure 2 As shown, specifically, before training the generator and the discriminator, a large number of image training sample sets must be obtained first. Since the generator needs two training images when generating a fused image, the discriminator needs to discriminate the generated fused image based on the panoramic image actually taken. Therefore, the image training sample set in this embodiment includes multiple image triplets, so that multiple trainings can be performed to obtain a mature generator and discriminator. Furthermore, each image triplet includes a first training image and a second training image for input into the generator for training. The first training image is taken by the first camera 31, and the second training image is taken by the second camera 32; each image triplet also includes a training panoramic image covering the first image acquisition area and the second image acquisition area for input into the discriminator for training.
[0076] In one embodiment of the present invention, during the training of the generator and the discriminator, the control device 3 further includes a third image acquisition device for capturing a training panoramic image covering the first image acquisition area and the second image acquisition area, wherein the third image acquisition device is a third camera 33, such as Figure 5As shown, the vertical distance from the third camera 33 to the stove 2 is greater than the vertical distance from the first camera 31 or the second camera 32 to the stove 2. In one embodiment of the present invention, the vertical distance from the third camera 33 to the stove 2 is twice the vertical distance from the first camera 31 or the second camera 32 to the stove 2, so that the acquisition area of the third camera 33 can cover the entire stove 2 below, that is, the image acquisition area of the third camera 33 completely covers the image acquisition areas of the first camera 31 and the second camera 32.
[0077] Step 202: Input the first training image and the second training image in the image triplet into a generator to generate a panoramic image that fuses the first training image and the second training image.
[0078] It can be understood that the first camera 31 captures the first training image and transmits it to the processor 34. Similarly, the second camera 32 captures the second training image and transmits it to the processor 34. The processor 34 then inputs the first training image and the second training image into the generator for training. During the training phase, the generator generates a panoramic image that fuses the first training image and the second training image based on the first training image and the second training image.
[0079] In one embodiment of the present invention, the size of the image captured by the first camera 31 and the second camera 32 is 640x512, and the size of the image captured by the third camera 33 is 1280x1024. Since the image captured by the third camera 33 includes content outside the shooting area of the first camera 31 and the second camera 32, and in order to enable the generative adversarial network to successfully train an image spliced from the first training image and the second training image, the processor 34 needs to crop the image captured by the third camera 33 into an image of size 1280x512.
[0080] Step 203: Input the generated panoramic image and the training panoramic image into the discriminator of the generative adversarial network to output the result.
[0081] It can be understood that after the panoramic image is generated, the processor 34 inputs it and the training panoramic image into the discriminator of the generative adversarial network to judge the authenticity of the generated panoramic image and the training panoramic image, and outputs the judgment result after the judgment is completed, that is, the discriminator judges whether the generated panoramic image is qualified based on the training panoramic image, and then guides the panoramic image generated by the generator to be closer and closer to the training panoramic image.
[0082] Specifically, Fig. 9As shown in the figure, the discriminator is composed of a series of convolutions, which generates a binary classifier (similar to the 0-1 classifier) through the sigmoid function to determine whether the input sample is true or false. In one embodiment of the present invention, the processor inputs the panoramic image and the training panoramic picture finally generated by the generator into the discriminator. After the panoramic image and the training panoramic image are subjected to nonlinear transformation of continuous convolution to extract features in the discriminator, the final dimension of the output becomes 1x1xC (here C refers to the output channel, that is, the output channel); further, each convolution operation in the discriminator is performed through a convolution kernel with a size of 3x3 and a step size of 2, so that after each convolution operation, the size of the feature map of the panoramic image and the training panoramic image is halved, and the channel is doubled, and then a batch normalization operation (BatchNormalization) is performed, and then a Leaky Relu nonlinear activation function is performed; finally, the feature map generated by the above process is subjected to a 1x1 convolution, the channel is changed to 1, and then a sigmoid function is connected for output, and the output result is used as a binary classifier for judging true or false.
[0083] In one embodiment of the present invention, the discriminator calculates the probability that the generated panoramic image is true based on the training panoramic image, and compares the probability with a preset threshold. In this implementation, the preset threshold is preferably 0.5, that is, when the probability is greater than or equal to 0.5, the discriminator determines that the generated panoramic image is true, that is, the generated panoramic image is judged to be qualified; when the probability is less than 0.5, the discriminator determines that the generated panoramic image is false, that is, the generated panoramic image is judged to be unqualified.
[0084] Step 204: Calculate the loss function of the generative adversarial network, where the loss function is associated with the generator and the discriminator.
[0085] It can be understood that the generative adversarial network is trained based on the loss function, and there is a correlation between the loss function and the generator and the discriminator. The model of the loss function in this embodiment is shown in the following formula:
[0086]
[0087] Where D is the discriminator; G is the generator; V(D, G) is the optimization target; x is the training panoramic image; D(x) is the probability that the discriminator judges the training panoramic image as true; z is the first training image and the second training image; G(z) is the panoramic image generated by the generator; D(G(z)) is the probability that the discriminator judges the generated panoramic image as true; To judge whether the generated panoramic image is real; The expectation of judging whether the generated panoramic image is fake.
[0088] From the above formula, we can see that the optimization target is V(D,G), which aims to make the probability D(G(z)) of the panoramic image G(z) generated by the generator being judged as true as small as possible (that is, 1-D(G(z)) is as large as possible), and to make the probability D(x) of the training panoramic image x being judged as true as large as possible.
[0089] Step 205: When the loss function satisfies the expected value, the training of the generator and the discriminator is terminated.
[0090] It is understandable that Figure 7 As shown, during the entire training process of the generative adversarial network, the generator and the discriminator are alternately trained to enable the entire generative adversarial network to reach a Nash equilibrium point. Furthermore, after the generative adversarial network reaches the Nash equilibrium point, it means that the generative adversarial network has matured. At this time, the discriminator determines that the panoramic image generated by the generator is a qualified image, that is, the generator can generate qualified panoramic images, and the processor 34 controls the training of the generator and the discriminator to end.
[0091] In an embodiment of the present invention, the first training image and the second training image in the image triplet are input into a generator to generate a panoramic image fused with the first training image and the second training image, comprising the following steps:
[0092] Step 301: inputting a first training image and a second training image into a generator;
[0093] Step 302: down-sample the input first training image and the second training image to extract a feature map.
[0094] It is understandable that Figure 8 As shown, the generator includes two parts, encoding and decoding, both of which are composed of deep convolutional neural networks. After receiving the first training image and the second training image, the processor 34 inputs them into the convolutional neural network of the generator to generate a panoramic image that fuses the first image and the second image.
[0095] For the encoding part, the images of different areas on the left and right sides of the stove 2 taken by the first camera 31 and the second camera 32 are first used as the input layer (Input Layers) of the convolutional neural network, that is, the first page training image and the second training image with a size of 640x512 are used as the input layer of the convolutional neural network. The dimensions of these two images are both 640x512x3, that is, the first training image and the second training image are both color 3-channel images.
[0096] In one embodiment of the present invention, in order to enhance the diversity of training samples, before training the generator, data augmentation is performed on the first training image and the second training image, so as to make the entire convolutional neural network model more robust and adapt to image inputs in more different scenarios. In this embodiment, for the kitchen stove scene, any random position transformation (such as random horizontal inversion, random cropping, etc.) of the first training image and the second training image is cancelled during data augmentation, and instead, random transformation of brightness and / or saturation is performed on them, and then the first training image and the second training image obtained after the above random transformation are input as the input layer into the convolutional neural network of the generator.
[0097] In one embodiment of the present invention, the encoder part of the generator performs a convolution process of continuous downsampling. After the encoder part performs continuous convolution to extract features of the first training image and the second training image, the final dimensions of the output of the first training image and the second training image become WxHxC, where W refers to the width of the feature, H refers to the height of the feature, and C refers to the output channel. Furthermore, according to different deep learning development frameworks, the positions of the channels may not be the same. Some frameworks may be CxWxH. In this embodiment, the channel is preferably placed in the last dimension.
[0098] In one embodiment of the present invention, each convolution operation during downsampling is performed through a convolution with a convolution kernel of size 3x3, followed by a batch normalization operation (Batch Normalization), and then a nonlinear activation function of Relu. Furthermore, the activation function is not necessarily limited to the Relu function, and functions such as Relu6, Swish or Mish can also be selected according to the actual training effect.
[0099] In one embodiment of the present invention, after downsampling the first training image and the second training image, each dimension of the first training image and the second training image may be changed, so as to extract feature maps from the first training image and the second training image.
[0100] Step 303: up-sample the feature map to generate a panoramic image that fuses the first training image and the second training image.
[0101] It can be understood that after the feature map is extracted, the decode part of the generator continues to upsample the feature map in order to generate a panoramic image of the same size as the training panoramic image.
[0102] Furthermore, the generated panoramic image and the training panoramic image are used together to form the loss function of the generative adversarial network, guiding the generative adversarial network to train in the back propagation, so that the panoramic image generated by the generator decode part and the training panoramic image tend to be consistent. On this basis, after multiple iterations and updates of the generative adversarial network, the generator can generate a qualified panoramic image that fuses the first training image and the second training image based on the first training image and the second training image.
[0103] In an embodiment of the present invention, downsampling the input first training image and the second training image to extract a feature map includes:
[0104] Step 401: down-sample the first training image and the second training image respectively to extract a first feature map and a second feature map respectively.
[0105] In one embodiment of the present invention, downsampling includes multiple layers of convolution operations, and the step size of the convolution operation is 2.
[0106] It can be understood that since both the first training image and the second training image are input into the generative adversarial network, the first training image and the second training image are downsampled in the encoder part of the generator, and the downsampling part does not need to adopt the usual pooling operation (Pooling), but adopts a multi-layer convolution operation, that is, the first training image and the second training image undergo multi-layer convolution operations in the encoder part of the generator, so that the dimensions of the first training image and the second training image are changed, so as to extract the first feature map from the first training image and the second feature map from the second training image. In one embodiment of the present invention, downsampling includes a multi-layer convolution operation with a step size of 2, so that the feature size of the first training image and the second training image is halved. After the first training image and the second training image undergo 5 downsamplings in the encoder part of the generator, the size of the extracted first feature map and the second feature map becomes 20x16x512.
[0107] Step 402: Process the first feature map and the second feature map to generate a feature map.
[0108] It can be understood that after extracting the first feature map and the second feature map of size 20x16x512, the first feature map and the second feature map are processed accordingly, and the encoding part of the generator can generate a feature map including the contents of the first feature map and the second feature map.
[0109] In an embodiment of the present invention, processing the first feature map and the second feature map to generate a feature map includes:
[0110] concatenate the first feature map and the second feature map to generate a feature map; or,
[0111] The first feature map and the second feature map are added to generate a feature map.
[0112] It can be understood that the encoding part of the generator can generate feature maps in a variety of ways, including concatenating the first feature map and the second feature map, both of which have a size of 20x16x512 (i.e., concatenation operation) together so that the generated feature map has 2C features; or performing an addition operation (i.e., add operation) on the first feature map and the second feature map, and the size of the feature map generated after the first feature map and the second feature map are added is still 20x16x512. The method of generating the feature map can be selected according to actual needs.
[0113] In an embodiment of the present invention, adding the first feature map and the second feature map to generate a feature map includes:
[0114] Step 501: Add the first feature map and the second feature map to obtain a third feature map.
[0115] Step 502: Change the height and the number of channels of the third feature map to generate a feature map.
[0116] In one embodiment of the present invention, a feature map is generated by adding the first feature map and the second feature map. Specifically, the first feature map and the second feature map are added to obtain a third feature map of size 20x16x512. Then, a 3x3 convolution operation is performed on the third feature map of size 20x16x512. The row step size used in this convolution operation is 1 and the column step size is 2. After the convolution operation is completed, a feature map is obtained. Compared with the third feature map, the width W of the feature map does not change, but the height H is reduced by half, and the channel is doubled, that is, the size of the feature map becomes 20x8x1024.
[0117] In an embodiment of the present invention, the first training image and the second training image in the image triplet are input into a generator to generate a panoramic image fused with the first training image and the second training image, further comprising:
[0118] The feature map is convolved to perform a nonlinear transformation without changing the feature size.
[0119] After the feature map is generated, a 3x3 convolution operation is performed on the feature map of size 20x8x1024 to perform a nonlinear transformation without changing the feature size, which is used to abstract more semantic information. After the 3x3 convolution operation, the size of the feature map is still 20x8x1024, and the feature map at this time can be used as the input of the decoding part.
[0120] Upsampling the feature map to generate a panoramic image that fuses the first training image and the second training image includes: upsampling the feature map that has undergone nonlinear transformation to generate a panoramic image that fuses the first training image and the second training image.
[0121] It can be understood that the decode part of the generator is a process of continuously performing convolution operations and upsampling. In one embodiment of the present invention, each convolution operation in this process is performed through a convolution with a convolution kernel of size 3x3, followed by a batch normalization operation (Batch Normalization), and then a nonlinear activation function.
[0122] The upsampling part can use a deconvolution operation or a bilinear interpolation upsampling + convolution operation to continuously enlarge the feature map, wherein each time the feature map is enlarged by one time, its channel is reduced by one time until an image of size 3x1280x512 is finally obtained. At this time, the image of size 3x1280x512 is a panoramic image that fuses the first training image and the second training image.
[0123] In an embodiment of the present invention, upsampling includes a combination of multi-layer bilinear interpolation and convolution operations.
[0124] It can be understood that in one embodiment of the present invention, the upsampling part does not use a deconvolution (deconv) operation to expand the feature map, but instead uses a bilinear interpolation upsampling + convolution operation to expand the feature map. This method is conducive to better maintaining the details of the feature map and can avoid the edge jaggedness or stacking effect caused by the deconvolution (deconv) operation.
[0125] In an embodiment of the present invention, the method for processing an image further includes:
[0126] Control the first image acquisition device and the second image acquisition device to shoot synchronously.
[0127] It can be understood that in one embodiment of the present invention, since it is necessary to count the information on the stove 2 at a certain moment, the processor 34 needs to control the first image acquisition device and the second image acquisition device to shoot synchronously, that is, in actual application, the first image and the second image are taken by the first image acquisition device and the second image acquisition device at the same time; in the process of generative adversarial network training, the first training image and the second training image are taken by the first image acquisition device and the second image acquisition device at the same time.
[0128] In summary, an embodiment of the present invention provides a method for processing an image. The method for processing an image overcomes the deviation between image pixels caused by the image fusion process in the traditional method by inputting two images into the generator of the generative adversarial network for fusion, making the fused image more realistic and facilitating more data analysis of the fused image to meet more usage requirements.
[0129] Another embodiment of the present invention provides a processor configured to execute the method for processing an image provided in the above embodiment.
[0130] In another embodiment of the present invention, a control device is provided, which is applied to household appliances. The control device includes a first image acquisition device, a second image acquisition device and a processor. The first image acquisition device and the second image acquisition device are both electrically connected to the processor, wherein the first image acquisition device is used to take a picture of a first image acquisition area; the second image acquisition device is used to take a picture of a second image acquisition area; and the processor is the processor provided in the above embodiment.
[0131] Another embodiment of the present invention provides a household appliance, which includes the control device provided in the above embodiment.
[0132] In one embodiment of the present invention, the household appliance includes one of a range hood 1 or an integrated stove, and the control device 3 includes a first image acquisition device and a second image acquisition device. In this embodiment, the first image acquisition device is a first camera 31, and the second image acquisition device is a second camera 32. The first camera 31 and the second camera 32 are respectively arranged on the left and right sides of the bottom of the range hood 1; an integrated stove is arranged below the range hood 1, and the integrated stove in this embodiment is the stove 2, then the first camera 31 is used to take pictures of the left area of the stove 2, and the image taken by the first camera 31 is the first image; the second camera 32 is used to take pictures of the right area of the stove 2, and the image taken by the second camera 32 is the second image, and the image acquisition areas of the first camera 31 and the second camera 32 have overlapping parts, so the first image and the second image have an intersecting part, and the intersecting part is the area between the left and right sides of the stove 2.
[0133] In one embodiment of the present invention, the control device 3 controls the stove 2 according to the first image and the second image taken, such as Figure 3 As shown, the control process includes the following steps:
[0134] Step 601: Determine whether the stove 2 enters the cooking mode.
[0135] It can be understood that the stove 2 in this embodiment includes a switch, which is electrically connected to the processor 34 in the control device 3. After the user turns on the stove 2 through the switch, the processor 34 will receive a signal that the stove 2 is turned on, and the processor 34 determines that the stove 2 enters the cooking mode based on the signal.
[0136] Step 602: Control the first image acquisition device and the second image acquisition device to start;
[0137] Step 603: synchronously acquire the first image and the second image.
[0138] It can be understood that after determining that the stove 2 enters the cooking mode, the processor 34 controls the first image acquisition device and the second image acquisition device to turn on, that is, controls the first camera 31 and the second camera 32 to turn on. After the first camera 31 and the second camera 32 are turned on, the processor 34 controls the first camera 31 and the second camera 32 to continuously capture the image of the stove 2 below. Further, the control device 3 in this embodiment also includes a memory with the processor 34. After the first camera 31 and the second camera 32 capture the first image and the second image, they are transferred to the memory for storage. When storing, the first image and the second image are named according to a specific naming rule, wherein the naming rule refers to naming the first image and the second image according to the shooting device and shooting time corresponding to the first image and the second image; the processor 34 can obtain the synchronized first image and the second image by querying the naming of the image.
[0139] Step 604: Generate a panoramic image of the stove 2 according to the first image and the second image.
[0140] It can be understood that the processor 34 includes a generator of a generative adversarial network, which is a maturely trained network model. After acquiring the first image and the second image, the processor 34 inputs them into the generator, and the generator generates a panoramic image of the stove 2 based on the first image and the second image.
[0141] Step 605: Sending service instruction information to the user according to the panoramic image of the stove 2.
[0142] It can be understood that the processor 34 analyzes the panoramic image of the stove 2 after it is generated, so as to extract effective information therefrom and guide the user's cooking operations based on the above effective information; further, the control device 3 also includes a display module or a voice module electrically connected to the processor 34. The processor 34 can display images or text information to the user by controlling the display module, or control the voice module to play voice information to prompt the user, thereby improving the user's usage experience.
[0143] Step 606: Determine whether the stove 2 exits the cooking mode;
[0144] Step 607: Control the first image acquisition device and the second image acquisition device to be turned off.
[0145] It can be understood that after the user turns off the switch of the stove 2, the processor 34 will receive a signal that the stove 2 is turned off. The processor 34 determines that the stove 2 exits the cooking mode based on the signal. At this time, the first camera 31 and the second camera 32 no longer need to continue to collect the image of the stove 2 below. Therefore, the processor 34 controls the first camera 31 and the second camera 32 to turn off after determining that the stove exits the cooking mode.
[0146] In one embodiment of the present invention, sending service instruction information to the user according to the panoramic image of the stove 2 includes the following steps:
[0147] Step 701: Obtain the user's cooking needs.
[0148] It can be understood that the stove 2 also includes a control panel electrically connected to the processor 34, and a plurality of function buttons (such as cooking, making soup, or cooking porridge, etc.) are provided on the control panel. After the user presses a function button, the control panel sends a corresponding function signal to the processor 34. After receiving the function signal, the processor 34 can determine the function selected by the user when cooking, thereby indirectly obtaining the user's cooking needs.
[0149] Step 702: Obtain a cooking demand analysis model according to cooking demand;
[0150] Step 703: Use the cooking demand analysis model to process and analyze the panoramic image of the stove 2.
[0151] It can be understood that a cooking demand analysis model is pre-stored in the processor 34. After obtaining the user's cooking demand, the processor 34 retrieves the cooking demand analysis model, and uses the cooking demand analysis model to process and analyze the generated panoramic image of the stove 2, that is, to extract effective information (such as food information, the degree of doneness of the food, the intensity of the fire, the user's posture or the user's identity, etc.) from the panoramic image of the stove 2 according to the user's cooking needs.
[0152] Step 704: Sending service instruction information to the user according to the result of the processing and analysis of the panoramic image of the stove 2.
[0153] It can be understood that after the processor 34 extracts valid information from the panoramic image of the stove 2, it sends service instruction information to the user based on the valid information. For example, when the user is cooking, the processor 34 extracts information from the panoramic image of the stove 2 that the ingredients in the pot are already eight-tenths cooked, and then prompts the user through the display module or the voice module to lower the heat to avoid overcooking the ingredients due to long-term high heat and affecting the taste; or, while the user is cooking, the processor 34 extracts information from the panoramic image of the stove 2 that a child is close to the stove 2, and then prompts the user through the display module or the voice module to pay attention to child safety to prevent children from accidentally touching the knives or getting close to the fire source and getting injured.
[0154] Another embodiment of the present invention provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the above-mentioned method for processing an image.
[0155] Another embodiment of the present invention provides a computer program product, comprising a computer program, which implements the above-mentioned method for processing an image when executed by a processor.
[0156] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0157] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0158] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0160] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0161] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0162] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0163] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0164] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
Claims
1. A method for processing an image, applied to a range hood or an integrated stove, characterized in that: include: Acquire a first image obtained by taking a picture of a first image acquisition area by a first image acquisition device, and a second image obtained by taking a picture of a second image acquisition area by a second image acquisition device, wherein the first image and the second image have an intersecting portion; Inputting the first image and the second image into a generator of a generative adversarial network respectively to generate a panoramic image that fuses the first image and the second image; Acquire an image training sample set, wherein the image training sample set includes a plurality of image triplets, each image triplet includes a first training image taken by the first image acquisition device, a second training image taken by the second image acquisition device, and a training panoramic image covering the first image acquisition area and the second image acquisition area; The first training image and the second training image in the image triplet are input into the generator to generate a panoramic image fused with the first training image and the second training image; wherein, The step of inputting the first training image and the second training image in the image triplet into the generator to generate a panoramic image fused with the first training image and the second training image comprises: Inputting the first training image and the second training image into the generator; Downsampling the first training image and the second training image input to extract feature maps, Wherein, the downsampling includes a multi-layer convolution operation, and the step size of the convolution operation is 2; The feature map is upsampled to generate a panoramic image that fuses the first training image and the second training image. picture.
2. The method according to claim 1, characterized in that Also includes: Inputting the generated panoramic image and the training panoramic image into a discriminator of a generative adversarial network to output a result; Calculating a loss function of the generative adversarial network, the loss function being associated with the generator and the discriminator; When the loss function satisfies the expected value, the training of the generator and the discriminator is terminated.
3. The method according to claim 2, characterized in that The step of downsampling the first training image and the second training image to extract a feature map includes: Downsampling the first training image and the second training image respectively to extract a first feature map and a second feature map respectively; The first feature map and the second feature map are processed to generate a feature map.
4. The method according to claim 3, characterized in that The processing of the first feature map and the second feature map to generate a feature map includes: concatenating the first feature map and the second feature map to generate a feature map; or The first feature map and the second feature map are added to generate a feature map.
5. The method according to claim 4, characterized in that The adding the first feature map and the second feature map to generate a feature map includes: Adding the first characteristic map and the second characteristic map to obtain a third characteristic map; The height and the number of channels of the third feature map are changed to generate a feature map.
6. The method according to claim 2, characterized in that The step of inputting the first training image and the second training image in the image triplet into the generator to generate a panoramic image fused with the first training image and the second training image further includes: Performing a convolution operation on the feature map to perform a nonlinear transformation without changing the feature size; The upsampling of the feature map to generate a panoramic image that fuses the first training image and the second training image includes: upsampling the feature map that has undergone nonlinear transformation to generate a panoramic image that fuses the first training image and the second training image.
7. The method according to claim 2, characterized in that The upsampling includes a combination of multi-layer bilinear interpolation and convolution operations.
8. The method according to claim 1, characterized in that Also includes: Control the first image acquisition device and the second image acquisition device to shoot synchronously.
9. A processor, characterized in that: The device is configured to perform the method for processing an image according to any one of claims 1 to 8.
10. A control device, applied to a range hood or an integrated stove, characterized in that: The control device comprises: A first image acquisition device, used to take a picture of a first image acquisition area; A second image acquisition device, used to take a picture of a second image acquisition area; and A processor according to claim 9.
11. A household appliance, characterized in that: Including the control device according to claim 10, the household appliance includes at least one of the range hood or the integrated stove.
12. A machine-readable storage medium having instructions stored thereon, characterized in that: The instructions are used to enable a machine to execute the method for processing an image according to any one of claims 1 to 8.
