Home appliance control method, image processing method, home appliance, and server

Abstract

This application proposes a control method for a home appliance, an image processing method, a home appliance, and a server. The control method for the home appliance includes: obtaining a first image of the home appliance; performing mask occlusion processing on the first image to generate a second image; and sending the second image to a server so that the server can obtain the first image based on the second image. In the technical solution of this application, on the one hand, the home appliance can transmit the image to the server, and the server performs image processing, which reduces the burden of image processing on the home appliance, improves the effect of image processing, and ensures the basic operation of the home appliance. On the other hand, through mask occlusion processing, only a small portion of the image data needs to be transmitted, and it is not necessary to transmit the entire image to the server. Therefore, it can greatly reduce the pressure on network transmission and shorten the image transmission time.

Description

Technical Field

[0001] The present application relates to the field of intelligent recommendation technology, and specifically to a control method for household appliances, an image processing method, a control device for household appliances, an image processing device, household appliances, a server, and a readable storage medium. Background Art

[0002] With the vigorous development of artificial intelligence technology, users' demand for smart homes is becoming increasingly strong. A home appliance with artificial intelligence functions can greatly improve product quality and user experience.

[0003] For example, controlling home appliances through their captured images is often expensive due to the complex processes involved in designing and modifying the size and structure of the appliances, as well as the increasing cost of smart chips. Consequently, image processing on remote servers is often performed, but the resulting network transmission burden is a pressing issue. Summary of the Invention

[0004] This application aims to solve at least one of the technical problems existing in the prior art or related art.

[0005] To this end, the first aspect of the present application is to propose a control method for household electrical appliances.

[0006] The second aspect of this application is to propose an image processing method.

[0007] The third aspect of the present application is to propose a control device for household appliances.

[0008] The fourth aspect of the present application is to propose an image processing device.

[0009] The fifth aspect of the present application is to propose a household appliance.

[0010] The sixth aspect of the present application is to propose a server.

[0011] The seventh aspect of the present application is to propose a readable storage medium.

[0012] In view of this, according to one aspect of the present application, a control method for a household appliance is proposed, including: obtaining a first image of the household appliance; performing masking processing on the first image to generate a second image; and sending the second image to a server so that the server can obtain the first image based on the second image.

[0013] In this technical solution, the home appliance is installed with an image acquisition device capable of acquiring a first image of the home appliance, or the home appliance is installed with a communication device capable of receiving a first image provided by an external image acquisition device.

[0014] It should be noted that household appliances include dishwashers, range hoods, refrigerators, induction cookers, rice cookers, ovens, air fryers, air conditioners, televisions, etc., and the first image includes an internal image or an external image of the household appliance. For example, if the household appliance is a dishwasher, the first image is an internal image of the dishwasher (that is, an image including tableware placed inside the dishwasher); if the household appliance is a refrigerator, the first image is an internal image of the refrigerator (that is, an image including food placed inside the refrigerator); if the household appliance is an air conditioner, the first image is an overall image of the air conditioner.

[0015] Furthermore, the first image is masked, that is, a part of the sub-image in the first image is masked using a preset mask to obtain a second image, and then the second image is transmitted to the server, so that the server can restore the first image based on the second image, and then perform image recognition to obtain control parameters for the home appliance.

[0016] In the technical solution of the present application, on the one hand, the home appliance can transmit the image to the server and perform image recognition on the server, which reduces the burden of image recognition on the home appliance, improves the effect of image recognition, and ensures the basic operation of the home appliance; on the other hand, through mask occlusion processing, only a small part of the image data needs to be transmitted, and there is no need to transmit the entire image to the server, which can greatly reduce the network transmission pressure and shorten the image transmission time.

[0017] The control method of the household appliance according to the present application may also have the following additional technical features:

[0018] In the above technical solution, the method further includes: receiving control parameters from a server; controlling the operation of the home appliance according to the control parameters; wherein the control parameters are generated by the server based on the first image.

[0019] In this technical solution, after receiving the second image, the server can restore the first image based on the second image, further perform image processing, obtain control parameters for the home appliance, and send the control parameters to the home appliance. After receiving the control parameters, the home appliance operates according to the control parameters.

[0020] For example, when the household appliance is a dishwasher, the image of tableware inside the dishwasher (i.e., the first image) is monitored in real time, and the tableware image is masked and sent to the server. The server restores the image and performs image recognition to obtain the control parameters of the dishwasher, such as the flushing intensity, oil removal degree, and tableware protection, so as to automatically adjust the flushing intensity of the dishwasher, control oil stains, and protect tableware.

[0021] Through the above method, it is possible to realize automatic control of home appliances based on image recognition of home appliances, improve the control effect of home appliances, and facilitate user use.

[0022] In any of the above technical solutions, the first image is subjected to masking and occlusion processing to generate a second image, including: identifying the background image and the target object image of the first image; performing masking and occlusion processing on the background image without intervals through a preset mask, and performing masking and occlusion processing on the target object image according to preset intervals to generate a second image.

[0023] In this technical solution, the first image includes a background image and a target object image, where the background image is an invalid area and the target object image is a valid area. For example, in an image of the interior of a dishwasher, the dishware is the target object, and the image of the dishwasher's internal shelves or the inner wall is the background image; in an image of an air conditioner, the air conditioner itself is the target object, and the image of the wall is the background image.

[0024] If the first image is an image with a width and height of W×H, a preset mask of m×m (e.g., 16×16) is used to fill the first image in a regular manner. Specifically, the invalid areas of the first image are uniformly filled with a preset mask of m×m without any spacing. The valid areas of the first image are filled with a preset mask of m×m at preset intervals. The preset intervals can be the same as the preset mask width and height, or multiples of the preset mask width or height.

[0025] Through the above method, the invalid area of the first image is completely masked and the valid area is masked at intervals, so that the image in the invalid area is not transmitted at all and a small part of the valid image data is transmitted. This can greatly reduce the network transmission pressure and improve the image transmission speed.

[0026] In any of the above technical solutions, sending the second image to the server includes: generating a sub-image queue based on multiple sub-images included in the second image that are not blocked by preset masks; and sending the sub-image queue to the server.

[0027] In this technical solution, the sub-images in the first image that are not blocked by the preset mask are regularly arranged one by one to generate a sub-image queue, which is then transmitted as a whole to a remote server.

[0028] In the technical solution of the present application, on the one hand, these sub-images are much smaller than the size of the original first image, which will greatly reduce the network transmission pressure and speed up the image transmission speed; on the other hand, the sub-images are sent to the server in a queue, so that the server can determine the order of the sub-images and achieve accurate image restoration.

[0029] According to the second aspect of the present application, an image processing method is proposed, comprising: receiving a second image from a household appliance, wherein the second image is obtained by the household appliance performing mask occlusion processing on the first image; and generating the first image based on the second image.

[0030] In this technical solution, the home appliance masks the first image to obtain a second image, which is then transmitted to a server. After receiving the second image, the server can restore the masked portion of the second image to obtain the first image, and then perform image recognition on the first image.

[0031] In this application's technical solution, on the one hand, the home appliance uses masking to process the image, requiring it to transmit only a small portion of the image data, rather than the entire image, to the server. This significantly reduces network transmission pressure and shortens image transmission time. On the other hand, the server deploys a decoding network to recover the first image and then perform image recognition, reducing the burden on the home appliance for image recognition, improving the image recognition effect, and ensuring the basic operation of the home appliance.

[0032] The image processing method of the present application may also have the following additional technical features:

[0033] In the above technical solution, the method further includes: determining control parameters of the home appliance according to the first image.

[0034] In this technical solution, image recognition is performed on the first image to obtain information such as control parameters and current working status of the home appliance.

[0035] Furthermore, the control parameters can be sent to the home appliances to control the home appliances to operate according to the control parameters, thereby realizing intelligent control of the home appliances.

[0036] In addition, the current status of the home appliance may be sent to a user terminal associated with the home appliance, so that the user can understand the current working status of the home appliance.

[0037] In any of the above technical solutions, the second image is a sub-image queue of multiple sub-images that are not blocked by a preset mask; based on the second image, a first image is generated, including: based on the preset mask and the sub-image queue, a target image queue is generated; based on the target image queue, a first embedding vector is generated, and based on the first embedding vector, a first position coding vector is generated according to a preset coding method, wherein the preset coding method is to use sine coding for the even positions of the first embedding vector and cosine coding for the odd positions of the first embedding vector; the first embedding vector and the first position coding vector are added to generate a first target vector; the first target vector is decoded by a decoding model to generate a first image.

[0038] In this technical solution, the home appliance arranges the sub-images in the first image that are not blocked by the preset mask one by one in a regular manner to generate a sub-image queue, and transmits the sub-image queue as a whole to the server.

[0039] The server receives the sub-image queue transmitted from the home appliance and sequentially inserts a preset m×m mask into each row from left to right and top to bottom according to pre-defined positions, thus forming a new target image queue. The target image queue then undergoes linear projection to generate n corresponding first embedding vectors.

[0040] For these n first embedding vectors, the corresponding first position encoding vectors are generated according to the preset encoding method (i.e., formula (1)). Formula (1) is:

[0041]

[0042] Here, p refers to the element in the first position encoding vector, k refers to the position of the current vector within the entire n first embedding vectors, and i refers to the index of each value in the first embedding vector. That is, sine encoding is used at even positions and cosine encoding is used at odd positions. d = 768 represents the length of the first embedding vector.

[0043] Furthermore, the first embedding vector is summed with its corresponding first position encoding vector to obtain a first target vector, so that each sub-image contains its position information in the image. The first target vector is then input into the decoding model to restore the original first image.

[0044] Through the above method, accurate restoration of the first image is achieved.

[0045] In any of the above technical solutions, the method also includes: obtaining a first sampling image, and performing occlusion processing on the first sampling image through a preset mask to generate a second sampling image; generating a second embedding vector based on the second sampling image, and generating a second position coding vector based on the second embedding vector; adding the second embedding vector and the second position coding vector to generate a second target vector; inputting the second target vector into a preset model and outputting a training image; subtracting multiple pixel values of the training image from multiple pixel values of the first sampling image to obtain multiple difference values, and summing the multiple difference values to generate a target loss function; training the preset model according to the target loss function to generate a decoding model.

[0046] In this technical solution, a first sampling image of a series of household appliances is obtained, and the first sampling image is masked to obtain a second sampling image. The sub-images of the second sampling image are arranged in a queue, and the sub-image queue of the second sampling image is inserted into a preset mask to form a new image queue. The image queue is then linearly projected to generate n corresponding second embedding vectors. For the second embedding vector, a corresponding second position coding vector is generated according to a preset coding method. The second embedding vector and the corresponding second position coding vector are summed to obtain a second target vector, so that each sub-image contains its position information in the image. The second target vector is then input into the decoding model to output a training image, which is the image restored from the first sampling image.

[0047] Furthermore, the differences between multiple pixel values of the training image and multiple pixel values of the first sampling image are calculated respectively to obtain multiple differences, and then the sum of these multiple differences is obtained, that is, the target loss function is obtained. Finally, the preset model is trained using the target loss function so that the final target loss function of the model reaches the ideal threshold or after a specified round of training, the decoding model is obtained.

[0048] Through this approach, a precise decoding model is established, enabling accurate restoration of masked images. Furthermore, this model-building method only requires a sampled image as input, without requiring additional parameters or human assistance, to generate a decoding model. This streamlines training and deployment, significantly reducing the cost of the entire framework. It also improves the quality of the first image, ensuring high-quality image input for subsequent imaging tasks.

[0049] According to the third aspect of the present application, a control device for household appliance is proposed, including: an acquisition module for acquiring a first image of the household appliance; a processing module for performing masking processing on the first image to generate a second image; and a sending module for sending the second image to a server so that the server can acquire the first image based on the second image.

[0050] In this technical solution, an image acquisition device is installed on the home appliance, which can acquire a first image of the home appliance, or a communication device is installed on the home appliance, which can receive a first image provided by an external image acquisition device.

[0051] It should be noted that household appliances include dishwashers, range hoods, refrigerators, induction cookers, rice cookers, ovens, air fryers, air conditioners, televisions, etc., and the first image includes an internal image or an external image of the household appliance. For example, if the household appliance is a dishwasher, the first image is an internal image of the dishwasher (that is, an image including tableware placed inside the dishwasher); if the household appliance is a refrigerator, the first image is an internal image of the refrigerator (that is, an image including food placed inside the refrigerator); if the household appliance is an air conditioner, the first image is an overall image of the air conditioner.

[0052] Furthermore, the first image is masked, that is, a part of the sub-image in the first image is masked using a preset mask to obtain a second image, and then the second image is transmitted to the server, so that the server can restore the first image based on the second image, and then perform image processing to obtain control parameters for the home appliance.

[0053] In the technical solution of this scheme, on the one hand, home appliances can transmit images to the server and perform image processing on the server, which reduces the burden of image processing on home appliances, improves the effect of image processing, and ensures the basic operation of home appliances; on the other hand, through mask occlusion processing, only a small part of the image data needs to be transmitted, and there is no need to transmit the entire image to the server, which can greatly reduce network transmission pressure and shorten image transmission time.

[0054] According to the fourth aspect of the present application, an image processing device is proposed, including: a receiving module for receiving a second image from a household appliance, wherein the second image is obtained by the household appliance performing mask occlusion processing on the first image; and a generating module for generating a first image based on the second image.

[0055] In this technical solution, the home appliance masks the first image to obtain a second image, which is then transmitted to a server. After receiving the second image, the server can restore the masked portion of the second image to obtain the first image, and then perform image recognition on the first image.

[0056] In this application's technical solution, on the one hand, the home appliance uses masking to process the image, requiring it to transmit only a small portion of the image data, rather than the entire image, to the server. This significantly reduces network transmission pressure and shortens image transmission time. On the other hand, the server deploys a decoding network to recover the first image and then perform image recognition, reducing the burden on the home appliance for image recognition, improving the image recognition effect, and ensuring the basic operation of the home appliance.

[0057] According to the fifth aspect of the present application, a household appliance is proposed, comprising: a memory storing programs or instructions; and a processor, which implements the steps of a method for controlling the household appliance of any of the above technical solutions when executing the programs or instructions.

[0058] The home appliance provided by the present application implements the steps of the image processing method of any of the above technical solutions when the processor executes the computer program. Therefore, the home appliance includes all the beneficial effects of the image processing method of any of the above technical solutions.

[0059] The above-mentioned household appliance according to the present application may also have the following additional technical features:

[0060] In the above technical solution, the home appliance further includes: a communication device connected to the processor, configured to send the second image to the server and receive control parameters from the server.

[0061] In this technical solution, the home appliance further includes a communication device connected to the processor, which can send a second image and receive control parameters to and from the server, thereby ensuring normal data interaction between the home appliance and the server.

[0062] In any of the above technical solutions, the processor is a single-chip microcomputer.

[0063] In this technical solution, the microcontroller is a low-cost chip that can reduce the cost of home appliances when used in them. However, due to its inherent computing power limitations, it cannot run complex models.

[0064] The technical solution of this application proposes an image mask-based image compression method that can be run on a single-chip microcomputer. The image is masked locally on the home appliance and then transmitted to the server for image restoration. Because the server has sufficient computational examples, it can use a deep learning-based self-encoding and decoding network to restore the image.

[0065] Therefore, the processor in the home appliance can directly utilize the microcontroller to reduce the cost of the home appliance.

[0066] According to the sixth aspect of the present application, a server is proposed, comprising: a memory storing programs or instructions; and a processor, which implements the steps of the image processing method of any of the above technical solutions when executing the programs or instructions.

[0067] The server provided by the present application implements the steps of the image processing method of any of the above technical solutions when the processor executes the computer program, so the server includes all the beneficial effects of the image processing method of any of the above technical solutions.

[0068] According to the seventh aspect of the present application, a readable storage medium is proposed, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the control method of the household appliance device as in any of the above-mentioned technical solutions or the steps of the image processing method as in any of the above-mentioned technical solutions are implemented.

[0069] The readable storage medium provided in this application implements the steps of the control method of the household appliance device as any of the above-mentioned technical solutions when the program or instruction is executed by the processor. Therefore, the readable storage medium includes all the beneficial effects of the control method of the household appliance device as any of the above-mentioned technical solutions.

[0070] The readable storage medium provided by the present application implements the steps of the image processing method of any of the above technical solutions when the program or instruction is executed by the processor. Therefore, the readable storage medium includes all the beneficial effects of the image processing method of any of the above technical solutions.

[0071] Additional aspects and advantages of the present application will become apparent in the following description or may be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0073] Figure 1 A flowchart of a method for controlling a household appliance according to an embodiment of the present application is shown;

[0074] Figure 2 A second flow chart of a method for controlling a household appliance according to an embodiment of the present application is shown;

[0075] Figure 3 A schematic diagram showing an original image of a scene inside a dishwasher according to an embodiment of the present application;

[0076] Figure 4 A schematic diagram showing masking of an invalid area of an original image according to an embodiment of the present application is shown;

[0077] Figure 5One of the schematic diagrams showing masking of a valid area of an original image according to an embodiment of the present application is shown;

[0078] Figure 6 A schematic diagram showing an image queue according to an embodiment of the present application is shown;

[0079] Figure 7 A second schematic diagram showing masking of a valid area of an original image according to an embodiment of the present application is shown;

[0080] Figure 8 FIG1 shows one of the flow charts of the image processing method according to an embodiment of the present application;

[0081] Figure 9 FIG2 shows a second flow chart of the image processing method according to an embodiment of the present application;

[0082] Figure 10 A logical diagram of image restoration according to an embodiment of the present application is shown;

[0083] Figure 11 A schematic diagram showing a new image queue according to an embodiment of the present invention;

[0084] Figure 12 A schematic diagram of decoding logic of a decoder according to an embodiment of the present invention is shown;

[0085] Figure 13 A schematic diagram of the working logic of the self-attention module according to an embodiment of the present invention is shown;

[0086] Figure 14 A schematic block diagram showing a control device for a household appliance according to an embodiment of the present invention is shown;

[0087] Figure 15 A schematic block diagram of an image processing apparatus according to an embodiment of the present invention is shown;

[0088] Figure 16 A schematic block diagram of a household appliance according to an embodiment of the present invention is shown;

[0089] Figure 17 A schematic block diagram of a server according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0090] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0091] It should be noted that all directional indications in the embodiments of the present application (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0092] In addition, the terms "first," "second," and so on, used in this application are for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly specifying the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0093] In this application, unless otherwise specified or limited, the terms "connection" and "fixation" should be understood in a broad sense. For example, "fixation" can mean fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; internal communication between two elements or interaction between two elements, unless otherwise specified. For those skilled in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0094] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in this field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0095] Below, in combination with the accompanying drawings, the control method of the home appliance, the image processing method, the control device of the home appliance, the image processing device, the home appliance, the server and the readable storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0096] Example 1

[0097] The embodiment of the present invention provides a control method for household electrical appliances. Figure 1 A flow chart of a method for controlling a household appliance according to an embodiment of the present application is shown. The method includes:

[0098] Step 102: Acquire a first image of the home appliance;

[0099] Step 104: performing masking processing on the first image to obtain a second image;

[0100] Step 106: Send the second image to the server, so that the server restores the first image based on the second image.

[0101] In this technical solution, the home appliance is installed with an image acquisition device capable of acquiring a first image of the home appliance, or the home appliance is installed with a communication device capable of receiving a first image provided by an external image acquisition device.

[0102] It should be noted that household appliances include dishwashers, range hoods, refrigerators, induction cookers, rice cookers, ovens, air fryers, air conditioners, televisions, etc., and the first image includes an internal image or an external image of the household appliance. For example, if the household appliance is a dishwasher, the first image is an internal image of the dishwasher (that is, an image including tableware placed inside the dishwasher); if the household appliance is a refrigerator, the first image is an internal image of the refrigerator (that is, an image including food placed inside the refrigerator); if the household appliance is an air conditioner, the first image is an overall image of the air conditioner.

[0103] Furthermore, the first image is masked, that is, a part of the sub-image in the first image is masked using a preset mask to obtain a second image, and then the second image is transmitted to the server, so that the server can restore the first image based on the second image, and then perform image recognition to obtain control parameters for the home appliance.

[0104] In the technical solution of the present application, on the one hand, the home appliance can transmit the image to the server and perform image recognition on the server, which reduces the burden of image recognition on the home appliance, improves the effect of image recognition, and ensures the basic operation of the home appliance; on the other hand, through mask occlusion processing, only a small part of the image data needs to be transmitted, and there is no need to transmit the entire image to the server, which can greatly reduce the network transmission pressure and shorten the image transmission time.

[0105] Example 2

[0106] The embodiment of the present invention provides a control method for household electrical appliances. Figure 2 The second flow chart of the control method of the household appliance of the embodiment of the present application is shown. The method includes:

[0107] Step 202: Acquire a first image of the home appliance;

[0108] Step 204: performing masking processing on the first image to obtain a second image;

[0109] Step 206: sending the second image to the server so that the server can restore the first image based on the second image;

[0110] Step 208: Obtain control parameters sent by the server;

[0111] Step 210: Control the home appliance based on the control parameter.

[0112] The control parameter is generated by the server based on the first image.

[0113] In this technical solution, after receiving the second image, the server can restore the first image based on the second image, further perform image processing, obtain control parameters for the home appliance, and send the control parameters to the home appliance. After receiving the control parameters, the home appliance operates according to the control parameters.

[0114] For example, when the household appliance is a dishwasher, the image of tableware inside the dishwasher (i.e., the first image) is monitored in real time, and the tableware image is masked and sent to the server. The server restores the image and performs image recognition to obtain the control parameters of the dishwasher, such as the flushing intensity, oil removal degree, and tableware protection, so as to automatically adjust the flushing intensity of the dishwasher, control oil stains, and protect tableware.

[0115] Through the above method, it is possible to realize automatic control of home appliances based on image recognition of home appliances, improve the control effect of home appliances, and facilitate user use.

[0116] Example 3

[0117] In an embodiment of the present invention, the step of performing mask occlusion processing on the first image to obtain the second image specifically includes: determining the background image and the target object image in the first image; using a preset mask, performing mask occlusion processing on the background image without intervals, and at the same time, performing mask occlusion processing on the target object image according to preset intervals, to finally obtain the second image.

[0118] In this technical solution, the first image includes a background image and a target object image, where the background image is an invalid area and the target object image is a valid area. For example, in an image of the interior of a dishwasher, the dishware is the target object, and the image of the dishwasher's internal shelves or the inner wall is the background image; in an image of an air conditioner, the air conditioner itself is the target object, and the image of the wall is the background image.

[0119] If the first image is an image with a width and height of W×H, a preset mask of m×m (e.g., 16×16) is used to fill the first image in a regular manner. Specifically, the invalid areas of the first image are uniformly filled with a preset mask of m×m without any spacing. The valid areas of the first image are filled with a preset mask of m×m at preset intervals. The preset intervals can be the same as the preset mask width and height, or multiples of the preset mask width or height.

[0120] Through the above method, the invalid area of the first image is completely masked and the valid area is masked at intervals, so that the image in the invalid area is not transmitted at all and a small part of the valid image data is transmitted. This can greatly reduce the network transmission pressure and improve the image transmission speed.

[0121] Example 4

[0122] In the embodiment of the present invention, the step of sending the second image to the server specifically includes: generating a sub-image queue according to a plurality of sub-images in the second image that are not blocked by a preset mask, and sending the sub-image queue to the server.

[0123] In this technical solution, the sub-images in the first image that are not blocked by the preset mask are regularly arranged one by one to generate a sub-image queue, which is then transmitted as a whole to a remote server.

[0124] In the technical solution of the present application, on the one hand, these sub-images are much smaller than the size of the original first image, which will greatly reduce the network transmission pressure and speed up the image transmission speed; on the other hand, the sub-images are sent to the server in a queue, so that the server can determine the order of the sub-images and achieve accurate image restoration.

[0125] Example 5

[0126] In the embodiment of the present invention, a dishwasher is used as an example of a household appliance. A built-in camera is installed in the dishwasher. When the door of the dishwasher is closed, the back-end system (i.e., the single-chip microcomputer) of the dishwasher receives a signal and then initializes the camera. The camera starts to continuously transmit the collected image data to the back-end system for distribution and processing. The original image of the scene inside the dishwasher is as follows: Figure 3 shown.

[0127] Using its inherent rules to use mask to clear the original image, specifically including:

[0128] (1) Assuming that the original image is an image with a length and width of W×H, a 16×16 mask is used to fill the image regularly. Specifically, Figure 4 As shown in , the invalid area (ie background area) in the original image is uniformly masked in a non-interval manner. Figure 5 As shown, the effective area in the original image (ie, the image area of the tableware) can be masked in sequence using a step size with the same width and height as the mask.

[0129] (2) For the areas that are not masked in the original image, such as Figure 6 As shown, take it off piece by piece and photograph it into an image queue.

[0130] (3) The cut image queue is transmitted as a whole to a remote server.

[0131] These image blocks are much smaller than the original image size, thus greatly reducing the network transmission pressure and speeding up the image transmission.

[0132] It should be noted that there is no limitation on the way to use the mask to mask the valid area in the original image. For the dishwasher, a large number of dense masks are used for the areas of interest in the image. For applications where the entire image is of interest, the following can be used: Figure 7 The encoding shown may also adopt a random masking method according to a certain percentage, etc. The masking method can be flexibly selected according to the actual application scenario and the corresponding training process can be adjusted.

[0133] It should be noted that the image mask framework proposed in the embodiment of the present application can also be applied to any application involving image transmission, such as video conferencing, video surveillance, video chat, short videos, online audio and video playback, online live broadcast, etc.

[0134] Example 6

[0135] The embodiment of the present invention proposes an image processing method. Figure 8 A flowchart of an image processing method according to an embodiment of the present application is shown. The method includes:

[0136] Step 802: Acquire a second image sent by the home appliance, where the second image is generated by the home appliance performing masking processing on the first image;

[0137] Step 804: Decode the second image to obtain the first image.

[0138] In this technical solution, the home appliance masks the first image to obtain a second image, which is then transmitted to a server. After receiving the second image, the server can restore the masked portion of the second image to obtain the first image, and then perform image recognition on the first image.

[0139] In this application's technical solution, on the one hand, the home appliance uses masking to process the image, requiring it to transmit only a small portion of the image data, rather than the entire image, to the server. This significantly reduces network transmission pressure and shortens image transmission time. On the other hand, the server deploys a decoding network to recover the first image and then perform image recognition, reducing the burden on the home appliance for image recognition, improving the image recognition effect, and ensuring the basic operation of the home appliance.

[0140] Example 7

[0141] The embodiment of the present invention proposes an image processing method. Figure 9 The second flow chart of the image processing method according to the embodiment of the present application is shown. The method includes:

[0142] Step 902: Acquire a second image sent by the home appliance, where the second image is generated by the home appliance performing masking processing on the first image;

[0143] Step 904: Decode the second image to obtain the first image;

[0144] Step 906: Identify the first image and obtain control parameters of the home appliance.

[0145] In this technical solution, image recognition is performed on the first image to obtain information such as control parameters and current working status of the home appliance.

[0146] Furthermore, the control parameters can be sent to the home appliances to control the home appliances to operate according to the control parameters, thereby realizing intelligent control of the home appliances.

[0147] In addition, the current status of the home appliance may be sent to a user terminal associated with the home appliance, so that the user can understand the current working status of the home appliance.

[0148] Example 8

[0149] In the embodiment of the present invention, the second image refers to a sub-image queue of multiple sub-images in the first image that are not blocked by the preset mask.

[0150] The step of decoding the second image to obtain the first image specifically includes: generating a target image queue based on a preset mask and a sub-image queue; obtaining a first embedding vector based on the target image queue, and generating a first position coding vector based on the first embedding vector in a preset coding method; adding the first embedding vector and the generated first position coding vector to obtain a first target vector; and decoding the first target vector using a decoding model to obtain the first image.

[0151] The preset encoding method refers to a method of using sine encoding for even positions of the first embedded vector and using cosine encoding for odd positions of the first embedded vector.

[0152] In this technical solution, the home appliance arranges the sub-images in the first image that are not blocked by the preset mask one by one in a regular manner to generate a sub-image queue, and transmits the sub-image queue as a whole to the server.

[0153] The server receives the sub-image queue transmitted from the home appliance and sequentially inserts a preset m×m mask into each row from left to right and top to bottom according to pre-defined positions, thus forming a new target image queue. The target image queue then undergoes linear projection to generate n corresponding first embedding vectors.

[0154] For these n first embedding vectors, the corresponding first position encoding vectors are generated according to the preset encoding method (i.e., formula (1)). Formula (1) is:

[0155]

[0156] Here, p refers to the element in the first position encoding vector, k refers to the position of the current vector within the entire n first embedding vectors, and i refers to the index of each value in the first embedding vector. That is, sine encoding is used at even positions and cosine encoding is used at odd positions. d = 768 represents the length of the first embedding vector.

[0157] Furthermore, the first embedding vector is summed with its corresponding first position encoding vector to obtain a first target vector, so that each sub-image contains its position information in the image. The first target vector is then input into the decoding model to restore the original first image.

[0158] Through the above method, accurate restoration of the first image is achieved.

[0159] Embodiment 9

[0160] In an embodiment of the present invention, the method also includes: obtaining a first sampling image, and performing masking processing on the first sampling image through a preset mask to generate a second sampling image; generating a second embedding vector based on the second sampling image, and generating a second position coding vector based on the second embedding vector; then adding the second embedding vector and the second position coding vector to obtain a second target vector; inputting the second target vector into a preset model and outputting a training image; respectively calculating the difference between multiple pixel values of the training image and multiple pixel values of the first sampling image to obtain multiple differences, and then summing these multiple differences to obtain a target loss function; and using the target loss function to train the preset model to obtain a decoding model.

[0161] In this technical solution, a first sampling image of a series of household appliances is obtained, and the first sampling image is masked to obtain a second sampling image. The sub-images of the second sampling image are arranged in a queue, and the sub-image queue of the second sampling image is inserted into a preset mask to form a new image queue. The image queue is then linearly projected to generate n corresponding second embedding vectors. For the second embedding vector, a corresponding second position coding vector is generated according to a preset coding method. The second embedding vector and the corresponding second position coding vector are summed to obtain a second target vector, so that each sub-image contains its position information in the image. The second target vector is then input into the decoding model to output a training image, which is the image restored from the first sampling image.

[0162] Furthermore, the differences between multiple pixel values of the training image and multiple pixel values of the first sampling image are calculated respectively to obtain multiple differences, and then the sum of these multiple differences is obtained, that is, the target loss function is obtained. Finally, the preset model is trained using the target loss function so that the final target loss function of the model reaches the ideal threshold or after a specified round of training, the decoding model is obtained.

[0163] Through this approach, a precise decoding model is established, enabling accurate restoration of masked images. Furthermore, this model-building method only requires a sampled image as input, without requiring additional parameters or human assistance, to generate a decoding model. This streamlines training and deployment, significantly reducing the cost of the entire framework. It also improves the quality of the first image, ensuring high-quality image input for subsequent imaging tasks.

[0164] Example 10

[0165] In the embodiment of the present invention, on the server side, transformer is used to restore the mask part of the image. The specific logical process is as follows: Figure 10 Shown, including:

[0166] (1) Receive image queues from home appliances.

[0167] (2) According to the previously set mask position, insert a 16×16 size mask in each row from left to right and from top to bottom, so that Figure 11 As shown, a new image queue is formed, and each 16×16 image is an image block.

[0168] (3) Each image block is a 16×16 color 3-channel image, which is flattened into a one-dimensional vector, thereby obtaining n vectors of size 16×16×3=768.

[0169] (4) These n 768-dimensional vectors are linearly projected to generate n corresponding embedding vectors, which are also 768-dimensional.

[0170] (5) For these n 768-dimensional embedding vectors, the corresponding position encoding vector is generated according to formula (1), and the position encoding vector is also 768-dimensional.

[0171] (6) The 768-dimensional embedding vector and its corresponding position encoding vector are summed, so that each image block contains the position information in the image.

[0172] (7) These n 768-dimensional vectors containing position codes are input into the decoder (also known as the decoding model) to restore the original image.

[0173] (8) The restored image is input into various subsequent image recognition tasks.

[0174] The decoding logic of the entire Decoder is as follows Figure 12 As shown, the decoding logic process includes:

[0175] (1) Assume X1...X n Is a 768-dimensional vector obtained after position encoding fusion, and these n vectors containing position encoding information are input into the self-attention module. Among them, the working logic of the self-attention module is as follows: Figure 13 shown.

[0176] (2) The self-attention module contains a multi-head module, each of which is concerned with each input feature vector X i The eigenvectors of i 、k i 、v i There are 3 vectors in total, which are used as query, dictionary and value vectors respectively. Here, we explain a multi-head module and the first output Z1. iFinally, we get Z1...Z which are related to each other. n .

[0177] Specifically, each X i will generate the corresponding q i 、k i 、v i , each q i Will be with different k i Do the inner product, for example, q 1 Will and each k i Do the inner product, so n inner product values α will be generated 1,i ,in These n inner product values are normalized by classification (softmax) to obtain n normalized values in Next will and v i Multiply and add to get Z1, as shown in the following formula (2):

[0178]

[0179] In this way, we will get Z1...Z n , each Z i Will contain relationships with each other.

[0180] Input X1......X n and Z1......Z n Summation, this operation is the classic residual structure.

[0181] The summed result is normalized by Layer norm to obtain the new Z1...Z n .

[0182] The normalized Z1...Z n , after a forward MLP (Multilayer Perceptron, fully connected network), the results are then normalized and the Z1...Z n Sum the residuals and then perform layer norm normalization to obtain n outputs.

[0183] The n outputs of the previous step are used as inputs and repeatedly fed into the multi-head module. According to the experience of transformers, this operation is repeated 6 times, resulting in n 768-dimensional vectors, assuming Y1...Y n .

[0184] Then for each Y iPerform a reshape operation to turn it into a 3-channel color image of 16×16×3.

[0185] Finally, these n 16×16 images are stitched together in sequence to obtain the restored original image.

[0186] Example 11

[0187] In this embodiment of the present invention, the entire decoder uses the transformer as the basic framework and is trained in an unsupervised training mode. Taking a dishwasher as an example, the method for training the decoder includes:

[0188] (1) Take a series of original images inside a dishwasher.

[0189] (2) Perform mask processing on the original image.

[0190] (3) The masked image is then cut into image blocks.

[0191] (4) Flatten each image block into a one-dimensional vector.

[0192] (5) Obtain the embedded vector through linear projection.

[0193] (6) Calculate the position encoding corresponding to each embedded vector.

[0194] (7) Sum the embedding vector and the position encoding vector.

[0195] (8) The embedded vector of the position coding information obtained in the previous step is sent to the decoder to obtain the output vector.

[0196] (9) The obtained output vector is reshaped into a small image with the same size as the original image block.

[0197] (10) All small images are reassembled in order into an image of the same size as the original image.

[0198] (11) Sum the difference between each pixel of the newly generated image and the original image, and the resulting sum is used as the final target loss function.

[0199] (12) The entire decoder is trained according to the target loss function so that the final target loss function of the decoder reaches the ideal threshold or specified rounds.

[0200] (13) Save the decoder and deploy it to the server.

[0201] In this embodiment, the entire decoder training does not require data labeling, thereby greatly reducing the cost of manual labeling and making the training deployment extremely streamlined.

[0202] Example 12

[0203] The embodiment of the present invention provides a control device for household appliances. Figure 14 A schematic block diagram of a control device 1400 for a household appliance according to an embodiment of the present invention is shown. The control device 1400 for the household appliance includes an acquisition module 1402 , a processing module 1404 and a sending module 1406 .

[0204] Among them, the acquisition module 1402 can acquire a first image of the home appliance, the processing module 1404 can mask the first image to obtain a second image, and the sending module 1406 can send the second image to the server so that the server can restore the first image based on the second image.

[0205] In this technical solution, an image acquisition device is installed on the home appliance, which can acquire a first image of the home appliance, or a communication device is installed on the home appliance, which can receive a first image provided by an external image acquisition device.

[0206] It should be noted that household appliances include dishwashers, range hoods, refrigerators, induction cookers, rice cookers, ovens, air fryers, air conditioners, televisions, etc., and the first image includes an internal image or an external image of the household appliance. For example, if the household appliance is a dishwasher, the first image is an internal image of the dishwasher (that is, an image including tableware placed inside the dishwasher); if the household appliance is a refrigerator, the first image is an internal image of the refrigerator (that is, an image including food placed inside the refrigerator); if the household appliance is an air conditioner, the first image is an overall image of the air conditioner.

[0207] Furthermore, the first image is masked, that is, a part of the sub-image in the first image is masked using a preset mask to obtain a second image, and then the second image is transmitted to the server, so that the server can restore the first image based on the second image, and then perform image processing to obtain control parameters for the home appliance.

[0208] In the technical solution of this scheme, on the one hand, home appliances can transmit images to the server and perform image processing on the server, which reduces the burden of image processing on home appliances, improves the effect of image processing, and ensures the basic operation of home appliances; on the other hand, through mask occlusion processing, only a small part of the image data needs to be transmitted, and there is no need to transmit the entire image to the server, which can greatly reduce network transmission pressure and shorten image transmission time.

[0209] Example 13

[0210] The embodiment of the present invention provides an image processing device. Figure 15 1 shows a schematic block diagram of an image processing apparatus 1500 according to an embodiment of the present invention. The image processing apparatus 1500 includes a receiving module 1502 and a generating module 1504 .

[0211] The receiving module 1502 can obtain a second image sent by the home appliance, where the second image is generated by the home appliance performing masking processing on the first image. The generating module 1504 can decode the second image to obtain the first image.

[0212] In this technical solution, the home appliance masks the first image to obtain a second image, which is then transmitted to a server. After receiving the second image, the server can restore the masked portion of the second image to obtain the first image, and then perform image recognition on the first image.

[0213] In this application's technical solution, on the one hand, the home appliance uses masking to process the image, requiring it to transmit only a small portion of the image data, rather than the entire image, to the server. This significantly reduces network transmission pressure and shortens image transmission time. On the other hand, the server deploys a decoding network to recover the first image and then perform image recognition, reducing the burden on the home appliance for image recognition, improving the image recognition effect, and ensuring the basic operation of the home appliance.

[0214] Example 14

[0215] The embodiment of the present invention provides a household appliance. Figure 16 16 shows a schematic block diagram of a home appliance 1600 according to an embodiment of the present invention. The home appliance 1600 includes a memory 1602 and a processor 1604 .

[0216] Memory 1602 stores programs or instructions, and processor 1604 executes the programs or instructions to implement the steps of the control method for a household appliance according to any of the above technical solutions. Memory 1602 and processor 1604 may be connected via a bus or other means. Processor 1604 may include one or more processing units, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or a field programmable gate array (FPGA).

[0217] The home appliance 1600 provided in this application implements the steps of the image processing method of any of the above technical solutions when the processor 1604 executes the computer program. Therefore, the home appliance 1600 includes all the beneficial effects of the image processing method of any of the above technical solutions.

[0218] In the above technical solution, the home appliance further includes: a communication device connected to the processor 1604, used to send the second image and receive the control parameters with the server.

[0219] In this technical solution, the home appliance further includes a communication device connected to the processor, which can send a second image and receive control parameters to and from the server, thereby ensuring normal data interaction between the home appliance and the server.

[0220] In any of the above technical solutions, the processor may be a single-chip microcomputer.

[0221] In this technical solution, the microcontroller is a low-cost chip that can reduce the cost of home appliances when used in them. However, due to its inherent computing power limitations, it cannot run complex models.

[0222] The technical solution of this application proposes an image mask-based image compression method that can be run on a single-chip microcomputer. The image is masked locally on the home appliance and then transmitted to the server for image restoration. Because the server has sufficient computational examples, it can use a deep learning-based self-encoding and decoding network to restore the image.

[0223] Home appliances only perform simple masking operations on images, without involving complex model loading and inference. These processes place low demands on chip memory and performance, making them easily deployable on various inexpensive microcontrollers. Consequently, the processors in home appliances can directly utilize microcontrollers, reducing the cost of the appliances.

[0224] Example 15

[0225] The embodiment of the present invention provides a server. Figure 17 A schematic block diagram of a server 1700 according to an embodiment of the present invention is shown. The server 1700 includes a memory 1702 and a processor 1704 .

[0226] Memory 1702 stores programs or instructions, and processor 1704 executes the programs or instructions to implement the steps of the control method for a household appliance according to any of the above technical solutions. Memory 1702 and processor 1704 may be connected via a bus or other means. Processor 1704 may include one or more processing units, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or a field programmable gate array (FPGA).

[0227] The server 1700 provided in the present application implements the steps of the image processing method of any of the above technical solutions when the processor 1704 executes the computer program. Therefore, the server 1700 includes all the beneficial effects of the image processing method of any of the above technical solutions.

[0228] Example 16

[0229] An embodiment of the present invention provides a readable storage medium having a program or instruction stored thereon. When the program or instruction is executed by a processor, the steps of the control method of a household appliance as described in any of the above technical solutions or the steps of the image processing method as described in any of the above technical solutions are implemented.

[0230] The readable storage medium provided in this application implements the steps of the control method of the household appliance device as any of the above-mentioned technical solutions when the program or instruction is executed by the processor. Therefore, the readable storage medium includes all the beneficial effects of the control method of the household appliance device as any of the above-mentioned technical solutions.

[0231] The readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0232] The readable storage medium provided by the present application implements the steps of the image processing method of any of the above technical solutions when the program or instruction is executed by the processor. Therefore, the readable storage medium includes all the beneficial effects of the image processing method of any of the above technical solutions.

[0233] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for controlling a household appliance, characterized in that: include: Acquire a first image of the home appliance; performing masking processing on the first image to generate a second image; sending the second image to a server, so that the server can obtain the first image according to the second image; The second image is a sub-image queue of multiple sub-images not blocked by a preset mask; and obtaining the first image according to the second image includes: generating a target image queue according to the preset mask and the sub-image queue; Generating a first embedding vector according to the target image queue, and generating a first position coding vector according to a preset coding method based on the first embedding vector, wherein the preset coding method is a method of using sine coding for even positions of the first embedding vector and using cosine coding for odd positions of the first embedding vector; Adding the first embedding vector and the first position encoding vector to generate a first target vector; The first target vector is decoded by a decoding model to generate the first image.

2. The method according to claim 1, characterized in that Also includes: receiving control parameters from the server; Controlling the operation of the household appliance according to the control parameters; The control parameter is generated by the server based on the first image.

3. The method according to claim 1 or 2, characterized in that The performing masking processing on the first image to generate a second image includes: identifying a background image and a target object image of the first image; The background image is subjected to masking processing without intervals by using a preset mask, and the target object image is subjected to masking processing at preset intervals to generate the second image.

4. The method according to claim 3, characterized in that The sending the second image to the server includes: generating a sub-image queue based on a plurality of sub-images included in the second image and not blocked by the preset mask; The sub-image queue is sent to the server.

5. An image processing method, characterized in that: include: receiving a second image from a home appliance, wherein the second image is obtained by performing masking processing on the first image by the home appliance; generating the first image according to the second image; Wherein, the second image is a sub-image queue of multiple sub-images that are not blocked by a preset mask; Generating the first image according to the second image includes: generating a target image queue according to the preset mask and the sub-image queue; Generating a first embedding vector according to the target image queue, and generating a first position coding vector according to a preset coding method based on the first embedding vector, wherein the preset coding method is a method of using sine coding for even positions of the first embedding vector and using cosine coding for odd positions of the first embedding vector; Adding the first embedding vector and the first position encoding vector to generate a first target vector; The first target vector is decoded by a decoding model to generate the first image.

6. The method according to claim 5, characterized in that Also includes: Determine control parameters of the home appliance according to the first image.

7. The method according to claim 5, characterized in that Also includes: Acquire a first sample image, and perform occlusion processing on the first sample image using the preset mask to generate a second sample image; generating a second embedding vector based on the second sampled image, and generating a second position encoding vector based on the second embedding vector; Adding the second embedding vector and the second position encoding vector to generate a second target vector; Inputting the second target vector into a preset model and outputting a training image; subtracting a plurality of pixel values of the training image from a plurality of pixel values of the first sampling image to obtain a plurality of difference values, and summing the plurality of difference values to generate a target loss function; The preset model is trained according to the target loss function to generate the decoding model.

8. A control device for household appliances, characterized in that: include: An acquisition module, configured to acquire a first image of the home appliance; a processing module, configured to perform masking processing on the first image to generate a second image; a sending module, configured to send the second image to a server, so that the server can obtain the first image according to the second image; The second image is a sub-image queue of multiple sub-images not blocked by a preset mask; and obtaining the first image according to the second image includes: generating a target image queue according to the preset mask and the sub-image queue; Generating a first embedding vector according to the target image queue, and generating a first position coding vector according to a preset coding method based on the first embedding vector, wherein the preset coding method is a method of using sine coding for even positions of the first embedding vector and using cosine coding for odd positions of the first embedding vector; Adding the first embedding vector and the first position encoding vector to generate a first target vector; The first target vector is decoded by a decoding model to generate the first image.

9. An image processing device, characterized in that: include: a receiving module, configured to receive a second image from a home appliance, wherein the second image is obtained by performing masking processing on the first image by the home appliance; a generating module, configured to generate the first image according to the second image; Wherein, the second image is a sub-image queue of multiple sub-images that are not blocked by a preset mask; Generating the first image according to the second image includes: generating a target image queue according to the preset mask and the sub-image queue; Generating a first embedding vector according to the target image queue, and generating a first position coding vector according to a preset coding method based on the first embedding vector, wherein the preset coding method is a method of using sine coding for even positions of the first embedding vector and using cosine coding for odd positions of the first embedding vector; Adding the first embedding vector and the first position encoding vector to generate a first target vector; The first target vector is decoded by a decoding model to generate the first image.

10. A household appliance, characterized in that: include: Memory, storing programs or instructions; A processor, wherein when executing the program or instruction, the processor implements the steps of the method for controlling the household appliance according to any one of claims 1 to 4.

11. The household appliance according to claim 10, characterized in that: Also includes: The communication device is connected to the processor and is used to send the second image to a server and receive control parameters from the server.

12. The household appliance according to claim 10 or 11, characterized in that: The processor is a single chip microcomputer.

13. A server, characterized in that: include: Memory, storing programs or instructions; A processor, wherein when executing the program or instruction, the processor implements the steps of the image processing method according to any one of claims 5 to 7.

14. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the steps of the method for controlling a household appliance according to any one of claims 1 to 4 or the steps of the image processing method according to any one of claims 5 to 7 are implemented.

Images