Control method and device of household appliance and household appliance

Abstract

The present application relates to the field of household appliances, and provides a control method and device of a household appliance and the household appliance, the control method comprising: acquiring a to-be-processed image, the to-be-processed image comprising a target object; performing instance segmentation on the to-be-processed image to obtain target pixel regions corresponding to the target object and target category information; performing material identification on the to-be-processed image based on the target pixel regions to obtain target material information of the target object; and outputting a target control instruction to a target actuating mechanism of the household appliance based on the target category information and the target material information, the target control instruction being used to control the target actuating mechanism to act. The control method determines the target pixel regions through instance segmentation, so that the process of material identification is more focused on details, and the target material information with high accuracy is used to assist the operation control of the household appliance, which helps to improve the user experience.

Description

Technical Field

[0001] This invention relates to the field of household appliance technology, and more particularly to control methods, devices, and household appliances. Background Technology

[0002] With the development of technology, the level of intelligence of home appliances is gradually improving. By incorporating image detection and recognition technology or voice detection and recognition technology, home appliances provide users with more convenient and intelligent services.

[0003] Home appliances equipped with image detection and recognition technology have high requirements for recognizing details such as materials. The determination of the material of each object in the image affects the operation control and intelligence level of home appliances. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention proposes a control method for household appliances that improves the accuracy of material recognition, thereby enhancing the intelligence level of household appliances.

[0005] A method for controlling a household appliance according to a first aspect of the present invention includes:

[0006] Acquire an image to be processed, the image to be processed including a target object;

[0007] Instance segmentation is performed on the image to be processed to obtain the target pixel region and target category information corresponding to the target object;

[0008] Material identification is performed on the image to be processed based on the target pixel region to obtain the target material information of the target object;

[0009] Based on the target category information and target material information, a target control command is output to the target actuator of the household appliance. The target control command is used to control the action of the target actuator.

[0010] According to the control method for household appliances of the present invention, the target pixel region in the image to be processed is determined by instance segmentation, which makes the material recognition process more focused on details, improves the accuracy of target material information recognition, helps to accurately judge the material information of the target object, assists in the operation and control of household appliances, and enhances the user experience.

[0011] According to an embodiment of the present invention, the step of performing material recognition on the image to be processed based on the target pixel region to obtain the target material information of the target object includes:

[0012] If, based on the target category information, the image to be processed is determined to be an image that needs to be cropped, the image to be processed is cropped according to the target pixel region to obtain the target image;

[0013] Material identification is performed on the target image to obtain the target material information.

[0014] According to an embodiment of the present invention, cropping the image to be processed based on the target pixel region to obtain a target image includes:

[0015] Based on the target pixel region, determine the circumscribed rectangle;

[0016] The image to be processed is cropped according to the circumscribed rectangle to obtain the target image.

[0017] According to an embodiment of the present invention, the step of performing material recognition on the image to be processed based on the target pixel region to obtain the target material information of the target object includes:

[0018] Set the pixel values ​​outside the target pixel region in the image to be processed to 0 to obtain the first image;

[0019] Material identification is performed on the first image to obtain the target material information.

[0020] According to an embodiment of the present invention, the step of performing material recognition on the first image to obtain the target material information includes:

[0021] Set the pixel values ​​of the target boundary region in the target pixel region of the first image to 0 to obtain the second image;

[0022] Material identification is performed on the second image to obtain the target material information.

[0023] According to an embodiment of the present invention, the step of performing material recognition on the image to be processed based on the target pixel region to obtain the target material information of the target object includes:

[0024] The image to be processed is input into the material classification model to obtain the target material information output by the material classification model;

[0025] The material classification model is trained using a first sample image as a sample and pre-determined material information of a first sample object corresponding to the first sample image as a sample label.

[0026] According to an embodiment of the present invention, the step of performing instance segmentation on the image to be processed to obtain the target pixel region and target category information corresponding to the target object includes:

[0027] The image to be processed is input into an instance segmentation model to obtain the target pixel region and the target category information output by the instance segmentation model;

[0028] The instance segmentation model is trained using a second sample image as a sample and pre-determined pixel information and category information of a second sample object corresponding to the second sample image as sample labels.

[0029] A control device for a household appliance according to a second aspect embodiment of the present invention includes:

[0030] The acquisition module is used to acquire the image to be processed, which includes the target object;

[0031] The first processing module is used to perform instance segmentation on the image to be processed to obtain the target pixel region and target category information corresponding to the target object;

[0032] The second processing module is used to perform material recognition on the image to be processed based on the target pixel region to obtain the target material information of the target object.

[0033] The output module is used to output target control commands to the target actuator of the household appliance based on the target category information and target material information. The target control commands are used to control the action of the target actuator.

[0034] A household appliance according to a third aspect of the present invention includes:

[0035] The main body of the device is equipped with a camera and a target execution mechanism;

[0036] The control device for the household appliance described above is electrically connected to the camera and the target actuator.

[0037] An electronic device according to a fourth aspect of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the control method for any of the above-described household appliances.

[0038] According to a fifth aspect of the present invention, a non-transitory computer-readable storage medium is provided thereon storing a computer program that, when executed by a processor, implements the steps of the control method for any of the above-described household appliances.

[0039] A computer program product according to a sixth aspect of the present invention includes a computer program that, when executed by a processor, implements the steps of the control method for any of the household appliances described above.

[0040] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0041] By segmenting instances, the target pixel regions in the image to be processed are marked. Then, based on the target pixel regions, material recognition is performed on the image to be processed to obtain target material information. This makes the process of material and other detail recognition more focused on the details of the target pixel regions, thereby improving the accuracy of target material information recognition.

[0042] Furthermore, by segmenting instances to determine the target pixel region in the image to be processed, the material recognition process can focus more on details, improve the accuracy of target material information recognition, help to accurately determine the material information of the target object, assist in the operation and control of home appliances, and enhance the user experience.

[0043] Furthermore, by setting the pixel values ​​of non-target pixel areas to 0, the impact of some pixels in non-target pixel areas on material recognition can be effectively reduced, thereby reducing the computational load during material recognition and improving the accuracy and speed of material recognition.

[0044] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the control method for household appliances provided in an embodiment of the present invention;

[0047] Figure 2 This is one of the schematic diagrams of the material classification model processing flow in the control method for household appliances provided in this embodiment of the invention;

[0048] Figure 3 This is the second schematic diagram of the processing flow of the material classification model in the control method for household appliances provided in this embodiment of the invention;

[0049] Figure 4 This is the third schematic diagram of the processing flow of the material classification model in the control method for household appliances provided in this embodiment of the invention;

[0050] Figure 5This is the fourth schematic diagram of the processing flow of the material classification model in the control method for household appliances provided in this embodiment of the invention;

[0051] Figure 6 This is a schematic diagram of the structure and flow of the control device for household appliances provided in an embodiment of the present invention;

[0052] Figure 7 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0053] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0054] In the description of the embodiments of the present invention, it should be noted that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of the present invention. In addition, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0055] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention based on the specific circumstances.

[0056] In embodiments of the present invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0057] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0058] The following is combined Figures 1 to 5 This invention describes a method for controlling household appliances, which achieves intelligent control of the appliances by acquiring material information from an image, thereby improving the user experience.

[0059] The execution entity of the control method for household appliances in this embodiment of the invention can be the controller of the household appliance, or the cloud, or an edge server.

[0060] In this embodiment of the invention, the household appliance can be a dishwasher, a disinfection cabinet, a range hood, a smart mirror, a refrigerator, an electric iron, or a microwave oven, etc.

[0061] like Figure 1 As shown, the control method for household appliances in this embodiment of the invention includes steps 110 to 140.

[0062] Step 110: Obtain the image to be processed.

[0063] The image to be processed is an image acquired by a home appliance in a working environment through a built-in or external camera. The image to be processed includes a target object, which is the object for which the home appliance needs to obtain relevant information in order to perform the next action.

[0064] In practice, image recognition can be performed on the image to be processed to identify relevant information about the target object in the image, thereby controlling home appliances to respond.

[0065] For example, a household appliance is a dishwasher. The built-in camera of the dishwasher captures images of the inside of the dishwasher that need to be processed. These images include images of target objects such as pots, bowls, chopsticks, or spoons. By recognizing the target objects in the images, the water flow intensity and the type of detergent in the dishwasher can be controlled.

[0066] The household appliance is a range hood. The image to be processed is the image below the range hood captured by a camera installed at the range hood's exhaust vent. The image to be processed includes images of target objects such as pots, stoves, food, or flames below the range hood. By identifying the target objects in the image to be processed, various anomalies such as pot burn and dry pot, as well as the type and doneness of the food, can be accurately predicted, thus providing more accurate feedback to the cooking system.

[0067] The home appliance is a smart mirror. The image to be processed is captured by a camera behind the mirror. The image to be processed includes images of target objects such as the user's face or tongue. By recognizing the target objects in the image to be processed, the system can control the output of various health assessments such as health evaluation and nutritional status.

[0068] The household appliance is a refrigerator, and the image to be processed is an image captured by a camera installed inside the refrigerator's refrigeration or freezer compartment. The image to be processed includes images of target objects such as fruits, vegetables, or meat. By recognizing the target objects in the image to be processed, information such as the freshness of the fruits, vegetables, or meat inside the refrigerator can be controlled and output.

[0069] The household appliance is an electric iron, and the image to be processed is an image captured by the built-in camera under the electric iron. The image to be processed includes images of target objects such as clothing. The temperature and humidity of the electric iron can be controlled by recognizing the target objects in the image to be processed.

[0070] The household appliance is a microwave oven, and the image to be processed is an image captured by a camera installed inside the microwave oven. The image to be processed includes images of target objects such as pots, bowls, or food. By recognizing the target objects in the image to be processed, the heating time of different foods can be adjusted.

[0071] Step 120: Perform instance segmentation on the image to be processed to obtain the target pixel region and target category information corresponding to the target object.

[0072] Instance segmentation is the process of separating objects from the background. It involves detecting objects in an image and segmenting the entire image into different target pixel regions based on different objects.

[0073] Instance segmentation can also classify the category corresponding to each target object to obtain the target category information corresponding to the target object.

[0074] For example, if the household appliance is a dishwasher, the target pixel regions corresponding to target objects such as the dishwasher's inner pot, bowl, chopsticks, or spoon can be separated from the image to be processed through instance segmentation, and the target category information corresponding to each target object can be output.

[0075] The household appliance is a range hood. Instance segmentation can be used to separate the target pixel regions corresponding to target objects such as pots, stoves, food, or flames below the range hood from the image to be processed, and output the target category information corresponding to each target object.

[0076] The home appliance is a smart mirror that can separate the target pixel regions corresponding to target objects such as the user's face or tongue from the image to be processed through instance segmentation, and output the target category information corresponding to each target object.

[0077] The household appliance is a refrigerator. Instance segmentation can be used to separate the target pixel regions corresponding to target objects such as fruits, vegetables, or meat in the refrigerator from the image to be processed, and output the target category information corresponding to each target object.

[0078] The household appliance is a microwave oven. Instance segmentation can be used to separate the target pixel regions corresponding to target objects such as the inner pot, bowl, or food of the microwave oven from the image to be processed, and output the target category information corresponding to each target object.

[0079] It should be noted that instance segmentation can separate the target pixel regions corresponding to different individuals belonging to the same category.

[0080] For example, the image to be processed of a dishwasher includes three plates that belong to the same type of item. Instance segmentation can segment the image into three target pixel regions corresponding to the three plates.

[0081] It is understandable that the image to be processed may include multiple target objects. After instance segmentation, the target pixel regions and target category information of each target object can be obtained, thereby realizing pixel segmentation and category recognition of multiple target objects in the image to be processed.

[0082] Step 130: Based on the target pixel region obtained by instance segmentation, perform material identification on the image to be processed to obtain target material information.

[0083] In this step, instance segmentation defines the target pixel region corresponding to the target object. The target pixel region can be used as the boundary to perform material recognition on the pixels within the target pixel region in the image to be processed, thereby obtaining the target material information corresponding to the target object in the image to be processed.

[0084] In this embodiment, the target material information reflects the surface structure and arrangement properties of the target object. By identifying the target material information, the constituent materials and surface structure characteristics of the target object can be obtained, thereby enabling accurate identification of the target object and assisting in the operation and control of household appliances.

[0085] Take a dishwasher as an example.

[0086] The image to be processed inside the dishwasher is segmented to obtain the target pixel regions corresponding to the three plates inside the dishwasher. The pixels in the target pixel regions of the image to be processed are material-identified to obtain the target material information corresponding to the three plates inside the dishwasher, and then it is determined that the three plates are plastic plates, ceramic plates and metal plates.

[0087] Depending on the type of tableware, the water flow intensity and detergent in the dishwasher can be adjusted to improve cleaning performance and save water and electricity.

[0088] Among related technologies, some texture recognition frameworks based on multi-task have emerged. However, since the core of multi-task is the shared feature layer, the recognition process cannot focus on details, and the accuracy of recognizing details such as material is difficult to guarantee.

[0089] Among the related technologies, some structures based on anchor frame image cutting for texture judgment have emerged. For images with severely occluded overlapping objects, this can easily lead to misjudgment of material and other detailed information.

[0090] This invention, through instance segmentation, marks the target pixel region in the image to be processed, and then performs material recognition on the image to be processed based on the target pixel region to obtain the target material information. This makes the process of material and other detail recognition more focused on the details of the target pixel region, improves the accuracy of target material information recognition, helps to accurately determine the material of the target object, and assists in the operation and control of household appliances.

[0091] Step 140: Based on the target category information and target material information, output the target control command to the target actuator of the household appliance.

[0092] In this embodiment, the household appliance is equipped with a target actuator that performs corresponding actions. By outputting target control commands to control the action of the target actuator, the function to be performed by the household appliance can be realized.

[0093] In practice, the target implementing agency can be either the agency that performs the action or the agency that provides information feedback.

[0094] For example, in the case of a household appliance like a dishwasher, the target actuator could be a mechanism that controls the water flow intensity and detergent dispensing within the dishwasher.

[0095] The household appliance is a range hood, and the target actuator can be a speaker that controls the voice broadcast of whether the pot is burning, dry pot, type of ingredients, and doneness.

[0096] The home appliance is a smart mirror, and the target actuator can be a speaker or display screen that outputs various health assessments such as health evaluations and nutritional status.

[0097] The household appliance is a refrigerator, and the target actuator can be a speaker or display screen that controls and outputs information such as the freshness of fruits, vegetables or meats inside the refrigerator.

[0098] The household appliance is an electric iron, and the target actuator can be a heating mechanism or a humidifying mechanism that controls the temperature and humidity of the electric iron.

[0099] The household appliance is a microwave oven, and the target actuator can be a clock control device that controls and adjusts the heating time of different foods.

[0100] In this step, target control instructions are generated based on the target material information and target category information to control the action of the target actuator, and the corresponding target control instructions are output to the target actuator.

[0101] Take a dishwasher as an example.

[0102] Instance segmentation yields the target pixel regions of three target objects inside the dishwasher, and the target category information of the three target objects is obtained as plates.

[0103] Material identification was performed on the image to be processed based on three target pixel regions, resulting in three plates: a plastic plate, a ceramic plate, and a metal plate.

[0104] Based on the material of the three plates, target control commands are generated and output to the water inlet valve and detergent dispensing mechanism inside the dishwasher. This enables control over the water flow intensity and the type or quantity of detergent dispensed, improving the cleaning effect of the dishwasher and effectively saving water and electricity consumption.

[0105] According to the control method for household appliances provided by the present invention, the target pixel region in the image to be processed is determined by instance segmentation, which makes the material recognition process more focused on details, improves the accuracy of target material information recognition, helps to accurately judge the material information of the target object, assists in the operation and control of household appliances, and enhances the user experience.

[0106] In some embodiments, the target category information obtained from instance segmentation can be used to determine whether the image to be processed needs to be cropped.

[0107] In this embodiment, when the image to be processed is an image that needs to be cropped, the image to be processed is cropped according to the target pixel region obtained by instance segmentation to obtain the corresponding target image, and then the material recognition is performed on the target image.

[0108] It is understandable that by cropping based on the target pixel regions corresponding to multiple target objects in the image to be processed, multiple target images corresponding to multiple target objects can be obtained.

[0109] Take a dishwasher as an example.

[0110] After instance segmentation of the image to be processed captured by the internal camera of the dishwasher, three target pixel regions corresponding to three target objects can be obtained. After cutting or cropping, three target images corresponding to the three target objects can be obtained.

[0111] It is understandable that the target image is obtained by cropping the target pixel region of the image to be processed based on instance segmentation. The target image includes all pixels corresponding to the target object, and the pixel ratio of the target object in the target image is higher than that of the non-target object.

[0112] In this embodiment, the instance segmentation process has marked the target pixel regions corresponding to each target object in the image to be processed. The cropping process reduces the interference of non-target object pixels in the image to be processed when using the target image for material recognition, thereby improving the accuracy of material information recognition.

[0113] In some embodiments, by determining the circumscribed rectangle corresponding to the target pixel region in the image to be processed, the corresponding target image is obtained by cropping using the circumscribed rectangle.

[0114] In this embodiment, the target pixel region in the image to be processed is a closed region. Tangents can be drawn at different boundaries of the target pixel region, and the intersection of the four tangents can form an external rectangular frame.

[0115] In the image to be processed, different target pixel regions correspond to different bounding rectangles. By using multiple bounding rectangles to cut the corresponding target pixel regions, multiple different target images can be obtained.

[0116] It is understandable that the target image obtained by cutting using the circumscribed rectangle is also a rectangular image, and the size of the target image is the same as the size of the circumscribed rectangle.

[0117] After cropping the image to be processed, the proportion of the target pixel region in the target image is greater than that in the image to be processed.

[0118] In some embodiments, the pixel values ​​corresponding to the non-target object portion of the image to be processed can be set to 0 to remove the influence of the background portion of the non-target object.

[0119] In practice, the background of the non-target pixel area in the image to be processed can be completely blackened, that is, the pixel value corresponding to the point outside the target pixel area is set to 0, and the first image is obtained.

[0120] Setting the pixel values ​​of non-target pixel regions to 0 and removing the background from the image to be processed before performing material recognition can effectively reduce the impact of some pixels in non-target pixel regions on material recognition, reduce the amount of computation during material recognition, and improve the accuracy and speed of material recognition.

[0121] In some embodiments, the pixel values ​​corresponding to points in the target boundary region of the target pixel region can also be set to 0.

[0122] After setting the pixel values ​​of non-target pixel regions in the image to be processed to 0 to obtain the first image, the pixel values ​​of the target boundary regions in the target pixel regions of the first image are set to 0 to obtain the corresponding second image.

[0123] In this embodiment, the target boundary region refers to the set of edge points of the target pixel region. The edge of the target pixel region is the junction between the region corresponding to the target object and the region corresponding to the non-target object. It is the place where the regional attributes change abruptly and has uncertainty.

[0124] Setting the pixel value corresponding to the point in the target boundary region of the target pixel region to 0 can further reduce the interference of non-target object corresponding pixels in the image to be processed and improve the accuracy of material information recognition.

[0125] In practice, the boundary of the target pixel region in the first image obtained by removing the background can be reduced by 2 to 5 pixels, that is, the value of the boundary 2 to 5 pixels can be set to 0.

[0126] For example, the first image with the background removed is processed to reduce the boundary to obtain a second image with reduced boundary, and then the target image with reduced boundary is used for material recognition.

[0127] In practice, the target pixel area of ​​the first image with the background removed can be set to white. Morphological erosion is then used to reduce the white edge area. The black part of the eroded image is then directly applied to the original first image with the background removed to achieve the effect of reducing the boundary and obtaining the corresponding target image.

[0128] The following describes seven specific implementation examples of home appliance control.

[0129] 1. The household appliance is a dishwasher.

[0130] When the dishwasher door is closed, the material detection switch is triggered, and the built-in camera takes an image of the dishes inside the dishwasher, which is the image to be processed.

[0131] The image to be processed is segmented into instances, and the target category information of the tableware and the corresponding target pixel region are output based on the instance segmentation.

[0132] Based on the target category information of the tableware, determine whether the image to be processed needs to be cropped. If cropping is required, determine the target rectangle based on the target pixel area and crop accordingly.

[0133] Set the pixels in the cropped image that are not the target object to 0, and also set the instance segmentation edges, such as pixels 3 pixels away from the boundary, to 0 to obtain the target image.

[0134] The system performs material recognition on the target image, reads the database based on the target material and category information, sets the corresponding cleaning intensity and detergent, and controls the dishwasher to start cleaning.

[0135] 2. The household appliance is a disinfection cabinet.

[0136] When the disinfection cabinet door is closed, the material detection switch is triggered, and the built-in camera takes an image of the tableware inside the disinfection cabinet, which is the image to be processed.

[0137] The system performs instance segmentation and material recognition on the image to be processed. Based on the target material information and target category information, it reads the database and sets the corresponding disinfection program, such as whether it is heat disinfection or non-heat disinfection. Based on the number of tableware detected, it estimates the heating energy and controls the disinfection cabinet to start disinfection.

[0138] 3. Household appliances include stoves.

[0139] When a user turns on the gas valve, the camera on the range hood captures a real-time image of the stovetop. The camera identifies whether the pot is empty or without a lid, and then performs instance segmentation and material identification on the corresponding image to be processed.

[0140] If the ingredient is found to be one of the ingredients in the cooking expert database that requires testing for doneness, such as chicken wings, lamb chops, or steak, then the outer bounding rectangle of the cropped image is calculated, the material of the cropped image is identified, and the current doneness is output.

[0141] Based on the current level of doneness, if it is not yet cooked, the corresponding fire control temperature is read from the database and the corresponding stove firepower value is set; if it is cooked, it means it is ready to eat, and the fire is turned off.

[0142] IV. The household appliance is a smart mirror.

[0143] When the switch is turned on, the camera captures the dynamics in front of the mirror in real time. If a face is detected, the user is prompted to stick out their tongue. The camera then crops the face image, performs instance segmentation and material recognition, reads the corresponding health statistics from the database, and provides voice prompts and suggestions regarding the user's current health status.

[0144] 5. The household appliance is a refrigerator.

[0145] When the refrigerator is turned on, it uses a built-in camera to periodically take images of the food inside. The images are then segmented and their materials are identified to determine the freshness of the food. Based on this freshness, the refrigerator reminds the user of when the food will be stored or expire, thus preventing waste caused by the user forgetting to store it.

[0146] 6. The household appliance is an electric iron.

[0147] Turn on the power, use the built-in camera under the iron to take pictures of the clothes, perform instance segmentation and material recognition to obtain the category and material of the clothes, extract the recommended temperature and humidity from the database, automatically set the corresponding recommended temperature and humidity, and prompt the user.

[0148] 7. The household appliance is a microwave oven.

[0149] When the microwave oven door is closed, the material detection switch is triggered. The built-in camera takes pictures of the food inside the microwave oven, performs instance segmentation and material recognition, and adjusts the microwave heating intensity and time of the food according to its type and material.

[0150] In some embodiments, target material information can be obtained through a classification network.

[0151] In this embodiment, the input to the material classification model can be the target image or the image to be processed, and the output of the material classification model is the target material information.

[0152] For example, the target image is input into the material classification model, which performs material identification and classification, and outputs the target material information, which corresponds to the material as plastic.

[0153] The target image is input into the material classification model, which performs material identification and classification, and outputs the target material information, which corresponds to the material of ceramic.

[0154] The target image is input into the material classification model, which performs material identification and classification, and outputs the target material information, which indicates that the material is metal.

[0155] Material classification models belong to classification network models. Material classification models include, but are not limited to, classic classification network models such as VGG, Renet, and Inception, as well as lightweight network models such as MobileNet, ShuffleNet, and GhostNet.

[0156] In practice, the material classification model is trained using the first sample image.

[0157] During the training of the material classification model, the material information corresponding to the first sample object in each first sample image is manually labeled using the first sample image as a sample.

[0158] Take a dishwasher as an example.

[0159] A large number of images of the inside of the dishwasher are taken as the first sample images. The first sample images are used as the input of the material classification model. The material information of each sample object in the first sample images is labeled. The first sample images with material information are used as the output of the material classification model to train the material classification model.

[0160] In some embodiments, the material classification model includes: a first processing layer, a second processing layer, and a third processing layer.

[0161] The first processing layer is the feature extraction layer of the material classification model, which can be called the backbone network in the material classification model.

[0162] The input to the first processing layer is the target image, and the output of the first processing layer is the first feature vector corresponding to the target image. The dimension of the first feature vector is the first dimension.

[0163] In actual execution, the first processing layer can perform operations such as convolution, pooling, and global average pooling on the target image to obtain the first feature vector.

[0164] The purpose of convolution processing is to extract features from a target image. More complex features can be extracted iteratively from low-level features through multi-layer convolutional structures.

[0165] Pooling is a form of downsampling. Pooling divides the input target image into several rectangular regions and outputs the maximum value for each sub-region.

[0166] Pooling is applied to each input feature and reduces its size, which also reduces the number of parameters and computational cost, thus controlling overfitting to some extent. Pooling is periodically inserted between convolutional processes, and after layers of convolutional pooling operations, a feature tensor is obtained.

[0167] Global average pooling sums all pixel values ​​in a feature map and averages them to obtain a single value, which is then used to represent the corresponding feature map.

[0168] The following is a specific example.

[0169] The segmented target image is used as input and fed into the first processing layer. After layers of convolution and pooling operations in the first processing layer, features of dimension CxWxH are obtained.

[0170] Where C refers to the output channel, W refers to the width of the feature, and H refers to the height of the feature.

[0171] The features of dimension CxWxH are subjected to global average pooling to obtain a Cx1x1 tensor.

[0172] Reshape the Cx1x1 tensor to obtain a C-dimensional vector for output, which is also the first feature vector.

[0173] The second processing layer is a layer that performs dimensional transformation on the first feature vector. The input of the second processing layer is the first feature vector, and after the dimensional transformation, it outputs multiple second feature vectors with different dimensions.

[0174] The second processing layer's dimensionality processing of the first feature vector is essentially a process of proportionally truncating the first feature vector.

[0175] The following is a specific example.

[0176] The first feature vector is a vector of size 2048. The second processing layer performs dimensionality processing on the first feature vector, truncating 3 / 4, 1 / 2 and 1 / 4 of the first feature vector respectively, and outputs vector one of size 1536, vector two of size 1024 and vector three of size 512.

[0177] The second processing layer also outputs a vector of size 2048, which retains the original small dimension.

[0178] Vector 1 to Vector 4 are multiple second feature vectors output by the second processing layer, and the dimensions of the multiple second feature vectors are different.

[0179] The third processing layer performs batch normalization and ReLU nonlinear transformation on multiple second feature vectors. Finally, after fully connected processing, multiple vectors are restored to the same dimension. Then, the multiple vectors are averaged and summed. Using a classifier, the target material information is output.

[0180] Batch Normalization (BN) optimizes the variance and mean position before the activation function, making the new distribution more consistent with the true distribution of the data and ensuring the non-linear expressive power of the model.

[0181] The classifier that outputs the target material information can be a Softmax classifier. The Softmax classifier converts the score value obtained from linear classification into a probability value and performs multi-classification.

[0182] The following is a specific example.

[0183] like Figure 2 As shown, four second feature vectors are input to the third processing layer. The four second feature vectors are vector one with a size of 1536, vector two with a size of 1024, vector three with a size of 512, and vector four with a size of 2048.

[0184] The third processing layer performs batch normalization and ReLU nonlinear transformation on the four second feature vectors, and then passes them through a fully connected layer to restore them to 2048-dimensional vectors. Finally, the average sum of these four vectors is calculated to output the final target material information.

[0185] In some embodiments, an attention module is introduced into the material classification model to make the segmentation in the material classification model more reasonable by utilizing the attention mechanism.

[0186] The attention mechanism is a resource allocation mechanism that can be understood as redistributing weights to resources that were originally evenly distributed according to the importance of different objects.

[0187] In the material classification model, the weights of attention assignment are automatically derived by the attention module based on existing features, without the need for manual settings, making the material classification model more intelligent.

[0188] The following section introduces three specific attention modules.

[0189] I. Introduce the first attention module into the material classification model.

[0190] In this embodiment, the first attention module belongs to the segmented overall attention mechanism.

[0191] The first attention module performs a dimension change on the first feature vector output by the first processing layer to obtain a third feature vector, the dimension of which is the second dimension.

[0192] The first attention module then passes the third feature vector through a fully connected (FC) layer to obtain a vector whose dimension is related to the number of second feature vectors in the third processing layer.

[0193] For example, the number of second feature vectors in the third processing layer is 4, and the first attention module outputs a 4-dimensional vector after passing the third feature vector through a fully connected layer.

[0194] The first attention module performs a nonlinear transformation on each component of the vector obtained after full connection, resulting in multiple first weight values.

[0195] It is understandable that the number of multiple first weight values ​​is equal to the number of second feature vectors in the third processing layer.

[0196] The first attention module inputs the obtained first weight values ​​into the third processing layer, and the third processing layer assigns the first weight values ​​to the corresponding second feature vectors.

[0197] In the third processing layer, each second feature vector is multiplied by its corresponding first weight value. The third processing layer then performs batch normalization, ReLU nonlinear transformation, fully connected layer, and average summation (Avg) on ​​the vector obtained after multiplication, and outputs the final target material information.

[0198] For example, such as Figure 3 As shown, the first attention module pulls another branch from the last CxWxH dimension feature tensor of the backbone network and performs global average pooling to obtain a Cx1x1 tensor, which is uniformly set to C=2048 for convenience.

[0199] The 2048x1x1 tensor is reshaped into a 2048-dimensional vector, and then this 2048-dimensional vector is fully connected to output a 4-dimensional vector.

[0200] Perform a sigmoid nonlinear transformation on each component of this 4-dimensional vector to obtain four values ​​of 0 to 1. Apply these four values ​​to the four segments of the original third processing layer, that is, multiply the value of each segment by this 0 to 1 value.

[0201] Vectors of length other than 2048 in these four segments are all subjected to batch normalization and ReLU nonlinear transformation, and finally passed through a fully connected layer to be restored to vectors of 2048 dimensions.

[0202] Finally, the average sum of these four vectors is calculated, and then the softmax classifier is used to obtain the final material information classification output.

[0203] 2. Introduce a second attention module into the material classification model.

[0204] In this embodiment, the second attention module belongs to a local attention mechanism with shared weights.

[0205] The second attention module performs dimensionality processing on the first feature vector output by the first processing layer, and proportionally truncates the first feature vector to obtain multiple feature vectors of different dimensions.

[0206] The second attention module then performs a nonlinear transformation on each component of the feature vectors of multiple different dimensions to obtain multiple second weight values.

[0207] It is understandable that the second attention module truncates the first feature vector proportionally, and the truncation ratio and the size of the final feature vector correspond to the second feature vector output by the second processing layer.

[0208] In the third processing layer, each second feature vector is multiplied by its corresponding second weight value. The third processing layer then performs batch normalization, ReLU nonlinear transformation, fully connected layer, and average summation on the vector obtained after multiplication, and outputs the final target material information.

[0209] For example, such as Figure 4 As shown, the second attention module pulls another branch from the last CxWxH dimension feature tensor of the backbone network and performs global average pooling to obtain a Cx1x1 tensor, which is uniformly set to C=2048 for convenience.

[0210] The 2048x1x1 tensor is reshaped into a 2048-dimensional vector. Then, this 2048-dimensional vector is passed through a 4-way fully connected layer to output four vectors of different dimensions: 512, 1024, 1536, and 2048.

[0211] Then, perform a sigmoid nonlinear transformation on each component of these four vectors of different dimensions. The component values ​​of each vector are between 0 and 1.

[0212] Each component of these four vectors is multiplied by the second feature vector of the four segments in the third processing layer, and the value of each segment is assigned an attention weight.

[0213] Vectors of length other than 2048 in these four segments are all subjected to batch normalization and ReLU nonlinear transformation, and finally passed through a fully connected layer to be restored to vectors of 2048 dimensions.

[0214] Finally, the average of these four vectors is summed, and then softmax is applied to obtain the final classification output target material information.

[0215] Third, a third attention module is introduced into the material classification model.

[0216] In this embodiment, the third attention module belongs to a local attention mechanism that does not share weights.

[0217] The third attention module needs to obtain multiple first feature vectors to calculate the third weight value. The number of first feature vectors obtained by the third attention module is the same as the number of second feature vectors output by the second processing layer.

[0218] The third attention module performs dimensionality transformation on multiple first feature vectors, that is, it truncates multiple first feature vectors according to different proportions.

[0219] The proportion of the first feature vectors extracted by the third attention module and the size of the final feature vector correspond to the second feature vector output by the second processing layer.

[0220] In the third processing layer, each second feature vector is multiplied by its corresponding third weight value. The third processing layer then performs batch normalization, ReLU nonlinear transformation, fully connected layer, and average summation on the vector obtained after multiplication, and outputs the final target material information.

[0221] For example, such as Figure 5 As shown, the third attention module pulls four branches from the final CxWxH dimension feature tensor of the backbone network, performs global average pooling on each branch, and obtains four Cx1x1 tensors. For convenience, we set C=2048 here.

[0222] Then reshape all four 2048x1x1 tensors into 2048-dimensional vectors.

[0223] The 2048-dimensional vector is passed through a 4-full connection to output four vectors of different dimensions: 512, 1024, 1536, and 2048.

[0224] Then, perform a sigmoid nonlinear transformation on each component of these four vectors of different dimensions. The component values ​​of each vector are between 0 and 1.

[0225] Each component of these four vectors is multiplied by the value of the original four segments in the "Split-and-Share Module", so that the value of each segment is assigned an attention weight.

[0226] Vectors of length other than 2048 in these four segments are all subjected to batch normalization and ReLU nonlinear transformation, and finally passed through a fully connected layer to be restored to vectors of 2048 dimensions.

[0227] Finally, the average of these four vectors is summed, and then softmax is applied to obtain the final classification output target material information.

[0228] It should be noted that adding an attention mechanism module to the material classification model makes the material classification more accurate. The attention mechanism module is a pluggable design module, which will not affect the initial design of the feature extraction and classification operations of the material classification model. It is flexible in operation and simple to implement.

[0229] In some embodiments, step 120 can be implemented using an instance segmentation model.

[0230] In this embodiment, the input of the instance segmentation model is the image to be processed, and the output of the instance segmentation model is the target category information and target pixel region corresponding to the target object in the image to be processed.

[0231] For example, the image of the dishwasher to be processed is input into the instance segmentation model, which performs instance segmentation processing, inputting the three target pixel regions corresponding to the three plates respectively.

[0232] Instance segmentation models include, but are not limited to, mask-rcnn, YOLACT, BlendMask, PointRend, and SOLO.

[0233] It is understandable that the process of an instance segmentation model outputting the target pixel region is a process of defining the boundaries of pixels in the image to be processed.

[0234] In practice, the instance segmentation model is trained using a second sample image.

[0235] In the instance segmentation model, the second sample image is used as the sample, and the pixel information corresponding to the second sample object in the second sample image is used as the sample label.

[0236] For different application scenarios, the pixels corresponding to each second sample object in the second sample image can also be manually labeled with boundaries.

[0237] Take a dishwasher as an example.

[0238] A large number of images of the inside of the dishwasher are taken as second sample images. The second sample images are used as input to the instance segmentation model. The boundaries of each piece of tableware in the second sample images are labeled. The second sample images with boundary labels are used as the output of the second sample images to train the instance segmentation model.

[0239] The control device for household appliances provided in the embodiments of the present invention is described below. The control device for household appliances described below can be referred to in correspondence with the control method for household appliances described above.

[0240] like Figure 6 As shown, the control device for household appliances provided in this embodiment of the invention includes:

[0241] The acquisition module 610 is used to acquire the image to be processed, which includes the target object;

[0242] The first processing module 620 is used to perform instance segmentation on the image to be processed to obtain the target pixel region and target category information corresponding to the target object;

[0243] The second processing module 630 is used to perform material recognition on the image to be processed based on the target pixel region to obtain the target material information of the target object.

[0244] The output module 640 is used to output target control commands to the target actuator of the household appliance based on the target category information and target material information. The target control commands are used to control the action of the target actuator.

[0245] The control device for household appliances provided by the present invention determines the target pixel region in the image to be processed through instance segmentation, making the material recognition process more focused on details, improving the accuracy of target material information recognition, helping to accurately determine the material information of the target object, assisting in the operation and control of household appliances, and enhancing the user experience.

[0246] In some embodiments, the first processing module 620 is further configured to, when determining that the image to be processed is an image that needs to be cropped based on the target category information, crop the image to be processed according to the target pixel region to obtain a target image; and perform material recognition on the target image to obtain target material information.

[0247] In some embodiments, the first processing module 620 is further configured to determine an external bounding rectangle based on the target pixel region; and to crop the image to be processed according to the external bounding rectangle to obtain the target image.

[0248] In some embodiments, the second processing module 630 is further configured to set the pixel values ​​of points outside the target pixel region in the image to be processed to 0 to obtain a first image; and to perform material recognition on the first image to obtain target material information.

[0249] In some embodiments, the second processing module 630 is further configured to set the pixel values ​​of the target boundary region in the target pixel region of the first image to 0 to obtain a second image; and to perform material recognition on the second image to obtain target material information.

[0250] In some embodiments, the second processing module 630 is used to input the image to be processed into the material classification model to obtain the target material information output by the material classification model; wherein, the material classification model is trained using the first sample image as a sample and the material information of the first sample object corresponding to the first sample image as a predetermined sample label.

[0251] In some embodiments, the first processing module 620 is used to input the image to be processed into the instance segmentation model to obtain the target pixel region and target category information output by the instance segmentation model;

[0252] The instance segmentation model is trained using the second sample image as the sample and the pixel information and category information of the second sample object corresponding to the second sample image as the sample label.

[0253] This invention also provides a household appliance, which includes a device body and a control device for the household appliance as described above.

[0254] In this embodiment of the invention, the household appliance can be a dishwasher, a disinfection cabinet, a range hood, a smart mirror, a refrigerator, an electric iron, or a microwave oven, etc.

[0255] Home appliances have built-in or external cameras to acquire images of the appliances in their working environment.

[0256] Household appliances are equipped with target actuators that perform corresponding actions. By outputting target control commands to control the action of the target actuators, the functions that the household appliances are required to perform can be realized.

[0257] According to the household appliance provided by the present invention, the target pixel region in the image to be processed is determined by instance segmentation, which makes the material recognition process more focused on details, improves the accuracy of target material information recognition, helps to accurately judge the material information of the target object, assists in the operation and control of the household appliance, and enhances the user experience.

[0258] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions from the memory 730 to execute a control method for a home appliance. This method includes: acquiring an image to be processed, the image including a target object; performing instance segmentation on the image to be processed to obtain target pixel regions and target category information corresponding to the target object; performing material recognition on the image to be processed based on the target pixel regions to obtain target material information of the target object; and outputting target control instructions to the target actuator of the home appliance based on the target category information and target material information. The target control instructions are used to control the action of the target actuator.

[0259] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0260] Furthermore, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the control method for household appliances provided in the above-described method embodiments. The method includes: acquiring an image to be processed, the image to be processed including a target object; performing instance segmentation on the image to be processed to obtain target pixel regions and target category information corresponding to the target object; performing material recognition on the image to be processed based on the target pixel regions to obtain target material information of the target object; and outputting a target control command to the target actuator of the household appliance based on the target category information and the target material information, the target control command being used to control the action of the target actuator.

[0261] On the other hand, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the control method for the household appliances provided in the above embodiments. The method includes: acquiring an image to be processed, the image to be processed including a target object; performing instance segmentation on the image to be processed to obtain target pixel regions and target category information corresponding to the target object; performing material recognition on the image to be processed based on the target pixel regions to obtain target material information of the target object; and outputting a target control command to the target actuator of the household appliance based on the target category information and the target material information. The target control command is used to control the action of the target actuator.

[0262] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0263] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0264] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0265] The above embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Although the invention has been described in detail with reference to the embodiments, those skilled in the art should understand that various combinations, modifications, or equivalent substitutions of the technical solutions of the invention do not depart from the spirit and scope of the invention and should be covered within the scope of the claims of the invention.

Claims

1. A method for controlling a household appliance, characterized in that, include: Acquire an image to be processed, the image to be processed including a target object; Instance segmentation is performed on the image to be processed to obtain the target pixel region and target category information corresponding to the target object; Material identification is performed on the image to be processed based on the target pixel region to obtain the target material information of the target object, including: The image to be processed is input into a material classification model to obtain the target material information output by the material classification model. The material classification model includes a first processing layer, a second processing layer, an attention module, and a third processing layer. The first processing layer is used to output a first feature vector. The second processing layer is used to perform dimensionality transformation on the first feature vector to output multiple second feature vectors. The attention module is used to generate multiple weight values ​​based on the first feature vector and assign the multiple weight values ​​to the multiple second feature vectors respectively. The third processing layer is used to fuse the multiple second feature vectors after weight assignment to output the target material information. The multiple weight values ​​include at least one of segmented global attention weight, shared local attention weight, and non-shared local attention weight. The segmented overall attention weight is obtained by performing dimensionality transformation on the first feature vector using the first attention module to obtain the third feature vector, and then performing fully connected and nonlinear transformations on the third feature vector in sequence to obtain multiple first weight values. The shared local attention weight is obtained by processing the first feature vector with the second attention module and truncating it proportionally to obtain multiple feature vectors with different dimensions. Then, a nonlinear transformation is performed on each component of the multiple feature vectors with different dimensions to obtain multiple second weight values. The non-shared local attention weights are obtained by performing dimensionality transformation processing on multiple first feature vectors through a third attention module, resulting in multiple third weight values. Based on the target category information and target material information, a target control command is output to the target actuator of the household appliance. The target control command is used to control the action of the target actuator.

2. The control method for household appliances according to claim 1, characterized in that, The step of performing material recognition on the image to be processed based on the target pixel region to obtain the target material information of the target object includes: If, based on the target category information, the image to be processed is determined to be an image that needs to be cropped, the image to be processed is cropped according to the target pixel region to obtain the target image; Material identification is performed on the target image to obtain the target material information.

3. The control method for household appliances according to claim 2, characterized in that, The step of cropping the image to be processed based on the target pixel region to obtain the target image includes: Based on the target pixel region, determine the circumscribed rectangle; The image to be processed is cropped according to the circumscribed rectangle to obtain the target image.

4. The control method for household appliances according to claim 1, characterized in that, The step of performing material recognition on the image to be processed based on the target pixel region to obtain the target material information of the target object includes: Set the pixel values ​​outside the target pixel region in the image to be processed to 0 to obtain the first image; Material identification is performed on the first image to obtain the target material information.

5. The control method for household appliances according to claim 4, characterized in that, The step of performing material recognition on the first image to obtain the target material information includes: Set the pixel values ​​of the target boundary region in the target pixel region of the first image to 0 to obtain the second image; Material identification is performed on the second image to obtain the target material information.

6. The control method for household appliances according to any one of claims 1-5, characterized in that, The material classification model is trained using a first sample image as a sample and pre-determined material information of a first sample object corresponding to the first sample image as a sample label.

7. The control method for household appliances according to any one of claims 1-5, characterized in that, The step of performing instance segmentation on the image to be processed to obtain the target pixel region and target category information corresponding to the target object includes: The image to be processed is input into an instance segmentation model to obtain the target pixel region and the target category information output by the instance segmentation model; The instance segmentation model is trained using a second sample image as a sample and pre-determined pixel information and category information of a second sample object corresponding to the second sample image as sample labels.

8. A control device for a household appliance, characterized in that, include: The acquisition module is used to acquire the image to be processed, which includes the target object; The first processing module is used to perform instance segmentation on the image to be processed to obtain the target pixel region and target category information corresponding to the target object; The second processing module is used to perform material recognition on the image to be processed based on the target pixel region to obtain the target material information of the target object. The second processing module is further configured to input the image to be processed into a material classification model to obtain the target material information output by the material classification model; the material classification model includes a first processing layer, a second processing layer, an attention module, and a third processing layer; the first processing layer is configured to output a first feature vector; the second processing layer is configured to perform dimensionality transformation on the first feature vector to output multiple second feature vectors; the attention module is configured to generate multiple weight values ​​based on the first feature vector and assign the multiple weight values ​​to the multiple second feature vectors respectively; the third processing layer is configured to fuse the multiple second feature vectors after weight assignment to output the target material information; the multiple weight values ​​include at least one of segmented global attention weight, shared local attention weight, and non-shared local attention weight; The segmented overall attention weight is obtained by performing dimensionality transformation on the first feature vector using the first attention module to obtain the third feature vector, and then performing fully connected and nonlinear transformations on the third feature vector in sequence to obtain multiple first weight values. The shared local attention weight is obtained by processing the first feature vector with the second attention module and truncating it proportionally to obtain multiple feature vectors with different dimensions. Then, a nonlinear transformation is performed on each component of the multiple feature vectors with different dimensions to obtain multiple second weight values. The non-shared local attention weights are obtained by performing dimensionality transformation processing on multiple first feature vectors through a third attention module, resulting in multiple third weight values. The output module is used to output target control commands to the target actuator of the household appliance based on the target category information and target material information. The target control commands are used to control the action of the target actuator.

9. A household appliance, characterized in that, include: The main body of the device is equipped with a camera and a target execution mechanism; The control device for a household appliance as described in claim 8 is electrically connected to the camera and the target actuator.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the control method for the household appliance as described in any one of claims 1 to 7.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the control method for the household appliance as described in any one of claims 1 to 7.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the control method for the household appliance as described in any one of claims 1 to 7.

Images