Household appliance, control method and device thereof and storage medium

By collecting multimodal information in home appliances to build a portrait of dynamic objects and adjusting execution instructions, the problem of collision between home appliances and dynamic objects is solved, and the intelligence and user experience of smart home appliances are improved.

CN118963179APending Publication Date: 2024-11-15MIDEA ROBOZONE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411001194.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing home appliances collide with dynamic objects (such as pets and children) during work, reducing the harmony between smart home appliances and home dynamic objects, and it is difficult to effectively avoid disturbing the normal life of dynamic objects.

Method used

By collecting multimodal information (video, audio, odor, etc.) in home appliances, building a portrait of dynamic objects, and adjusting execution instructions based on the analysis results to avoid disturbing the normal life of dynamic objects and improving the intelligence of home appliances.

Benefits of technology

It realizes the normal life of dynamic objects during the work of home appliances, improves the intelligence and user experience of home appliances, and enhances the harmonious relationship with dynamic objects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118963179A_ABST
    Figure CN118963179A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of household appliance control, and particularly discloses a household appliance and a control method and device thereof, and a storage medium, and the control method comprises the steps: collecting the multi-mode information of the environment where the household appliance is located when the household appliance works according to an execution instruction; constructing a portrait of a dynamic object in the environment according to the multi-modal information; performing state analysis on the dynamic object according to the portrait to obtain an analysis result; and the execution instruction is adjusted according to the analysis result, and the household appliance is controlled to continue working according to the adjusted execution instruction, so that the normal life of the dynamic object can be prevented from being interfered in the working process, and the intelligence of the household appliance is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of household appliance control technology, and in particular to a household appliance device and a control method and device thereof, and a storage medium. Background Art

[0002] As people's living standards continue to improve, more and more families choose to keep pets to enrich their lives and meet their companionship needs, and home appliances are becoming more and more intelligent. In this context, dynamic objects such as pets will affect the operation of home appliances. For example, during the operation of a sweeping robot, it often collides with dynamic objects such as cats and dogs, greatly reducing the harmony between pets and smart home appliances. Even when there are many children at home, the sweeping robot may affect the children's play. Therefore, developing a home appliance control method to enable smart home appliances to get along harmoniously with dynamic objects at home is an urgent problem to be solved. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the first object of the present invention is to provide a control method for a household appliance, which can identify dynamic objects in the environment during operation and adjust the execution instructions of the household appliance according to the image information feedback of the dynamic objects to avoid interfering with the normal life of the dynamic objects and improve the intelligence of the household appliance.

[0004] A second object of the present invention is to provide a computer-readable storage medium.

[0005] The third objective of the present invention is to provide a control device for household electrical appliances.

[0006] A fourth objective of the present invention is to provide a household appliance.

[0007] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present invention proposes a method for controlling a household appliance, the method comprising: when the household appliance is working to execute instructions, collecting multimodal information of the environment in which the household appliance is located; constructing a portrait of a dynamic object in the environment based on the multimodal information; performing a state analysis on the dynamic object based on the portrait to obtain an analysis result; adjusting the execution instruction based on the analysis result, and controlling the household appliance to continue working according to the adjusted execution instruction.

[0008] The control method of the household appliance in the embodiment of the present invention first collects multimodal information of the environment in which the household appliance is located during the operation of the household appliance, then constructs a portrait of a dynamic object based on the multimodal information, and analyzes the dynamic object based on the portrait, and finally adjusts the execution instructions of the household appliance based on the results of the analysis, so that the household appliance continues to work according to the adjusted execution instructions, thereby avoiding interference with the normal life of the dynamic object during the operation process and improving the intelligence of the household appliance.

[0009] In some embodiments of the present invention, the multimodal information includes video information, audio information and odor information.

[0010] In some embodiments of the present invention, a portrait of the dynamic object in the environment is constructed based on the multimodal information, including: performing target detection and target re-identification processing on the dynamic objects in the video information to obtain detection results and feature vectors of each detection result; performing target association and tracking on the dynamic objects based on the detection results and the feature vectors to determine the target information of the dynamic objects; and constructing a portrait of the dynamic object in the environment based on the target information, the audio information and the odor information.

[0011] In some embodiments of the present invention, the method further includes: issuing an early warning message for reminder when it is determined that the dynamic object makes an abnormal sound based on the audio information, determines that the dynamic object is a strange object based on the target information of the dynamic object, or determines that the dynamic object behaves abnormally based on the portrait.

[0012] In some embodiments of the present invention, when the dynamic object is a pet, the method further includes: obtaining basic information of the pet, the basic information including maintenance information of the pet; and constructing a portrait of the pet based on the target information, the audio information, the odor information and the basic information.

[0013] In some embodiments of the present invention, the analysis results include health analysis results and / or behavior analysis results.

[0014] In some embodiments of the present invention, when the analysis result includes the health analysis result, adjusting the execution instruction according to the analysis result includes: replacing the execution instruction according to the health analysis result to improve the health status of the dynamic object, or when the analysis result includes the behavior analysis result, adjusting the execution instruction according to the behavior analysis result to avoid the dynamic object.

[0015] In some embodiments of the present invention, before constructing a portrait of a dynamic object in the environment based on the multimodal information, the method further includes: preprocessing the multimodal information, wherein the preprocessing includes information filtering, information enhancement, and information classification.

[0016] In some embodiments of the present invention, the household appliance is connected to a mobile terminal device, and the method further includes: storing the image of the dynamic object in the mobile terminal device, so as to display the information of the dynamic object through the mobile terminal device.

[0017] To achieve the above objectives, a second aspect of the present invention provides a computer-readable storage medium on which a control program for a household appliance is stored. When the control program is executed by a processor, the control method for the household appliance described in any one of the above embodiments is implemented.

[0018] The computer-readable storage medium of an embodiment of the present invention executes the control program of the home appliance stored thereon through a processor, so that the home appliance can identify dynamic objects in the environment during operation and adjust the execution instructions of the home appliance according to the image information feedback of the dynamic objects to avoid interfering with the normal life of the dynamic objects and improve the intelligence of the home appliance.

[0019] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a control device for household appliances, wherein the device includes: a collection module, which is used to collect multimodal information of the environment in which the household appliance is located when the household appliance is working to execute instructions; a construction module, which is used to construct a portrait of the dynamic objects in the environment according to the multimodal information; an analysis module, which is used to perform state analysis on the dynamic objects according to the portrait to obtain an analysis result; and a control module, which is used to adjust the execution instruction according to the analysis result, and control the household appliance to continue working according to the adjusted execution instruction.

[0020] The control device of the household appliance in the embodiment of the present invention includes a collection module, a construction module, an analysis module and a control module. First, the collection module is used to collect multimodal information of the environment in which the household appliance is located during the operation of the household appliance. Then, the construction module is used to construct a dynamic object portrait based on the multimodal information, and the analysis module is used to analyze the dynamic object based on the portrait. Finally, the control module is used to adjust the execution instructions of the household appliance according to the results of the analysis, so that the household appliance continues to work according to the adjusted execution instructions, thereby avoiding interference with the normal life of the dynamic object during the operation process and improving the intelligence of the household appliance.

[0021] To achieve the above-mentioned purpose, a fourth aspect of the present invention provides a household appliance, wherein the household appliance includes the control device of the household appliance described in the above-mentioned embodiment.

[0022] The household appliance of the embodiment of the present invention can identify dynamic objects in the environment during operation through the control device of the household appliance of the above embodiment, and adjust the execution instructions of the household appliance according to the image information feedback of the dynamic objects to avoid interfering with the normal life of the dynamic objects and improve the intelligence of the household appliance.

[0023] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of a method for controlling a household appliance in one embodiment of the present invention;

[0025] Figure 2 is a flow chart of a method for controlling a household appliance in another embodiment of the present invention;

[0026] Figure 3 is a flow chart of a control method for household electrical appliances in a specific embodiment of the present invention;

[0027] Figure 4 is a schematic diagram of an application of a control method for household electrical appliances in a specific embodiment of the present invention;

[0028] Figure 5 is a block diagram of a control device for household electrical appliances in an embodiment of the present invention;

[0029] Figure 6 It is a structural block diagram of a household appliance in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0031] The following describes a household appliance, a control method and device thereof, and a storage medium according to an embodiment of the present invention with reference to the accompanying drawings.

[0032] Figure 1 It is a flow chart of a method for controlling a household appliance in one embodiment of the present invention.

[0033] like Figure 1 As shown, the present invention proposes a control method for household electrical appliances, the control method comprising the following steps:

[0034] S10, when the home appliance is working to execute the instruction, multimodal information of the environment in which the home appliance is located is collected.

[0035] Specifically, the household appliances in this embodiment may be sweeping robots, air conditioners, gas stoves, range hoods, etc., and the household appliances are equipped with acquisition devices for collecting multimodal information of the current environment. In some examples, the multimodal information may include video information, audio information, and odor information. It should be noted that the multimodal information may also include other information, such as ambient temperature and humidity information, etc., which may not be specifically limited here, and for different household appliances, the corresponding multimodal information may be different. For example, a range hood may include video information, odor information, audio information, and temperature information, while a sweeping robot may only include video information, odor information, and audio information.

[0036] More specifically, the acquisition devices installed on the home appliance may include camera devices, microphone devices, odor sensors, etc., among which the camera devices correspond to collecting video information, the microphone devices correspond to collecting audio information, and the odor sensors correspond to collecting odor information.

[0037] In some embodiments, after acquiring multimodal information, since multimodal information needs to be used as the basis for subsequent steps to establish a portrait and generate relevant output data, in order to improve the accuracy of the output data, the embodiment can also pre-process the multimodal information, wherein the pre-processing includes information filtering, information enhancement and information classification. Specifically, blurred, overexposed, and dark video information can be filtered out so that subsequent processing can be performed on high-quality data. For the captured audio information, noise reduction, echo cancellation, and irrelevant audio can be removed to highlight the sound emitted by dynamic objects. The odor information can also be filtered to eliminate the influence of useless data and extract useful odor features. Information enhancement can rotate, scale, mirror, and crop the video information to increase the diversity of the data, play the audio information at a variable speed, change the volume, add background noise, etc., and combine odor molecules to create new odor data. By enhancing the information of multimodal information, the diversity of relevant information can be increased to improve the generalization ability of the portrait model. Information classification is specifically for the convenience of model training. The collected video information, audio information and odor information are annotated. The main annotation objects can be the position, category, sound category of human bodies and pets in the image, and the type of pet odor. In other words, dynamic objects appearing in video information can be explained through information classification.

[0038] S20, constructing a portrait of the dynamic object in the environment based on the multimodal information.

[0039] Specifically, after collecting the multimodal information in the environment, the dynamic objects appearing in the environment can be determined, and the portrait of the dynamic object can be constructed through the multimodal information. In this embodiment, portraits can be established for different dynamic objects, so in step S10, a data set is also produced, and a multimodal data set is constructed specifically according to the annotated multimodal data, wherein the image data set may include an image file and its corresponding image label file, and the image label file may include information such as the category and location of the target; the sound data set may also include an audio file and its corresponding audio label file, and the audio label file may include the type of sound made by a human voice or a pet; the odor data set also includes an odor data file and its corresponding odor label file, and the odor label file includes the odor type of the dynamic object, etc. The image data set can also be divided into a target detection data set and a re-identification data set, wherein the target detection data set adopts the YOLO (You Only Look Once) format, and the re-identification data set is obtained by intercepting the image in the detection box in the target detection data set, and is divided according to the individual dynamic objects. In addition, in some examples, the creation of a portrait of a dynamic object can also be achieved by the user directly inputting relevant information into a mobile terminal device, wherein the mobile terminal device is connected to a home appliance, and the user can input relevant information of the dynamic object into the home appliance through an application installed on the mobile terminal device (such as a smart home APP), so that the home appliance can successfully complete the creation of the portrait of the dynamic object.

[0040] S30, performing state analysis on the dynamic object according to the image to obtain an analysis result.

[0041] Specifically, after constructing the portrait of each dynamic object, the state analysis of the dynamic object is further performed according to the corresponding portrait to determine the current state of the dynamic object and generate an analysis result. In some examples, the analysis results include health analysis results and / or behavior analysis results. In this embodiment, the state analysis can be performed by comparing with a database to query the database according to the current state of the dynamic object, and the machine learning algorithm of the decision tree can be used to analyze the dynamic object to obtain the corresponding analysis result. Of course, other analysis methods can also be used. For example, when the dynamic object is a person, it can also be determined by detecting the body temperature to determine whether there is a problem with the person's health status, etc., to ensure the accuracy of the analysis results.

[0042] S40, adjusting the execution instruction according to the analysis result, and controlling the home appliance to continue working according to the adjusted execution instruction.

[0043] Specifically, after obtaining the analysis results of the dynamic object, the health status and / or behavior status of the dynamic object can be determined based on the analysis results, and then the execution instructions of the home appliance can be adjusted according to the obtained health status and / or behavior status, and then the home appliance can be controlled to continue working according to the adjusted execution instructions, thereby avoiding interference with the dynamic object and improving the harmonious coexistence of the dynamic object and the home appliance.

[0044] In some embodiments, adjusting execution instructions according to analysis results includes: when the analysis results include health analysis results, replacing execution instructions according to the health analysis results to improve the health status of dynamic objects, or when the analysis results include behavior analysis results, adjusting execution instructions according to the behavior analysis results to avoid dynamic objects.

[0045] Specifically, take the air conditioner as an example of household appliances for explanation. When it is determined that the person or pet in the current environment has a fever or a cold according to the health analysis results, the refrigeration temperature of the air conditioner can be controlled to rise or the air conditioner can be turned off directly. Of course, if it is determined that the person in the current environment has a dry cough, the execution instruction of the air conditioner can also be adjusted to humidification to improve the health status of dynamic objects. Take the sweeping robot as an example for explanation. When it is determined that the person or pet in the current environment is in the path to be cleaned according to the behavior analysis results, the execution instruction of the sweeping robot can be adjusted to avoid the person or pet to avoid colliding with the person or pet. Of course, if the pet is detected to be playing, a laser can be fired or a sound can be made to amuse the pet. This embodiment can increase the functions of household appliances, meet the diverse needs of users and pets, and enable users to reduce the purchase of other devices with the same functions, reducing user expenses.

[0046] In some embodiments of the present invention, Figure 2 As shown in the figure, a portrait of a dynamic object in the environment is constructed based on multimodal information, including:

[0047] S201, performing target detection and target re-identification processing on dynamic objects in video information to obtain detection results and feature vectors of each detection result.

[0048] Specifically, this embodiment can first participate in the construction of a target detection model used in target detection and a re-identification model involved in target re-identification, wherein the YOLOv7 algorithm can be used to perform real-time human and pet detection on the input image. As a target detection algorithm, YOLOv7 achieves higher accuracy while maintaining real-time performance. The algorithm divides the input image into S×S networks, and assigns B bounding boxes and C category probabilities to each network, and predicts the specific values ​​of B and C through convolutional neural networks (CNN). For each bounding box, the output includes x, y, w, h, and a confidence score, wherein S×S represents the number of grids in the feature map, B represents the number of bounding boxes in each grid, (x, y) represents the center of the box, and (w, h) represents the width and height of the box. The loss function of the neural network can be loss = λ_coord*(coordinate_loss+scale_loss)+λ_noobj*confidence_loss+class_loss, where λ_coord and λ_noobj are parameters for adjusting coordinate and no-target losses, coordinate_loss, scale_loss, confidence_loss and class_loss are coordinate loss, scale loss, confidence loss and category loss respectively.

[0049] In the process of target re-identification, it is necessary to first perform target recognition, then extract human and pet images from the target recognition results, and extract feature vectors based on ResNet50 and Triplet Loss. Triplet Loss is a commonly used metric learning loss function that aims to make the distance between samples of the same type as close as possible, and the distance between samples of different types as far as possible, so that the trained model can obtain a good feature space and map samples with similarity to close positions. The Triplet Loss formula in metric learning is as follows: L(A, P, N) = max(d(A, P) - d(A, N) + margin, 0), where A represents the anchor sample, P represents the same type of sample, N represents the different type of sample, and d represents the distance between the two samples. Margin is a hyperparameter used to ensure that there is a certain interval between positive samples and negative samples.

[0050] S202, target association and tracking are performed on the dynamic object according to the detection result and the feature vector to determine the target information of the dynamic object.

[0051] Specifically, after obtaining the detection results and feature vectors, the dynamic object can be associated and tracked based on the detection results and feature vectors. More specifically, for each target in the current frame, the distance between samples is calculated. If the distance is less than a preset threshold, it is considered that a match is found, that is, the re-identification is successful; if no matching sample is found for the target detected in the current frame, it means that the target may be a new target, that is, the dynamic object is an unfamiliar object, such as a stranger. Through the target association and tracking method of this embodiment, the target information of the dynamic object, that is, the person or pet appearing in the video information, can be determined.

[0052] S203, constructing a portrait of a dynamic object in the environment based on the target information, audio information and odor information.

[0053] Specifically, after determining that the target information is obtained, a portrait of the dynamic object in the environment can be constructed based on the target information and the audio information and odor information in the multimodal information. The portrait can reflect the behavior, activity patterns, preferences, sound characteristics, and odor characteristics of the dynamic object. More specifically, taking the dynamic object as a pet, based on the target information, the mini-gpt4 visual large model recognition technology can be used to analyze and identify the behavioral characteristics of the pet, such as running, calling, playing, resting, eating, defecating, biting wires, vomiting, etc.; based on the audio information, the Transformer deep learning algorithm can be used to analyze and identify the sounds made by the pet, and detect abnormal sounds or emotional changes, such as pain, fear, anxiety, etc.; based on the odor information, SVM (Support Vector Machines) can be used to classify and identify the pet's odor to monitor potential health problems.

[0054] In this embodiment, if the dynamic object is a pet, such as a cat or a dog, the basic information of the pet can be included in the construction of the pet portrait to further improve the accuracy of the portrait. More specifically, the user can input the basic information of the pet, such as breed, age, gender, etc., through the smart home application, thereby facilitating accurate management of the pet and reducing the probability of error.

[0055] In some embodiments of the present invention, the control method further includes: issuing an early warning message to remind when it is determined that a dynamic object makes an abnormal sound based on audio information, determines that the dynamic object is a strange object based on target information of the dynamic object, or determines that the dynamic object behaves abnormally based on a portrait.

[0056] Specifically, the present embodiment is also provided with an early warning mechanism. According to the audio information, it can be determined that the dynamic object makes abnormal sounds, such as screams of fear, pain, etc., and then an early warning message can be issued. The early warning message can serve as a reminder to the user, and can even provide some helpful guidance for alleviating abnormal information. More specifically, the present embodiment can use acoustic event detection (AED, Acoustic Event Detection) technology to identify abnormal sounds made by pets and crying sounds of babies. The AED method is usually based on convolutional neural networks (CNN) or long short-term memory networks (LSTM, Long Short-Term Memory), which can learn the acoustic characteristics of different types of sounds. When a sound significantly different from a normal sound is detected (such as a pet's pain, panic cry or a baby's cry), an early warning message is issued for reminder. When the dynamic object in the current environment is determined to be a strange object according to the target information of the dynamic object, the strange object can be further tracked, and an early warning message can be issued in time to remind, so as to play a warning role. In addition, when the dynamic object is a pet, the abnormal behavior can be a pet biting an electric wire, a pet vomiting, etc. Of course, if the dynamic object is not a pet, it can also be a child crying, etc. When an abnormality is detected in a dynamic object, an early warning is also issued. When the home appliance of this embodiment detects an abnormality in the current environment through audio-visual information, image information, etc., an early warning can be issued in time to remind the user, and the user is notified to handle the abnormality in time, thereby reducing the accident rate and improving the user's experience.

[0057] In a specific embodiment of the present invention, Figure 3As shown, first, the household appliance collects the data of the environment in which it is located, then cleans the data, and determines whether the cleaned data meets the standard. If it meets the standard, data enhancement is performed, otherwise it is deleted. For the standard judgment, a threshold can be set for comparison, and the thresholds corresponding to different types of data are different. After completing the data enhancement, the data is annotated and processed, and then the corresponding data set is made. The data set can include an image data set (i.e., a video data set), an audio data set, and an odor data set. After the data set is made, the target detection and re-identification processing can be further performed according to the image data set to obtain the detection result and the feature vector, and then the target association and tracking are performed according to the detection result and the feature vector to obtain the target information. It can be understood that the target information can be used to determine whether there is a stranger in the environment. And the basic information of the user is queried online through pet care, and then the pet portrait is constructed according to the target information, basic information, audio information and odor information. After the construction of the pet portrait is completed, it can be further determined whether the current pet has abnormal behavior, and health analysis can also be performed according to the pet portrait, and then the behavior strategy of the household appliance is adjusted according to the health analysis result, and then the actuator in the household appliance is assigned tasks according to the adjusted behavior strategy. This embodiment can also determine whether there is an abnormal sound in the dynamic object through audio data. If the dynamic object has an abnormal sound, abnormal behavior or is a stranger, then intelligent monitoring and health warning can be further performed, and the actuator in the home appliance can also perform corresponding operations according to the control signal provided by the intelligent monitoring and health warning. The functional diversity of home appliances is fully improved, and the execution instructions can be automatically and intelligently adjusted. It can also promptly alarm and notify users when an abnormality occurs, so as to improve the user experience.

[0058] It should be noted that the household appliance of this embodiment may also include a speech analysis module, which may specifically adopt an end-to-end automatic speech recognition (ASR) and natural language understanding (NLU) system to analyze and understand the collected human speech. The system combines deep learning technologies, such as recurrent neural network (RNN) or Transformer architecture, and attention mechanism, which can directly capture context-related information in the speech from the speech signal, and can also retain the speaker's tone and emotional information to determine the owner's intention and parameters. For example, after determining that the owner's intention is to summon the sweeping robot to clean, it is necessary to further determine the room type or specific area.

[0059] The online query of pet care in this embodiment can be queried in the smart home application. A pet knowledge base is built in the reference program. Specifically, pet breeding tutorials, nutrition manuals, disease prevention and other related materials can be queried from the Internet to form a structured pet knowledge base. Specifically, TF-IDF (Term Frequency-Inverse Document Frequency) information retrieval technology can be used to find content related to user needs in the pet knowledge base. In addition, knowledge graph technology and reasoning algorithms can also be applied to mine the associated information implicit in the knowledge base to better meet user needs. Moreover, based on the knowledge obtained by retrieval and reasoning, the GPT series model can be used to convert this knowledge information into easy-to-understand natural language text for users to read.

[0060] Furthermore, if Figure 4 As shown, the control method of the above embodiment can be applied to the sweeping robot in the figure. The sweeping robot can obtain video information, audio information and odor information of the environment through odor sensors, camera devices, and microphone devices. Then, according to the information input into the smart home application by the user's online query, the smart home application can send the pet information to the sweeping robot, and then the sweeping robot can control the cleaning module, navigation module and / or interactive module to work. In addition, the user can also directly send voice commands to the sweeping robot to control the sweeping robot, and the sweeping robot can send notifications or alarms to the user, thereby improving the functional diversity of the sweeping robot and thus improving the user experience. The specific working method of the sweeping robot can refer to the specific description of the control method of the household appliance in the above embodiment, which will not be repeated here.

[0061] In addition, in some examples, the home appliance device can also be connected to the mobile terminal device, and the home appliance device can also store the image of the dynamic object in the mobile terminal device to display the information of the dynamic object through the mobile terminal device.

[0062] Specifically, taking a mobile terminal device as a mobile phone device as an example, an APP (Application) can be set on the mobile phone device, and multiple icons corresponding to different dynamic objects can be designed on the APP. After the mobile phone device is connected to the home appliance, the icon corresponding to the corresponding dynamic object can be selected through the APP. After clicking the icon, multiple information of the dynamic object can be displayed on the mobile phone device, such as location information, status information, age and gender information, etc., so that the user can obtain the information of each pet in time and manage the pet better.

[0063] In summary, the control method of the household appliance in the embodiment of the present invention can identify dynamic objects in the environment during operation, and adjust the execution instructions of the household appliance according to the portrait information feedback of the dynamic objects to avoid interfering with the normal life of the dynamic objects and improve the intelligence of the household appliance.

[0064] Furthermore, the present invention proposes a computer-readable storage medium on which a control program for a household appliance is stored. When the control program is executed by a processor, the control method for the household appliance of any one of the above embodiments is implemented.

[0065] The computer-readable storage medium of an embodiment of the present invention executes the control program of the home appliance stored thereon through a processor, so that the home appliance can identify dynamic objects in the environment during operation and adjust the execution instructions of the home appliance according to the image information feedback of the dynamic objects to avoid interfering with the normal life of the dynamic objects and improve the intelligence of the home appliance.

[0066] Figure 5 4 is a block diagram of a control device for household electrical appliances in an embodiment of the present invention.

[0067] Furthermore, if Figure 5 As shown, the present invention proposes a control device 100 for a household appliance. The control device 100 includes a collection module 101 , a construction module 102 , an analysis module 103 and a control module 104 .

[0068] Among them, the acquisition module 101 is used to collect multimodal information of the environment in which the home appliance is located when the home appliance is working to execute instructions; the construction module 102 is used to construct a portrait of the dynamic objects in the environment based on the multimodal information; the analysis module 103 is used to perform state analysis on the dynamic object based on the portrait to obtain an analysis result; the control module 104 is used to adjust the execution instruction according to the analysis result, and control the home appliance to continue working according to the adjusted execution instruction.

[0069] In some embodiments of the present invention, the multimodal information includes video information, audio information and scent information.

[0070] In some embodiments of the present invention, the construction module 102 is specifically used to: perform target detection and target re-identification processing on dynamic objects in video information to obtain detection results and feature vectors of each detection result; perform target association and tracking on dynamic objects based on the detection results and feature vectors to determine target information of the dynamic objects; and construct portraits of dynamic objects in the environment based on target information, audio information and odor information.

[0071] In some embodiments of the present invention, the control module 104 is also used to issue an early warning message to provide a reminder when it is determined that a dynamic object makes an abnormal sound based on audio information, determines that the dynamic object is a strange object based on target information of the dynamic object, or determines that the dynamic object behaves abnormally based on a portrait.

[0072] In some embodiments of the present invention, when the dynamic object is a pet, the construction module 102 is also used to: obtain basic information of the pet, including the pet's maintenance information; and construct a portrait of the pet based on the target information, audio information, odor information and basic information.

[0073] In some embodiments of the present invention, the analysis results include health analysis results and / or behavior analysis results.

[0074] In some embodiments of the present invention, the control module 104 is specifically used to: change the execution instruction according to the health analysis result to improve the health status of the dynamic object, or adjust the execution instruction according to the behavior analysis result to avoid the dynamic object.

[0075] In some embodiments of the present invention, the control module 104 is further used to: pre-process the multimodal information before constructing a portrait of a dynamic object in the environment based on the multimodal information, wherein the pre-processing includes information filtering, information enhancement and information classification.

[0076] It should be noted that the specific implementation of the control device of the household appliance in the embodiment of the present invention can refer to the specific implementation of the control method of the household appliance in the above embodiment, and will not be described again here to avoid redundancy.

[0077] In summary, the control device of the household appliance in the embodiment of the present invention can enable the household appliance to identify dynamic objects in the environment during operation, and adjust the execution instructions of the household appliance according to the image information feedback of the dynamic objects to avoid interfering with the normal life of the dynamic objects and improve the intelligence of the household appliance.

[0078] Figure 6 It is a structural block diagram of a household appliance in an embodiment of the present invention.

[0079] Furthermore, if Figure 6 As shown, the present invention proposes a household appliance 200, which includes the control device 100 of the household appliance in the above embodiment.

[0080] The household appliance of the embodiment of the present invention can identify dynamic objects in the environment during operation through the control device of the household appliance of the above embodiment, and adjust the execution instructions of the household appliance according to the image information feedback of the dynamic objects to avoid interfering with the normal life of the dynamic objects and improve the intelligence of the household appliance.

[0081] In addition, other structures and functions of the household appliance in the embodiment of the present invention are known to those skilled in the art and will not be described in detail here to reduce redundancy.

[0082] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0083] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0084] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0085] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0086] In addition, the terms "first", "second", etc. used in the embodiments of the present invention are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the present embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present invention can explicitly or implicitly indicate that the embodiment includes at least one of the features. In the description of the present invention, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0087] In the present invention, unless otherwise clearly specified or limited in the embodiments, the terms "installed", "connected", "connected" and "fixed" etc. in the embodiments should be understood in a broad sense. For example, the connection can be a fixed connection, a detachable connection, or an integrated connection. It can be understood that it can also be a mechanical connection, an electrical connection, etc.; of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements, or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific implementation situation.

[0088] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0089] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for controlling a household appliance, characterized in that: The method comprises: When the household appliance is working to execute the instruction, collecting multimodal information of the environment in which the household appliance is located; Constructing a portrait of a dynamic object in the environment according to the multimodal information; Performing state analysis on the dynamic object according to the image to obtain an analysis result; The execution instruction is adjusted according to the analysis result, and the household appliance is controlled to continue working according to the adjusted execution instruction.

2. The control method according to claim 1, characterized in that: The multimodal information includes video information, audio information and smell information.

3. The control method according to claim 2, characterized in that: Constructing a portrait of the dynamic object in the environment according to the multimodal information includes: Performing target detection and target re-identification processing on the dynamic objects in the video information to obtain detection results and feature vectors of each detection result; Performing target association and tracking on the dynamic object according to the detection result and the feature vector to determine target information of the dynamic object; A portrait of the dynamic object in the environment is constructed according to the target information, the audio information and the odor information.

4. The control method according to claim 3, characterized in that: The method further comprises: When it is determined according to the audio information that the dynamic object makes an abnormal sound, when it is determined according to the target information of the dynamic object that the dynamic object is a strange object, or when it is determined according to the portrait that the dynamic object makes an abnormal behavior, an early warning message is issued for reminder.

5. The control method according to claim 3, characterized in that: When the dynamic object is a pet, the method further includes: Obtaining basic information of the pet, wherein the basic information includes maintenance information of the pet; A portrait of the pet is constructed according to the target information, the audio information, the odor information and the basic information.

6. The control method according to claim 1, characterized in that: The analysis results include health analysis results and / or behavior analysis results.

7. The control method according to claim 6, characterized in that: Adjusting the execution instruction according to the analysis result includes: When the analysis result includes the health analysis result, replacing the execution instruction according to the health analysis result to improve the health state of the dynamic object, or When the analysis result includes the behavior analysis result, the execution instruction is adjusted according to the behavior analysis result to avoid the dynamic object.

8. The control method according to any one of claims 1 to 7, characterized in that: Before constructing a portrait of the dynamic object in the environment according to the multimodal information, the method further includes: The multimodal information is preprocessed, wherein the preprocessing includes information filtering, information enhancement and information classification.

9. The control method according to any one of claims 1 to 7, characterized in that: The household appliance is connected to a mobile terminal device, and the method further comprises: The image of the dynamic object is stored in the mobile terminal device, so that the information of the dynamic object can be displayed through the mobile terminal device.

10. A computer-readable storage medium, characterized in that: A control program for a household appliance is stored thereon, and when the control program is executed by a processor, the control method for the household appliance described in any one of claims 1 to 9 is implemented.

11. A control device for household electrical appliances, characterized in that: The device comprises: A collection module, used for collecting multimodal information of the environment in which the household appliance is located when the household appliance is working to execute instructions; A construction module, used to construct a portrait of the dynamic object in the environment according to the multimodal information; An analysis module, used for performing state analysis on the dynamic object according to the image to obtain an analysis result; The control module is used to adjust the execution instruction according to the analysis result, and control the home appliance to continue working according to the adjusted execution instruction.

12. A household appliance, characterized in that: The household appliance includes the control device for the household appliance according to claim 11.