Equipment control method and device, air conditioning equipment and storage medium

Abstract

The invention relates to an equipment control method and device, air conditioning equipment and a storage medium, and the equipment control method comprises the steps that audio data and a depth image of a space where first equipment is located are obtained; determining object information of an object in the space according to the audio data and the depth image; determining whether the object in the space meets a mode switching trigger condition or not according to the object information; and if it is determined that the object in the space meets the mode switching triggering condition, controlling the first device to be switched to a target working mode. According to the embodiment of the invention, through a multi-modal data fusion technology, the object information of the object in the space is accurately identified, and when the object in the space meets the mode switching trigger condition, the working mode of the first equipment is automatically switched, so that a personalized service mode is provided for the object in the space, and convenience is brought to a user.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a device control method, apparatus, air conditioning equipment and storage medium. Background Art

[0002] In the prior art, some air-conditioning products use infrared sensors or cameras as core detection elements. Among them, infrared sensors mainly determine whether there is someone in the room by capturing infrared rays of specific wavelengths emitted by the human body. When a human body enters its sensing range, the sensor receives the infrared signal radiated by the human body, and then triggers the corresponding detection mechanism. The camera uses image recognition technology to analyze the captured image and identify whether there are features such as human outlines in the image to determine whether there is someone in the room. Based on these detection results, the air conditioner can intelligently switch operating modes. For example, when it is detected that there is no one in the room, it automatically switches to energy-saving mode to reduce energy consumption and achieve efficient use of energy; or the automatic start-stop function is activated to avoid unnecessary power loss caused by continuous operation when no one is in the room.

[0003] However, this type of air conditioning product has great limitations in terms of personnel detection. They can only provide simple presence information of whether there is someone in the room. From a technical point of view, the existing algorithms and functions of infrared sensors and cameras can only complete the binary judgment of whether there is a person. Infrared sensors cannot subdivide the characteristics of different human bodies, and the image recognition technology of cameras only stays at the basic level of human body contour recognition, and it is difficult to go deep into the precise distinction of user identities. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides an equipment control method, device, air-conditioning equipment and storage medium.

[0005] In a first aspect, the present application provides a device control method, comprising:

[0006] Acquire audio data and a depth image of a space where the first device is located;

[0007] Determine object information of an object in the space according to the audio data and the depth image;

[0008] determining, according to the object information, whether the object in the space satisfies a mode switching trigger condition;

[0009] If it is determined that the object in the space meets the mode switching trigger condition, the first device is controlled to switch to the target working mode.

[0010] Optionally, determining object information of an object in the space according to the audio data and the depth image includes:

[0011] Performing sound detection on the audio data to obtain a sound detection result;

[0012] identifying three-dimensional contour information in the depth image;

[0013] The object information of each object in the space is determined according to the sound detection result and the three-dimensional contour information.

[0014] Optionally, determining whether the object in the space satisfies a mode switching trigger condition according to the object information includes:

[0015] Determining whether the space contains a target object according to the object information;

[0016] If the space contains the target object, determining whether the space contains only the target object within a preset time period according to the object information;

[0017] If the space only includes the target object within a preset time period, it is determined that the object in the space meets the mode switching trigger condition.

[0018] Optionally, the method further comprises:

[0019] Performing scene recognition on the depth image to obtain scene type information corresponding to the space;

[0020] determining an object state of the target object according to the scene type information, the audio data, and the depth image;

[0021] A second device associated with the object state is controlled to switch to a target working mode corresponding to the object state.

[0022] Optionally, determining the object state of the target object according to the scene type information, the audio data, and the depth image includes:

[0023] If the space type corresponding to the scene type information is an entertainment space, determining that the object state of the target object is an entertainment state;

[0024] Alternatively, if the space type corresponding to the scene type information is a non-entertainment space, the object state of the target object is determined according to the audio data and the depth image.

[0025] Optionally, determining the object state of the target object according to the audio data and the depth image includes:

[0026] Acquire a sound detection result obtained by performing sound detection on the audio data and three-dimensional contour information obtained by performing contour detection on the depth image;

[0027] If it is determined according to the sound detection result that the volume of the sound in the space is within the quiet volume range and the posture of the target object is determined to be a learning posture according to the three-dimensional contour information, the object state of the target object is determined to be a learning state; or, if it is determined according to the sound detection result that the volume of the sound in the space is within the quiet volume range and the posture of the target object is determined to be a resting posture according to the three-dimensional contour information, the object state of the target object is determined to be a resting state.

[0028] Optionally, controlling a second device associated with the object state to switch to a target working mode corresponding to the object state includes:

[0029] Acquire a device set associated with the first device;

[0030] Determining a second device corresponding to the object state in the device set;

[0031] Acquire a target operating mode corresponding to the object state for each second device;

[0032] Control the second device to switch to the target working mode.

[0033] In a second aspect, the present application provides a device control apparatus, comprising:

[0034] A first acquisition module, used to acquire audio data and a depth image of a space where the first device is located;

[0035] A first determining module, configured to determine object information of an object in the space according to the audio data and the depth image;

[0036] A second determination module, configured to determine whether an object in the space satisfies a mode switching trigger condition according to the object information;

[0037] The first control module is used to control the first device to switch to a target working mode if it is determined that the object in the space meets the mode switching trigger condition.

[0038] Optionally, the first determining module includes:

[0039] A sound detection unit, used to perform sound detection on the audio data to obtain a sound detection result;

[0040] A contour recognition unit, used to recognize three-dimensional contour information in the depth image;

[0041] The first determining unit is used to determine the object information of each object in the space according to the sound detection result and the three-dimensional contour information.

[0042] Optionally, the second determining module includes:

[0043] A second determining unit, configured to determine whether the space contains a target object according to the object information;

[0044] a third determining unit, configured to determine, if the space contains a target object, whether the space contains only the target object within a preset time period according to the object information;

[0045] The fourth determining unit is configured to determine whether the object in the space satisfies a mode switching trigger condition if the space only contains the target object within a preset time period.

[0046] Optionally, the method further comprises:

[0047] A recognition module, used to perform scene recognition on the depth image to obtain scene type information corresponding to the space;

[0048] A third determination module, configured to determine an object state of the target object according to the scene type information, the audio data and the depth image;

[0049] The second control module is used to control a second device associated with the object state to switch to a target working mode corresponding to the object state.

[0050] Optionally, the third determining module includes:

[0051] A fifth determining unit, configured to determine that the object state of the target object is an entertainment state if the space type corresponding to the scene type information is an entertainment space;

[0052] Alternatively, a sixth determination unit is configured to determine the object state of the target object according to the audio data and the depth image if the space type corresponding to the scene type information is a non-entertainment space.

[0053] Optionally, the sixth determining unit includes:

[0054] an acquisition subunit, configured to acquire a sound detection result obtained by performing sound detection on the audio data and three-dimensional contour information obtained by performing contour detection on the depth image;

[0055] A determination subunit is used to determine that the object state of the target object is a learning state if it is determined according to the sound detection result that the volume of the sound in the space is within a quiet volume range and the posture of the target object is determined according to the three-dimensional contour information to be a learning posture; or to determine that the object state of the target object is a resting state if it is determined according to the sound detection result that the volume of the sound in the space is within a quiet volume range and the posture of the target object is determined according to the three-dimensional contour information to be a resting posture.

[0056] Optionally, the second control module includes:

[0057] A first acquiring unit, configured to acquire a device set associated with the first device;

[0058] a seventh determining unit, configured to determine a second device corresponding to the object state in the device set;

[0059] A second acquisition unit, configured to acquire a target operating mode corresponding to the object state for each second device;

[0060] A control unit is used to control the second device to switch to the target working mode.

[0061] In a third aspect, the present application provides an air conditioning device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0062] Memory, used to store computer programs;

[0063] The processor is used to implement any device control method described in the first aspect when executing the program stored in the memory.

[0064] In a fourth aspect, the present application provides a computer-readable storage medium, on which a program of a device control method is stored. When the program of the device control method is executed by a processor, the steps of the device control method described in any one of the first aspects are implemented.

[0065] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0066] The embodiments of the present application can determine the object information of objects in the space based on the audio data and depth images collected in the space, and when it is determined that the first device meets the mode switching condition based on the object information, control the first device to switch to the target working mode, and through multimodal data fusion technology, accurately identify the object information of objects in the space, and automatically switch the working mode of the first device when the objects in the space meet the mode switching trigger condition, so as to provide personalized service mode for the objects in the space, which is convenient for users to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0069] Figure 1 A flow chart of a device control method provided in an embodiment of the present application;

[0070] Figure 2 A flowchart of a device control method in a practical application provided by an embodiment of the present application;

[0071] Figure 3 A structural diagram of a device control device provided in an embodiment of the present application;

[0072] Figure 4 A structural diagram of an air conditioning device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0074] Current air conditioning products have great limitations in terms of personnel detection. They can only provide simple presence information of whether there is someone in the room. From a technical point of view, the existing algorithms and functions of infrared sensors and cameras can only complete the binary judgment of whether there is a person. Infrared sensors cannot subdivide the characteristics of different human bodies, and the image recognition technology of cameras only stays at the basic level of human body contour recognition, and it is difficult to go deep into the accurate distinction of user identities. To this end, the embodiments of the present application provide a device control method, device, air conditioning equipment and storage medium.

[0075] The embodiment of the present application provides a smart home system, which includes one or more smart devices, and each smart device communicates and transmits data through wireless communication. The home smart system, with multi-scenario adaptation as the core, deeply integrates the interconnection and interoperability of children's safety, health and home smart devices. All smart devices are equipped with wireless communication module units, and respond in linkage based on their respective sensors, edge algorithms and rule engines. Parents can remotely monitor and intervene through mobile phone apps, and can understand the situation of children being alone in the family through the APP.

[0076] The air-conditioning equipment provided in the embodiment of the present application includes: a main control unit, a sound sensor unit, a voice module unit, a TOF sensor module unit, an edge computing module unit, a wireless communication module unit, etc.

[0077] Among them, the main control unit is responsible for logical scheduling and control; the sound sensor unit, that is, the microphone array, is mainly responsible for collecting real-time audio signals in the room; the voice module unit is responsible for sound processing and analysis, identifying the age level classification of sound, and is also responsible for voice recognition, voice broadcast processing, etc.; TOF sensor module unit, TOF calculates the distance through the light flight time, generates a depth image or three-dimensional outline of objects in the room, and can combine the algorithm to judge the height and volume characteristics, and can also analyze the child's sitting posture, head and hand movement characteristics; the edge computing module unit is responsible for image recognition, performs image recognition on the room depth map generated by TOF, identifies the scene type, identifies scenes such as living room, bedroom or study, and is also configured with a multimodal data fusion analysis algorithm. The multimodal data fusion analysis algorithm can identify the child's status, such as learning status, non-learning status, etc.; the wireless module communication unit is responsible for communication between devices, and the communication method can select WiFi, Bluetooth and other communication methods.

[0078] In an embodiment of the present application, the air-conditioning equipment can be used as a central control device, relying on wireless communication (such as device networking) to achieve connection with smart devices such as smart lights, door locks, and televisions.

[0079] The present application embodiment provides a device control method, such as Figure 1 As shown, the first device may include the following steps:

[0080] Step S101, obtaining audio data and a depth image of a space where a first device is located;

[0081] In an embodiment of the present application, the first device may refer to an air-conditioning device. In actual applications, the first device may also refer to other smart devices in the home, such as smart TVs, smart lamps, smart panels, etc. The space where the first device is located refers to the space where the first device is installed, such as a living room, bedroom, study, etc.

[0082] Audio data can be collected through a microphone, and depth images can be collected through a TOF sensor. The TOF sensor can measure the distance between itself and an object by the time it takes for pulsed light to reflect from the surface of an object and return, and generate a depth image based on the returned distance.

[0083] Step S102, determining object information of an object in the space according to the audio data and the depth image;

[0084] In this step, multimodal fusion analysis can be performed on the audio data and depth image, that is, the audio data and the depth image are time-aligned, and the object information of each object in the space is determined by combining the sound information (pitch, loudness, timbre, etc.) carried in the audio data and the motion posture information carried in the depth image.

[0085] In one implementation of the present application, determining object information of the object in the space according to the audio data and the depth image includes:

[0086] Performing sound detection on the audio data to obtain a sound detection result; identifying three-dimensional contour information in the depth image; and determining object information of each object in the space according to the sound detection result and the three-dimensional contour information.

[0087] The information carried in the sound detection results and the three-dimensional contour information can be directly added to the object information, and the information page obtained by analyzing the sound detection results and the three-dimensional contour information can also be added to the object information. The object information may include the number, age group, pitch, loudness, timbre, movement posture and other information of the objects.

[0088] Step S103, determining whether the object in the space meets the mode switching trigger condition according to the object information;

[0089] In the embodiment of the present application, the mode switching trigger condition may refer to the space containing the target object or only containing the target object, and the target object may refer to children or other special groups.

[0090] In one implementation of the present application, determining whether the object in the space meets the mode switching trigger condition according to the object information includes:

[0091] Determine whether the space contains a target object according to the object information; if the space contains a target object, determine that the object in the space meets a mode switching trigger condition.

[0092] In another implementation of the present application, determining whether the object in the space satisfies the mode switching trigger condition according to the object information includes:

[0093] Determine whether the space contains the target object according to the object information; if the space contains the target object, determine whether the space only contains the target object within a preset time period according to the three-dimensional contour information; if the space only contains the target object within the preset time period, determine that the object in the space meets the mode switching trigger condition.

[0094] In another implementation of the present application, determining whether the object in the space satisfies the mode switching trigger condition according to the object information includes:

[0095] Determine whether the space contains the target object according to the object information; if the space contains the target object, determine whether the space only contains the target object within a preset time period according to the sound detection result; if the space only contains the target object within the preset time period, determine that the object in the space meets the mode switching trigger condition.

[0096] Through the above implementation methods, scenes that contain the target object or only contain the target object in the space can be accurately screened to trigger switching of the working mode of the first device, thereby providing personalized services for the target object, so as to provide the target object with an adaptive living environment and ensure the safety and comfort of the target object.

[0097] Step S104: If it is determined that the object in the space meets the mode switching trigger condition, control the first device to switch to the target working mode.

[0098] In an embodiment of the present application, the target working mode may refer to a mode suitable for children, and the control parameters of the target working mode are different from the control parameters of the ordinary adult mode.

[0099] In this step, control parameters of the target operating mode may be acquired, and the first device may be controlled to operate according to the control parameters of the target operating mode.

[0100] The embodiments of the present application can determine the object information of the objects in the space based on the audio data and depth images collected in the space, and when it is determined that the first device meets the mode switching condition based on the object information, the first device is controlled to switch to the target working mode, and through the multimodal data fusion technology, the object information of the objects in the space can be accurately identified, and when the objects in the space meet the mode switching trigger condition, the working mode of the first device is automatically switched to provide a personalized service mode for the objects in the space, which is convenient for users to use. Moreover, since the audio data and depth images are analyzed, the user privacy can be protected to avoid the situation where the video collected in the space exposes the user privacy.

[0101] In another embodiment of the present application, the method further includes:

[0102] Step S201, performing scene recognition on the depth image to obtain scene type information corresponding to the space;

[0103] In the embodiment of the present application, the scene type of the space where the first device is located can be identified through scene recognition, such as: study, living room, bedroom, etc. The scene type information is the identification of the scene type.

[0104] Step S202, determining the object state of the target object according to the scene type information, the audio data and the depth image;

[0105] In this step, the object state of the target object may be determined according to the audio data and the depth image for different scene types.

[0106] In one implementation of the present application, determining the object state of the target object according to the scene type information, the audio data, and the depth image includes:

[0107] If the space type corresponding to the scene type information is an entertainment space, the object state of the target object is determined to be an entertainment state; or, if the space type corresponding to the scene type information is a non-entertainment space, the object state of the target object is determined based on the audio data and the depth image.

[0108] That is to say, when the scene type corresponding to the space is an entertainment space (such as a toy room, living room, etc.), the object state of the target object can be directly determined as an entertainment state; when the scene type corresponding to the space is a non-entertainment space (such as a bedroom, study, etc.), the object state of the target object can be further determined based on the audio data and the depth image.

[0109] Further, determining the object state of the target object according to the audio data and the depth image includes:

[0110] Acquire a sound detection result obtained by performing sound detection on the audio data and three-dimensional contour information obtained by performing contour detection on the depth image;

[0111] Among them, sound detection on audio data can detect the pitch, loudness, timbre, etc. of the object in the audio to obtain the sound detection result, and contour detection is performed on the depth image to obtain three-dimensional contour information of objects around the first device.

[0112] If it is determined according to the sound detection result that the volume of the sound in the space is within the quiet volume range and the posture of the target object is determined to be a learning posture according to the three-dimensional contour information, the object state of the target object is determined to be a learning state; or, if it is determined according to the sound detection result that the volume of the sound in the space is within the quiet volume range and the posture of the target object is determined to be a resting posture according to the three-dimensional contour information, the object state of the target object is determined to be a resting state.

[0113] That is to say, after obtaining the three-dimensional contour information, the three-dimensional contour information can be classified. If there is an object in a learning posture in the space, and the volume of the sound in the space is within the quiet volume range at this time, it can be determined that the target object is in a learning state; if there is an object in a resting posture in the space, and the volume of the sound in the space is within the quiet volume range at this time, it can be determined that the target object is in a resting state.

[0114] Step S203: Control a second device associated with the object state to switch to a target working mode corresponding to the object state.

[0115] In one implementation of the present application, step S203 controls the second device associated with the object state to switch to a target working mode corresponding to the object state, including:

[0116] Acquire a device set associated with the first device; determine a second device corresponding to the object state in the device set; acquire a target operating mode corresponding to the object state for each second device; and control the second device to switch to the target operating mode.

[0117] To facilitate linkage control, the first device may be communicatively connected with multiple second devices, and the multiple second devices constitute a device set. Several of the multiple second devices may have working modes that match the object state. Therefore, a second device corresponding to the object state may be selected from the device set, and the working mode corresponding to the object state in the second device may be obtained as the target working mode, and the second device may be switched to the target working mode to match the working mode of the second device with the object state of the target object.

[0118] The embodiments of the present application directly determine the object state of the target object according to the scene type corresponding to the space, or determine the object state of the target object according to audio data and a depth image, and then control the second device to switch to the corresponding target working mode according to the object state, thereby realizing linkage control of other smart devices adapted to the object state matching with the first device as the core control node, for example: realizing intelligent linkage between air conditioners and smart TVs, smart door locks, alarm systems and lighting equipment.

[0119] For ease of understanding, the following combination Figure 2 An embodiment in practical application is provided.

[0120] First, the sound sensor collects audio data, which is then transmitted to the air conditioner's voice module for real-time analysis and age classification. If only children's voices are detected for a certain period of time, it is determined that there are only children in the room.

[0121] Then, the TOF sensor is combined to further confirm that there are only children in the room. The TOF depth sensing technology is used to detect the body shape and height of people and distinguish between children and adults. The TOF sensor collects scene depth data, generates the outline of the target object, and classifies the outline through an algorithm to determine whether it is a child's feature. If it is a child's feature, the sound sensor and the TOF sensor can be combined to accurately determine that there are only children in the environment. At this time, the air conditioner switches to child mode.

[0122] The air conditioner then determines whether it is a living room, bedroom or study based on the room depth image generated by the TOF sensor. If it is a living room, the air conditioner will simultaneously send a child mode command to the smart devices in the living room (such as: smart TV displaying children's resources, smart door locks opening child locks, smart cameras, etc.). After receiving the command, each smart device will automatically switch to child mode.

[0123] If it is a bedroom or study, the air conditioner main control feedback edge computing module unit identifies the status of the child in the bedroom scene. If the multimodal data fusion analysis algorithm identifies that the child is in a learning state, the air conditioner switches to the learning optimization mode, and the air conditioner operating parameters are adjusted accordingly. At the same time, the air conditioner links the smart lights to intelligently adjust the lighting in the learning environment, and monitors and reminds the child's study time. If the child studies for a long time, a voice broadcast will remind him to relax and play relaxing music; if the child is not concentrating on studying, a voice broadcast will remind him to concentrate on studying.

[0124] Among them, the quiet state can be detected through sound recognition, and the ambient sound in the room can be collected by sound sensors, and the signal processing algorithm is used for analysis to extract features such as sound frequency, intensity (decibel), and duration. Use a noise suppression algorithm to separate background noise and target sound; calculate the average decibel value (such as 40dB or less can be considered quiet). At the same time, a machine learning model is also used to classify sounds: active sounds such as speaking, knocking, and moving, and silent or only low-intensity sounds such as pen tip sliding and book pages turning. If no obvious active sounds are detected within a certain period of time (such as 5 minutes) or only slight learning-related sounds (such as book turning and writing) are detected, the environment is considered to be in a quiet state.

[0125] The learning state can be detected by the TOF sensor, which generates a depth image or three-dimensional outline of objects in the room and analyzes the child's sitting posture, head and hand movement characteristics. In the learning state, the child's head and upper body movements are less, and the main hand movements are concentrated in the desk area. If the TOF data detects a stable sitting posture and a slightly lowered head for a long time, and the hands move in a fixed area (such as turning a book or writing), it can be judged as a learning state. At the same time, regional constraints are used for judgment, and the TOF sensor is used to determine whether the child's activity range is near the desk. If the child leaves the learning area for more than a set time (such as 10 minutes), it is determined that the child is not in the learning state.

[0126] Then fuse the sound with the TOF sensor data. Synchronize the data and time-align the sound detection results (quiet or learning-related sounds) with the behavior detection data of the TOF sensor. If the sound sensor detects the sound of writing or turning pages, and the TOF sensor recognizes a fixed sitting posture and stable head movement, it is comprehensively judged as a learning state. If there is only a TOF detection result without sound characteristics, it may be other quiet activities (such as playing with toys). If TOF detects frequent standing and walking movements, or the sound detects loud activity sounds (such as playing sounds), it is judged as a non-learning state. If a quiet state is detected but there is no obvious learning-related behavior (such as hand movements) for a long time, a prompt will be pushed to the parents.

[0127] The following provides two linkage control examples corresponding to different scene types:

[0128] 1. Living room scenario: Application example of detecting that only children are present in the living room

[0129] Scenario: A 5-year-old child is alone in the living room. The air conditioner detects that there are no adults in the room and automatically switches to child mode and interacts with other smart devices to ensure safety and comfort.

[0130] The air conditioner automatically performs intelligent adjustment and equipment linkage functions, as follows:

[0131] (1) Activate the child lock mode of the air conditioner and optimize the environmental parameters

[0132] The air conditioner detects that there is only a child in the room through voice recognition and TOF sensor, and then switches to child lock mode: only basic adjustment functions are available (such as temperature range set at 24-26℃, wind speed limited to low gear). Complex function buttons are shielded to prevent children from operating them by mistake. The air conditioner activates the air purification function to optimize the air quality. If the humidity is lower than 40%, the humidification function is automatically turned on.

[0133] (2) Linking with smart TVs to push children’s content

[0134] The air conditioner is connected to the smart TV through the home Internet of Things: a child mode switch command is sent to the TV. The smart TV activates the preset child mode and automatically plays educational or entertainment content (such as cartoons and educational games). If the TV is not turned on, the air conditioner attracts the child's attention through voice prompts (such as "Turn on the TV and watch children's content"). If it is our company's Painting Era II air conditioner, the Painting Era II air conditioner TV function is automatically turned on, and the TV function automatically switches to the preset child mode, automatically playing educational or entertainment content (such as cartoons and educational games).

[0135] (3) Linked with smart door locks to prevent children from opening the door and leaving alone

[0136] The air conditioner sends a command to the smart door lock through the home gateway: Enable "child lock mode" to lock the home's outer door, and only authorized mobile phones or passwords can unlock it. If it detects repeated attempts to open the door lock, the air conditioner notifies the parent through the mobile phone app and plays a voice warning on the door lock: "Children, please do not try to open the door."

[0137] (4) Linking smart cameras and alarm systems

[0138] The air conditioner is linked to the smart camera. When the smart camera is turned on, the camera monitors the children's activity area in real time and turns on the "child safety mode": if the camera detects abnormal behavior (such as children trying to climb the windowsill), the alarm is triggered immediately, the air conditioner is notified through the home gateway, the air conditioner voice plays a voice prompt, and the smart camera video is pushed to the parent's mobile phone in real time. If the door and window sensor detects an abnormal opening, the alarm system is activated: the air conditioner voice broadcasts a warning tone to remind the child to stay away from the window. The mobile phone receives notifications on the status of doors and windows. If there are smart door and window sensors, parents can close the doors and windows remotely.

[0139] 2. Bedroom or study scene: Application example of detecting that only children are present in the room

[0140] Scenario: A 10-year-old child is studying alone in the study room. The air conditioner recognizes this and switches to a mode suitable for studying. It also links with other devices to create a healthy and safe learning environment.

[0141] The air conditioner automatically performs intelligent adjustment and equipment linkage functions, as follows:

[0142] (1) Optimized adjustment of air conditioning mode switching

[0143] The air conditioner detects the presence of a child through sound and TOF sensors, and identifies the child as being in a learning state through multimodal data fusion analysis methods. The air conditioner then switches to learning optimization mode: the temperature is set at 25°C to ensure concentration and comfort, the wind direction is automatically adjusted to avoid direct blowing to prevent colds or distractions, the wind speed is adjusted to the lowest, and the low-noise operation mode (20-25dB) is turned on.

[0144] (2) Link with intelligent lighting to adjust the ambient atmosphere

[0145] The air conditioner sends adjustment instructions to the smart light through the gateway: turn on the learning mode, set the light to gradually adjust to cool white light (eye protection), and dynamically adjust the light brightness according to the natural light brightness detected by the ambient light sensor (for example, automatically increase the brightness at night). If the child is detected to be sleeping (such as being quiet for a long time or the TOF sensor detects a lying position), the air conditioner automatically switches to sleep mode and sends instructions: the smart light switches to night light mode, the brightness gradually decreases, and the color temperature is adjusted to warm light (sleep).

[0146] (3) Learning status monitoring

[0147] At the same time, the TOF sensor detects whether the child is near the study table. If it is found that the child has left the study area for a long time (such as more than 30 minutes), the air conditioner will play a voice reminder: "Please stay focused and continue studying." A reminder notification will be sent to the parent's mobile phone to inform the child's learning status. If it is found that the child has been in the study area for a long time (such as more than 2 hours), the air conditioner will play a voice reminder: "If you are tired of studying, you can take a break and relax." At the same time, the air conditioner will play a soothing relaxation audio.

[0148] This application uses a combination of sound sensors and TOF sensors to accurately detect the type of users in the room and their behavior status. The air conditioner is used as the core control node to link smart TVs, smart door locks, alarm systems and smart lighting equipment to provide children with a safe, comfortable and scene-adaptive environment, and each device automatically switches to child mode.

[0149] The embodiments of the present application can achieve high-precision detection: based on the multimodal data fusion of sound and TOF sensors, the identity and status of children in the room can be accurately identified; environmental optimization can be achieved: low-noise operation, suitable temperature, humidity and light personalized adjustment functions are provided for children's learning and resting scenes; safety protection for children can be achieved: by linking smart door locks and alarm systems, accidents can be prevented when children are left unattended; home device linkage can be achieved: combined with smart TVs, lighting equipment, etc., the device automatically switches to child mode, and provides learning, entertainment, and educational content support to further enhance the user experience.

[0150] In another embodiment of the present application, a device control apparatus is also provided, such as Figure 3 As shown, including:

[0151] A first acquisition module 11 is used to acquire audio data and a depth image of a space where a first device is located;

[0152] A first determining module 12, configured to determine object information of an object in the space according to the audio data and the depth image;

[0153] A second determination module 3, used to determine whether the object in the space meets the mode switching trigger condition according to the object information;

[0154] The first control module 14 is configured to control the first device to switch to a target working mode if the device meets a mode switching condition.

[0155] Optionally, the first determining module includes:

[0156] A sound detection unit, used to perform sound detection on the audio data to obtain a sound detection result;

[0157] A contour recognition unit, used to recognize three-dimensional contour information in the depth image;

[0158] The first determining unit is used to determine the object information of each object in the space according to the sound detection result and the three-dimensional contour information.

[0159] Optionally, the second determining module includes:

[0160] A second determining unit, configured to determine whether the space contains a target object according to the object information;

[0161] a third determining unit, configured to determine, if the space contains a target object, whether the space contains only the target object within a preset time period according to the object information;

[0162] The fourth determining unit is configured to determine whether the object in the space satisfies a mode switching trigger condition if the space only contains the target object within a preset time period.

[0163] Optionally, the method further comprises:

[0164] A recognition module, used to perform scene recognition on the depth image to obtain scene type information corresponding to the space;

[0165] A third determination module, configured to determine an object state of the target object according to the scene type information, the audio data and the depth image;

[0166] The second control module is used to control a second device associated with the object state to switch to a target working mode corresponding to the object state.

[0167] Optionally, the third determining module includes:

[0168] A fifth determining unit, configured to determine that the object state of the target object is an entertainment state if the space type corresponding to the scene type information is an entertainment space;

[0169] Alternatively, a sixth determination unit is configured to determine the object state of the target object according to the audio data and the depth image if the space type corresponding to the scene type information is a non-entertainment space.

[0170] Optionally, the sixth determining unit includes:

[0171] an acquisition subunit, configured to acquire a sound detection result obtained by performing sound detection on the audio data and three-dimensional contour information obtained by performing contour detection on the depth image;

[0172] A determination subunit is used to determine that the object state of the target object is a learning state if it is determined according to the sound detection result that the volume of the sound in the space is within a quiet volume range and the posture of the target object is determined according to the three-dimensional contour information to be a learning posture; or to determine that the object state of the target object is a resting state if it is determined according to the sound detection result that the volume of the sound in the space is within a quiet volume range and the posture of the target object is determined according to the three-dimensional contour information to be a resting posture.

[0173] Optionally, the second control module includes:

[0174] A first acquiring unit, configured to acquire a device set associated with the first device;

[0175] a seventh determining unit, configured to determine a second device corresponding to the object state in the device set;

[0176] A second acquisition unit, configured to acquire a target operating mode corresponding to the object state for each second device;

[0177] A control unit is used to control the second device to switch to the target working mode.

[0178] In another embodiment of the present application, an air conditioning device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0179] Memory, used to store computer programs;

[0180] The processor is used to implement the device control method described in any of the above method embodiments when executing the program stored in the memory.

[0181] In the air-conditioning equipment provided by an embodiment of the present invention, the processor can determine the object information of objects in the space according to the audio data and depth images collected in the space by executing the program stored in the memory, and when it is determined that the first device meets the mode switching condition based on the object information, the first device is controlled to switch to the target working mode, and through multimodal data fusion technology, the object information of the objects in the space can be accurately identified, and when the objects in the space meet the mode switching trigger condition, the working mode of the first device is automatically switched, thereby providing a personalized service mode for the objects in the space, which is convenient for users to use.

[0182] The communication bus 1140 mentioned in the above air conditioning device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0183] The communication interface 1120 is used for communication between the air conditioning device and other devices.

[0184] The memory 1130 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0185] The above-mentioned processor 1110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0186] In another embodiment of the present application, a computer-readable storage medium is provided, on which a program of a device control method is stored. When the program of the device control method is executed by a processor, the steps of the device control method described in any of the aforementioned method embodiments are implemented.

[0187] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0188] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A device control method, characterized in that: include: Acquire audio data and a depth image of a space where the first device is located; determining object information of an object in the space according to the audio data and the depth image; determining, according to the object information, whether the object in the space satisfies a mode switching trigger condition; If it is determined that the object in the space meets the mode switching trigger condition, the first device is controlled to switch to the target working mode.

2. The device control method according to claim 1, characterized in that: Determining object information of an object in the space according to the audio data and the depth image includes: Performing sound detection on the audio data to obtain a sound detection result; identifying three-dimensional contour information in the depth image; The object information of each object in the space is determined according to the sound detection result and the three-dimensional contour information.

3. The device control method according to claim 1, characterized in that: Determining whether the object in the space satisfies a mode switching trigger condition according to the object information includes: Determining whether the space contains a target object according to the object information; If the space contains the target object, determining whether the space contains only the target object within a preset time period according to the object information; If the space only includes the target object within a preset time period, it is determined that the object in the space meets the mode switching trigger condition.

4. The device control method according to claim 1, characterized in that: The method further comprises: Performing scene recognition on the depth image to obtain scene type information corresponding to the space; determining an object state of a target object according to the scene type information, the audio data, and the depth image; A second device associated with the object state is controlled to switch to a target working mode corresponding to the object state.

5. The device control method according to claim 4, characterized in that: Determining the object state of the target object according to the scene type information, the audio data, and the depth image includes: If the space type corresponding to the scene type information is an entertainment space, determining that the object state of the target object is an entertainment state; Alternatively, if the space type corresponding to the scene type information is a non-entertainment space, the object state of the target object is determined according to the audio data and the depth image.

6. The device control method according to claim 5, characterized in that: Determining the object state of the target object according to the audio data and the depth image includes: Acquire a sound detection result obtained by performing sound detection on the audio data and three-dimensional contour information obtained by performing contour detection on the depth image; If it is determined according to the sound detection result that the volume of the sound in the space is within the quiet volume range and the posture of the target object is determined to be a learning posture according to the three-dimensional contour information, the object state of the target object is determined to be a learning state; or, if it is determined according to the sound detection result that the volume of the sound in the space is within the quiet volume range and the posture of the target object is determined to be a resting posture according to the three-dimensional contour information, the object state of the target object is determined to be a resting state.

7. The device control method according to claim 4, characterized in that: Controlling a second device associated with the object state to switch to a target working mode corresponding to the object state includes: Acquire a device set associated with the first device; Determining a second device corresponding to the object state in the device set; Acquire a target operating mode corresponding to the object state for each second device; Control the second device to switch to the target working mode.

8. A device control device, characterized in that: include: A first acquisition module, used to acquire audio data and a depth image of a space where the first device is located; A first determining module, configured to determine object information of an object in the space according to the audio data and the depth image; A second determination module, configured to determine whether an object in the space satisfies a mode switching trigger condition according to the object information; The first control module is used to control the first device to switch to a target working mode if it is determined that the object in the space meets the mode switching trigger condition.

9. An air conditioning device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the device control method described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program of the device control method, and when the program of the device control method is executed by the processor, the steps of the device control method described in any one of claims 1 to 7 are implemented.

Images