Smart home linkage control method, device and equipment and computer readable medium

Abstract

The invention relates to a linkage control method, device and equipment for smart home and a computer readable medium. The method comprises the steps that under the condition that sound information is collected, acoustic features of the sound information are extracted, and the sound information comprises sound generated by at least one of a user, an environment and intelligent home equipment in a target space; comparing the acoustic features with associated features of each library instruction in a database to determine a target library instruction triggered by the sound information; determining a scene linkage strategy corresponding to the target library instruction and at least one target smart home device; and issuing the target library instruction to the target smart home device to control the target smart home device to execute the scene linkage strategy according to the target library instruction. According to the invention, the limitation that the sound control of the traditional smart home equipment can only be based on a specific voice instruction is broken through, and the sound generated by the user behavior, the environment sound and even the sound generated by the home equipment can identify the specific application scene and trigger the linkage instruction of the corresponding scene.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a linkage control method, device, equipment and computer-readable medium for a smart home. Background Art

[0002] With the continuous development of smart home technology, smart products are increasingly used in homes, such as smart curtains, smart switches, smart sockets, smart air purifiers, etc. These smart products can provide users with a more convenient and comfortable life experience. However, there are certain limitations in the smart home control methods currently on the market, especially in the voice control method. The voice control of existing smart homes all needs to rely on specific voice commands, that is, the user needs to clearly state the control command, and the smart home determines the specific control command after voice recognition and semantic recognition. As a result, the voice control method of existing smart homes still has great limitations.

[0003] The existing voice control of smart homes all relies on specific voice commands and has great limitations. No effective solution has been proposed yet. Summary of the invention

[0004] The present application provides a linkage control method, device, equipment and computer-readable medium for a smart home to solve the technical problem that the voice control of existing smart homes needs to rely on specific voice commands and has great limitations.

[0005] According to one aspect of an embodiment of the present application, the present application provides a linkage control method for a smart home, comprising: when sound information is collected, extracting acoustic features of the sound information, wherein the sound information includes sounds generated by at least one of a user, an environment, and a smart home device in a target space; comparing the acoustic features with associated features of each library instruction in a database to determine a target library instruction triggered by the sound information; determining a scene linkage strategy and at least one target smart home device corresponding to the target library instruction; and sending the target library instruction to the target smart home device to control the target smart home device to execute the scene linkage strategy according to the target library instruction.

[0006] Optionally, the method also includes pre-establishing the library instructions in the following manner: collecting multiple sound samples of different scenes through a voice-controlled sensor; dividing the multiple sound samples according to the scenes to obtain multiple sample sets corresponding to different scenes; performing feature extraction on the sound samples in each sample set, and taking the average value of each feature as the standard feature value; splicing multiple standard feature values ​​of multiple features in the same sample set to obtain a standard feature vector for each sample set; determining the scene linkage strategy corresponding to each scene, and generating the corresponding library instructions according to the scene linkage strategy, and associating the library instructions and the standard feature vector under the same scene to determine the standard feature vector as the associated feature of the library instructions.

[0007] Optionally, determining the scene linkage strategy corresponding to each scene includes: obtaining historical behavior data of the user's activities in the target space and historical operation data of multiple smart home devices in the target space; determining multiple scenes of the user's activities in the target space based on the historical behavior data, and determining target time periods corresponding to different scenes; determining the first smart home device with an instruction execution record in each of the target time periods based on the historical operation data; calculating the correlation between the instructions executed by the first smart home device in the target time period and the scene corresponding to the target time period; selecting a second smart home device whose correlation is greater than or equal to a set threshold and a target instruction executed by the second smart home device from the first smart home device; and determining the second smart home device and the target instruction under the same scene as the scene linkage strategy corresponding to the current scene.

[0008] Optionally, generating the corresponding library instruction according to the scene linkage strategy includes: inserting an identification code of the second smart home device that executes the target instruction into each of the target instructions in the same scene to obtain multiple instruction segments; splicing the multiple instruction segments to obtain the library instruction, wherein the identification code is used for the second smart home device to determine the target instruction to be executed according to the identification code when receiving the complete library instruction.

[0009] Optionally, determining the scene linkage strategy corresponding to each scene also includes: acquiring video data of multiple time periods of the target space and historical operation data of multiple smart home devices in the target space in the same time period; after aligning the video data with the historical operation data in time, inputting the target large language model, so as to use the target large language model to analyze and summarize the user's behavior habits according to the video data, and determine the scenes corresponding to different behavior habits, and determine the smart home devices that have an associated relationship with different scenes; sending a linkage strategy generation instruction to the target large language model, so that the target large language model generates the corresponding scene linkage strategy based on the user's behavior habits, the scenes, and the smart home devices that have an associated relationship with different scenes.

[0010] Optionally, generating the corresponding library instructions according to the scene linkage strategy includes: sending a library instruction generation instruction to the target large language model, so that the target large language model analyzes the linkage logic of multiple smart home devices in the same scene based on the scene linkage strategy, and determines the execution instructions and trigger conditions of each smart home device based on the linkage logic, and generates the library instructions corresponding to the scene linkage strategy based on the execution instructions and the trigger conditions.

[0011] Optionally, determining the target library instruction triggered by the sound information also includes: capturing the user's facial expressions and gestures in the target space through a camera, combining with the sound information, and extracting multimodal features; comparing the multimodal features with the multimodal association features of each library instruction in the database to determine the target library instruction triggered by the sound information and the multimodal features.

[0012] According to another aspect of an embodiment of the present application, the present application provides a linkage control device for a smart home, comprising: a collection and extraction module, for extracting the acoustic features of the sound information when the sound information is collected, wherein the sound information includes the sound generated by at least one of the user, the environment and the smart home device in the target space; an instruction comparison module, for comparing the acoustic features with the associated features of each library instruction in the database to determine the target library instruction triggered by the sound information; a determination module, for determining the scene linkage strategy and at least one target smart home device corresponding to the target library instruction; a linkage control module, for sending the target library instruction to the target smart home device to control the target smart home device to execute the scene linkage strategy according to the target library instruction.

[0013] According to another aspect of an embodiment of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the steps of the above method are implemented when the processor executes the computer program.

[0014] According to another aspect of an embodiment of the present application, the present application also provides a computer-readable medium having a non-volatile program code executable by a processor, and the program code enables the processor to execute the above method.

[0015] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0016] The present application provides a linkage control method for a smart home, comprising: extracting acoustic features of the sound information when sound information is collected, wherein the sound information includes the sound generated by at least one of the user, environment, and smart home device in the target space; comparing the acoustic features with the associated features of each library instruction in the database to determine the target library instruction triggered by the sound information; determining the scene linkage strategy and at least one target smart home device corresponding to the target library instruction; sending the target library instruction to the target smart home device to control the target smart home device to execute the scene linkage strategy according to the target library instruction. The present application breaks the limitation that the voice control of traditional smart home devices can only be based on specific voice instructions. The sound generated by user behavior, environmental sound, and even the sound generated by home devices can identify specific application scenarios and trigger linkage instructions for corresponding scenarios, making the linkage control of smart home devices more flexible and more in line with the actual needs of users, greatly improving the user experience, and solving the technical problem that the voice control of existing smart homes needs to rely on specific voice instructions and has great limitations. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 A schematic diagram of a hardware environment of an optional smart home linkage control method provided according to an embodiment of the present application;

[0020] Figure 2 A schematic flow chart of an optional smart home linkage control method provided according to an embodiment of the present application;

[0021] Figure 3 A block diagram of an optional smart home linkage control device provided according to an embodiment of the present application;

[0022] Figure 4 A schematic diagram of an optional electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0024] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0025] In order to solve the problems mentioned in the background technology, according to one aspect of the embodiments of the present application, an embodiment of a linkage control method of a smart home is provided.

[0026] Optionally, in the embodiment of the present application, the above-mentioned smart home linkage control method can be applied to Figure 1 In the hardware environment composed of the terminal 101 and the server 103 shown in FIG. Figure 1 As shown, the server 103 is connected to the terminal 101 via a network, and can be used to provide services for the terminal or a client installed on the terminal. A database 105 can be set on the server or independently of the server to provide data storage services for the server 103. The above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network or a local area network, and the terminal 101 includes but is not limited to smart home devices and PCs, mobile phones, tablet computers, etc. connected to the smart home devices.

[0027] A smart home linkage control method in the embodiment of the present application can be executed by the server 103, or can be executed by the server 103 and the terminal 101 together, such as Figure 2 As shown, the method may include the following steps:

[0028] Step S202, when sound information is collected, extracting acoustic features of the sound information, wherein the sound information includes sound generated by at least one of a user, an environment, and a smart home device in a target space;

[0029] Step S204, comparing the acoustic feature with the associated features of each library instruction in the database to determine the target library instruction triggered by the sound information;

[0030] Step S206, determining a scene linkage strategy and at least one target smart home device corresponding to the target library instruction;

[0031] Step S208: Send the target library instruction to the target smart home device to control the target smart home device to execute the scene linkage strategy according to the target library instruction.

[0032] In an embodiment of the present application, multiple sound sensors can be arranged in the target space (such as each room of the family). These sensors should have high sensitivity and broadband response characteristics, and can accurately capture sound signals of different frequencies and intensities. For example, a sound sensor is installed in each of the four corners of the living room and the center of the ceiling, and sound sensors are installed on both sides of the bedside of the bedroom, on the top of the wardrobe, and above the door to ensure full coverage of the space range and not miss any key sound information. When a sound is generated, the sensor converts the sound signal into an electrical signal and transmits it to the processing unit. The collected sound signal can be pre-processed, and the background noise interference can be removed by adaptive noise elimination technology. For example, in the daytime environment, there is traffic noise, electrical appliance operation noise, etc. The adaptive noise elimination algorithm generates a reverse signal according to the real-time characteristics of the ambient noise, which is subtracted from the original sound signal, thereby highlighting the effective sound information.

[0033] Then, acoustic feature extraction is performed. Linear predictive coding (LPC) technology is used to analyze the sound signal and extract acoustic features including fundamental frequency, formant frequency, and vocal channel filter coefficient. For example, for the user's voice, LPC analysis can be used to obtain a fundamental frequency of about 5-10 milliseconds (the specific value depends on the speaker's gender, age, and speaking speed), and the formant frequency is distributed within a certain range (such as the first formant frequency of a male voice may be between 500-800Hz). The vocal channel filter coefficient reflects the shape and characteristics of the vocal channel. Taking footsteps as an example, it appears as a short-term impact signal in the time domain and may have a specific frequency component distribution in the frequency domain, such as strong energy in the low frequency band (related to the impact frequency of footsteps on the ground). The extracted acoustic features include fundamental frequency, harmonic structure, spectral centroid, zero-crossing rate, etc. These features are combined into a feature vector, such as [fundamental frequency value, harmonic coefficient 1, harmonic coefficient 2, ..., spectral centroid value, zero-crossing rate value]. These acoustic features are combined into a feature vector, which is then used for subsequent matching and comparison with the associated features of the library instructions in the database.

[0034] For example, suppose that in the bedroom, when the user turns over at night, the bed makes a slight shaking sound, and the smart air purifier in the room is running at a low wind speed. After the sound sensor collects the sound information, it is processed by wavelet noise reduction to highlight the bed shaking sound signal. The acoustic features are extracted by LPC technology. The fundamental period of the bed shaking sound may not be obvious (because it is not a periodic sound, but some characteristic parameters can still be extracted), the resonance peak frequency distribution is wide and has more low-frequency components, and the channel filter coefficient shows characteristics related to the ambient sound and the bed structure. These features are combined into feature vectors, such as [fundamental period related parameters (special mark when not obvious), resonance peak frequency eigenvalue, channel filter coefficient eigenvalue, etc.], which are used for subsequent comparison and judgment of the features associated with library instructions.

[0035] The system then compares the extracted acoustic feature vectors with the associated features of each library instruction stored in the database one by one. The associated features in the database are feature models obtained by pre-collecting and analyzing a large number of sound samples in different scenarios. For example, in the associated features of the "bedroom rest scene", for slight breathing sounds in a quiet environment, the spectral centroid may be in a lower frequency range and have a lower zero-crossing rate; while for the shaking sound of the bed when getting up, the fundamental frequency is lower and the spectral energy distribution has its own specific pattern.

[0036] In an embodiment of the present application, a distance measurement algorithm (such as Euclidean distance or cosine similarity) can be used to calculate the similarity between the acoustic feature vector and the associated features of each library instruction. Assuming that the currently collected sound feature vector is [fundamental frequency = 30 Hz, harmonic coefficient 1 = 0.2, ...], the similarity is calculated with the associated feature vector of the bed shaking sound in the "bedroom rest scene" [fundamental frequency = 35 Hz, harmonic coefficient 1 = 0.22, ...]. If the similarity exceeds the set threshold (such as 0.8, which can be adjusted according to actual conditions), it is determined that the sound information may trigger a library instruction related to the "bedroom rest scene".

[0037] After determining the triggered target library instruction, the system searches for the scene linkage strategy corresponding to the library instruction according to the pre-set mapping relationship. Taking the "bedroom rest scene" as an example, its linkage strategy may include executing a series of operations when a sound that matches the rest scene is detected (such as turning off the lights, going to bed, etc.).

[0038] Then determine the target smart home devices involved in the scene linkage strategy. In the "bedroom rest scene", the target smart home devices may include smart lights, smart curtains, smart air conditioners, etc. For example, smart lighting devices are used to adjust the brightness according to scene requirements (such as dimming or turning off the lights when going to sleep), smart curtain devices are responsible for closing or opening curtains at specific times (such as automatically closing curtains when going to bed at night), and smart air conditioners can adjust the bedroom temperature according to the ambient temperature and user preferences (such as maintaining a suitable temperature in sleep mode).

[0039] Finally, the system sends the determined target library instructions to the corresponding target smart home devices through wireless communication methods (such as Wi-Fi, Bluetooth or ZigBee, etc.). For example, for smart lighting devices, the instructions may include information such as light brightness adjustment value (such as dimming to 10% brightness), color mode (such as switching to warm yellow light), etc. For smart curtain devices, the instructions include parameters such as curtain opening and closing direction (such as closing) and opening and closing degree (such as completely closing); for smart air conditioning equipment, the instructions involve temperature settings (such as adjusting to 26°C), operating modes (such as sleep mode), etc.

[0040] After the target smart home device receives the library instruction, the microprocessor inside the device parses the instruction and performs the corresponding operation according to the operation code and parameters in the instruction. For example, the smart lighting device adjusts the light brightness and color according to the instruction by controlling the light driving circuit; the smart curtain device drives the motor to operate and drives the curtain track to open and close the curtain; the smart air conditioning device adjusts the compressor frequency, fan speed, etc. to adjust the temperature and operation mode.

[0041] Assume that at 10 o'clock in the evening, the user is in the bedroom preparing to go to bed. The user turns off the light and goes to bed. The sound sensor collects the sound of turning off the light (a short click) and the slight shaking sound of the bed when going to bed.

[0042] After processing in S202, the acoustic features of the sound of turning off the light (such as a short peak in the high frequency band of the spectrum and a high zero-crossing rate) and the acoustic features of the sound of the bed shaking (such as a fundamental frequency of about 20 Hz and a low spectrum centroid) are extracted and combined into a feature vector.

[0043] In S204, the instructions in the target library are compared with the associated features of the "bedroom rest scene" in the database, and after calculating the similarity, it is found that the trigger condition is met, and it is determined that the target library instruction is related to the "bedroom rest scene".

[0044] For S206, the system searches for the scene linkage strategy and determines that the target smart home devices are smart lights, smart curtains, and smart air conditioners.

[0045] In S208, the system sends a command to the smart light to dim the light brightness to 5% and turn it off (assuming the current brightness is 50%); sends a command to the smart curtain to fully close it; sends a command to the smart air conditioner to adjust the temperature to 25°C and switch to sleep mode (assuming the current temperature is 28°C and the operating mode is normal mode). After receiving the command, the smart light device dims the light brightness by reducing the current until it turns off; the smart curtain device starts the motor and pulls the curtain track to fully close the curtain; the smart air conditioner adjusts the cooling capacity and fan speed to reduce the temperature to 25°C and enters sleep mode to reduce operating noise.

[0046] Through steps S202 to S208, the present application breaks the limitation that the voice control of traditional smart home devices can only be based on specific voice commands. The sounds generated by user behavior, environmental sounds and even the sounds generated by home devices can recognize specific application scenarios and trigger linkage commands of corresponding scenarios, making the linkage control of smart home devices more flexible and more in line with the actual needs of users, greatly improving the user experience, and solving the technical problem that the voice control of existing smart homes needs to rely on specific voice commands and has great limitations.

[0047] In an optional embodiment, the method further comprises pre-establishing the library instruction in the following manner:

[0048] Step 1, collecting multiple sound samples of different scenes through a sound control sensor;

[0049] Step 2, dividing the multiple sound samples according to scenes to obtain multiple sample sets corresponding to different scenes;

[0050] Step 3, extracting features from the sound samples in each sample set, and taking the average value of each feature as the standard feature value;

[0051] Step 4, concatenating multiple standard feature values ​​of multiple features in the same sample set to obtain a standard feature vector for each sample set;

[0052] Step 5, determine the scene linkage strategy corresponding to each scene, and generate the corresponding library instructions according to the scene linkage strategy, and associate the library instructions and the standard feature vector under the same scene to determine the standard feature vector as the association feature of the library instructions.

[0053] In the embodiment of the present application, sound samples can be collected for different scenes, such as the "morning waking scene", which collects the sound of the alarm clock ringing every morning in a week, the friction sound when the user gets up and dresses, the sound of the curtains opening, the footsteps going to the bathroom, etc.; the "dinner cooking scene", which collects the sound of the stove ignition, the sound of the range hood running, the sound of the knife cutting vegetables, the sound of the spatula frying, the sound of the tableware colliding, etc. The collected sound samples are classified according to the scene and stored in different sample sets.

[0054] In an embodiment of the present application, for the sound samples in each sample set, the Mel-Frequency Cepstral Coefficient (MFCC) algorithm can be used for feature extraction. MFCC can simulate the human ear's perception characteristics of sound frequency and effectively extract the key features of the sound. Taking the stove ignition sound sample as an example, calculating its MFCC feature vector may obtain a vector containing 12-20 coefficients, such as [0.2, 0.3, -0.1, ...] (assumption). The average value is calculated for the same feature of all samples in the same sample set to obtain the standard feature value. For example, in the "dinner cooking scene" sample set, the average value is calculated for the first coefficient of the MFCC feature of the stove ignition sound, assuming it is 0.25. The standard feature values ​​of multiple features are spliced ​​into a standard feature vector, such as [0.25, 0.32 (the standard feature value of another feature), ...].

[0055] In the embodiment of the present application, other feature extraction methods can also be used. For example, for time domain features, the duration, start time, end time, energy envelope, etc. of the sound are calculated; for frequency domain features, the spectrum of the sound is obtained by fast Fourier transform (FFT), and then the spectrum centroid, spectrum bandwidth, harmonic structure, etc. are calculated. Taking the sound sample of a TV program as an example, its duration may vary according to the type of program and the length of the broadcast, the spectrum centroid may be concentrated in the low and medium frequency bands (because of the frequency range of the voice and some background music in the TV program), and the spectrum bandwidth reflects the distribution range of the sound frequency.

[0056] In the embodiment of the present application, the scene linkage strategy can be determined according to the common behavior patterns of users in different scenes. Taking the "kitchen cooking scene" as an example, when the sound of stove ignition is detected (judged by acoustic feature matching), the linkage strategy can be to automatically start the range hood (set to high wind mode), turn on the kitchen lights (adjust to bright mode), adjust the smart oven to preheat state (if the user often preheats the oven before cooking), etc.

[0057] In an embodiment of the present application, a corresponding library instruction can be generated according to a determined scene linkage strategy. For example, for the linkage strategy of the above-mentioned "kitchen cooking scene", the generated library instruction may be "1010110010011010" (where different bit segments represent control instructions for different devices, assuming that the first four digits "1010" represent starting the range hood and setting it to a high wind gear, the middle four digits "1100" represent turning on the kitchen lights and adjusting them to a bright mode, and the last four digits "1001" represent adjusting the smart oven to a preheating state. The specific coding rules can be determined according to the system design). This library instruction is then associated with the standard feature vector of the "kitchen cooking scene" (such as the vector containing features such as the stove ignition sound generated previously) and stored in a database.

[0058] By comprehensively collecting sound samples of different scenes and processing them carefully, the feature model (standard feature vector) of each scene can be accurately constructed. The scene linkage strategy determined based on the user behavior pattern and the generated library instructions enable the smart home system to automatically identify the scene and perform corresponding linkage operations based on the actual collected sound. This method greatly improves the automation and intelligence of smart home control, reduces the tediousness of manual operation by users, makes the smart home system more in line with users' daily living habits, and improves the convenience and comfort of user experience.

[0059] In an optional embodiment, determining the scene linkage strategy corresponding to each scene includes:

[0060] Step 1, obtaining historical behavior data of the user's activities in the target space and historical operation data of multiple smart home devices in the target space;

[0061] Step 2, determining multiple scenes of the user's activities in the target space according to the historical behavior data, and determining target time periods corresponding to different scenes;

[0062] Step 3, determining the first smart home device having a command execution record in each of the target time periods according to the historical operation data;

[0063] Step 4, calculating the correlation between the instruction executed by the first smart home device in the target time period and the scene corresponding to the target time period;

[0064] Step 5, selecting a second smart home device whose correlation degree is greater than or equal to a set threshold from the first smart home device and a target instruction executed by the second smart home device;

[0065] Step 6: determine the second smart home device and the target instruction in the same scene as the scene linkage strategy corresponding to the current scene.

[0066] In the embodiment of the present application, the historical behavior data of the user's activities in the target space (such as a family home) can be obtained from the storage module of the smart home system. These data come from the records of various sensors and smart devices, such as smart door locks recording the time and method of the user's entry and exit, human infrared sensors recording the user's activity trajectory and stay time in each room, and smart appliances (such as TVs, refrigerators, washing machines, etc.) recording usage time and operation mode. For example, by analyzing the smart door lock data, it is found that the user usually goes home between 7 and 8 pm from Monday to Friday, which is a common behavior pattern. At the same time, the historical operation data of multiple smart home devices in the target space are obtained, including the opening time, closing time, and operating parameters (such as light brightness, air conditioning temperature, appliance power, etc.) of the device. For example, the smart lighting system records the brightness settings of different time periods every day, and the smart air conditioner records the temperature adjustment history. According to these data, the target time period corresponding to different scenes is determined. For example, according to the behavior pattern that the user usually moves in the living room after returning home from 7-8 pm, the target time period of the "living room leisure scene at night" is determined to be 19:00-20:00 (the time period range can be appropriately expanded according to the actual situation to include possible before and after activity time).

[0067] In an embodiment of the present application, the first smart home device with a record of command execution in each target time period can be found based on historical operation data. In the 19:00-20:00 time period of the "living room leisure scene at night", devices that may have command execution records include smart TVs (commands such as turning on and switching channels), smart lights (brightness adjustment commands), smart speakers (playing music commands), etc. Calculate the correlation between the instructions executed by the first smart home device in the target time period and the scene corresponding to the target time period. Using a probabilistic statistical method, for example, calculate the ratio of the execution frequency of the smart TV's command to turn on and switch channels during the time period to the frequency of occurrence of the "living room leisure scene at night". Assuming that in the past month, the "living room leisure scene at night" appeared 20 times, and the smart TV executed 15 commands to turn on and switch channels during this time period, the correlation is 15 / 20=0.75. Such calculations can be performed for each first smart home device.

[0068] In the embodiment of the present application, a correlation threshold value, such as 0.6 (adjustable according to actual conditions), can be pre-set. A second smart home device with a correlation greater than or equal to the set threshold value and the target instruction executed by it are selected from the first smart home device. In the above example, the smart TV (correlation degree 0.75) and the smart light (assuming the correlation degree is 0.65) meet the conditions, and their target instructions in this scene are to turn on and switch channels and adjust the brightness, respectively. The second smart home device and the target instruction in the same scene are determined as the scene linkage strategy corresponding to the current scene. For the "living room leisure scene at night", the scene linkage strategy is that when the system detects that it is in the target time period of the scene (19:00-20:00) and there is sound information that meets the characteristics of the scene (such as the sound of TV programs, the sound of the user sitting on the sofa, etc.), the smart TV is turned on and the channel switching instructions are executed (automatically switched according to the user's previous preferred channel settings or waiting for further user operations) and the brightness adjustment instructions of the smart light (adjusted to a brightness suitable for leisure activities, such as 70%-80% brightness).

[0069] By accurately screening smart home devices and their instructions that are highly relevant to the scene, the accuracy and practicality of scene linkage are improved. The system can automatically generate reasonable scene linkage strategies based on the common behaviors of users in a specific time period, avoiding the tedious process of manual settings by users, and also reducing the troubles caused to users by unreasonable linkage settings, further improving the intelligence level of the smart home system and user satisfaction.

[0070] In an optional embodiment, generating the corresponding library instruction according to the scenario linkage strategy includes:

[0071] Step 1, inserting the identification code of the second smart home device that executes the target instruction into each of the target instructions in the same scenario to obtain multiple instruction segments;

[0072] Step 2: splice the multiple instruction segments to obtain the library instruction, wherein the identification code is used for the second smart home device to determine the target instruction to be executed according to the identification code when receiving the complete library instruction.

[0073] In the embodiment of the present application, for each target instruction determined in the same scene, taking the "night sleeping scene" as an example, the target instructions include turning off the bedroom lights, drawing the curtains, turning off the smart speaker, adjusting the air conditioner to sleep mode, etc. For the instruction to turn off the bedroom lights, assuming that the identification code of the bedroom lighting device is "001" (the identification code can be set according to the device coding rules of the system and is unique), and the original light turning off instruction is "0101" (the assumed instruction code), the identification code is inserted into the original instruction to obtain "0010101". Similarly, for the instruction to draw the curtains, if the curtain device identification code is "010", the original instruction to draw the curtains is "1010", and it becomes "0101010" after the identification code is inserted; for the instruction to turn off the smart speaker, if the speaker device identification code is "100", the original instruction to turn off the speaker is "0011", and it becomes "1000011" after the identification code is inserted.

[0074] The multiple instruction segments after the identification code is inserted are spliced ​​in a predetermined order to obtain a complete library instruction. In the "night sleeping scene", the bedroom light instruction "0010101", curtain instruction "0101010", speaker instruction "1000011", and air conditioning instruction (assuming that the air conditioning equipment identification code is "011", the original air conditioning adjustment to sleep mode instruction is "1100", and after inserting the identification code is "0111100") are spliced ​​to obtain the library instruction "001010101010101010000110111100". This library instruction contains each target instruction and the corresponding device identification code, so that when the smart home device receives the library instruction, it can accurately determine the target instruction to be executed according to the identification code.

[0075] By inserting device identification codes into target instructions and splicing them into library instructions, precise control of smart home devices is achieved. When the system triggers scene linkage based on sound information and issues library instructions, each smart home device can quickly and accurately identify its own part of the instruction and perform the corresponding operation, avoiding the problem of instruction confusion and erroneous execution. This improves the reliability and accuracy of scene linkage control, ensures that the smart home system can operate in the way the user expects, and improves the stability and user experience of the entire smart home system. At the same time, this coding method facilitates the expansion and management of the system, and can be easily adjusted and updated when adding new smart home devices or modifying instructions.

[0076] In an optional embodiment, determining the scene linkage strategy corresponding to each scene further includes:

[0077] Step 1, obtaining video data of the target space in multiple time periods and historical operation data of multiple smart home devices in the target space in the same time period;

[0078] Step 2, after aligning the video data with the historical operation data according to time, inputting the target large language model, so as to use the target large language model to analyze and summarize the user's behavior habits according to the video data, and determine the scenes corresponding to different behavior habits, and determine the smart home devices associated with different scenes;

[0079] Step 3: Send a linkage strategy generation instruction to the target large language model, so that the target large language model generates the corresponding scene linkage strategy based on the user's behavior habits, the scene, and the smart home devices associated with different scenes.

[0080] In an embodiment of the present application, video data of multiple time periods can be obtained from the surveillance camera system of the target space (such as a home), and the video data should have a high resolution and frame rate to accurately capture the details of the user's behavior. For example, a high-definition camera is installed in the living room to obtain videos of different time periods from 8 am to 10 pm, covering weekdays and weekends. At the same time, the historical operation data of multiple smart home devices in the target space of the same time period is obtained to ensure that the video data and the device operation data can accurately correspond in time. For example, the timestamp of each video frame in the video data of the living room camera is accurately matched and aligned with the time record in the operation data of devices such as smart TVs, smart lights, and smart curtains for subsequent joint analysis.

[0081] Input the aligned video data and historical operation data into the target large language model. The target large language model should have strong semantic understanding and behavior analysis capabilities, such as the GPT series model based on deep learning or other similar advanced models. The model analyzes and summarizes the user's behavior habits based on the character actions (such as walking, sitting down, picking up items, etc.), behavior trajectories (such as the movement path from the living room to the kitchen), object interactions (such as using a remote control to control the TV, opening the refrigerator door to take items, etc.) in the video data, and the device state changes in the device operation data (such as turning on the TV and switching channels, adjusting the brightness of the lights, and changing the opening and closing states of the curtains, etc.). For example, if the user often sits on the sofa in the living room from 7 to 9 pm in the video, switching TV channels with the remote control and adjusting the brightness of the lights, the model can summarize the user's behavior habits in the "leisure and entertainment in the living room at night" scenario, including liking to adjust the light brightness first after entering the living room during a specific time period (7 to 9 pm), and then turning on the entertainment equipment (TV and speakers), and may further adjust the device parameters (such as TV volume, speaker sound effect mode, etc.) according to the viewing content.

[0082] In an embodiment of the present application, a linkage strategy generation instruction can be sent to the target large language model according to the determined scene and associated devices. The model generates a detailed scene linkage strategy based on the user's behavioral habits, scene characteristics, and device associations. For example, for the "living room leisure at night" scene, the linkage strategy is: when the camera detects that the user enters the living room between 7 and 9 p.m. and walks towards the sofa, the smart light is automatically dimmed to 50% brightness; when it is detected that the user picks up the remote control to turn on the TV, the smart speaker automatically plays music that matches the type of program the user often watches (if it is detected that a movie channel is turned on, surround sound movie soundtracks are played); when the TV volume changes, the volume of the smart speaker is automatically adjusted to a level that matches the TV volume to maintain a good audio experience.

[0083] By integrating video data and smart home device operation data and analyzing them using the target large language model, we can gain a deeper understanding of users' behavioral habits and scenario requirements. The scenario linkage strategy generated in this way is more in line with the user's actual life pattern and improves the personalization of the smart home system. For example, in a family dinner scene, the system can automatically create a suitable atmosphere (lighting and music), provide food management suggestions, and optimize the preparation of cooking equipment, which greatly improves the convenience and comfort of users in the smart home environment and enables the smart home system to serve users' daily lives more intelligently.

[0084] In an optional embodiment, generating the corresponding library instruction according to the scenario linkage strategy includes:

[0085] Sending a library instruction generation instruction to the target large language model, so that the target large language model analyzes the linkage logic of multiple smart home devices in the same scene based on the scene linkage strategy, and determines the execution instruction and trigger condition of each smart home device based on the linkage logic, and generates the library instruction corresponding to the scene linkage strategy based on the execution instruction and the trigger condition.

[0086] In an embodiment of the present application, after receiving the scene linkage strategy, a library instruction generation instruction can be sent to the target large language model. The model first conducts an in-depth analysis of the linkage logic of multiple smart home devices in the same scene. For example, in the "weekend family dinner" scenario, the operation sequence and relationship between smart speakers, smart lights, smart refrigerators, smart ovens and other devices are analyzed. Determine that the smart speaker should start playing music after the smart lights are turned on, and the music type should match the dinner atmosphere; the smart refrigerator's food management prompt should be triggered when someone opens the refrigerator door; the preheating operation of the smart oven should be accurately set according to the type of dish selected by the user.

[0087] The target large language model can determine the execution instructions and trigger conditions for each smart home device based on linkage logic. For smart lights, the execution instruction is to adjust the brightness to 80%, and the trigger condition is that family members enter the restaurant at around 12 noon on Saturday or Sunday and the number of people reaches 3 or more; for smart speakers, the execution instruction is to play cheerful background music (such as classical light music), and the trigger condition is that the smart lights are brightened; for smart refrigerators, the execution instruction is that the internal camera records the removal of ingredients and recommends additional ingredients based on historical data, and the trigger condition is detecting that the refrigerator door is open; for smart ovens, the execution instruction is to adjust the preheating temperature and time according to the preset dish type, and the trigger condition is receiving the user's preheating instruction.

[0088] According to the determined execution instructions and trigger conditions, the target large language model generates library instructions corresponding to the scene linkage strategy. A structured encoding method is adopted, for example, a specific encoding bit segment is assigned to each device, the first few bits indicate the device type, the middle few bits indicate the parameters of the execution instruction, and the last few bits indicate the encoding of the trigger condition. Taking smart lighting as an example, assuming that the device type is coded as "001", the instruction parameter for adjusting the brightness to 80% is coded as "01000" (the specific encoding rules can be determined according to the system design), and the trigger condition (family members enter the restaurant around 12 noon on Saturday or Sunday and the number of people reaches 3 or more) is coded as "110101", then the instruction code of the smart light is "00101000110101".

[0089] According to the order of the devices in the scene linkage, the command codes of each device are spliced ​​into a complete library command. In the "weekend family dinner" scene, the command codes of smart lights, smart speakers, smart refrigerators, smart ovens and other devices are spliced ​​together to obtain "0010100011010101001000101101100110100101110010" (assuming a complete library command, the actual code length and content are determined according to the specific device and scene).

[0090] By clearly encoding execution instructions and trigger conditions, the accuracy and executability of library instructions are improved. After receiving the library instructions, each device can quickly parse and execute the corresponding operation, avoiding the ambiguity of instructions and execution errors. At the same time, the structured coding method facilitates the system to manage and maintain library instructions. When the scene linkage strategy changes or the device function is updated, the library instructions can be easily modified and updated, which enhances the flexibility and adaptability of the smart home system and further improves the stability and reliability of the user experience.

[0091] In an optional embodiment, the determining the target library instruction triggered by the sound information further includes:

[0092] Step 1: Capture the user's facial expressions and gestures in the target space through a camera, combine them with sound information, and extract multimodal features;

[0093] Step 2: Compare the multimodal feature with the multimodal associated features of each library instruction in the database to determine the target library instruction triggered by the sound information and the multimodal feature.

[0094] In the embodiment of the present application, a high-definition camera can be installed in the target space (such as each room of a home), and the position of the camera should ensure that the user's facial expressions and gestures can be clearly captured. For example, a rotatable high-definition camera is installed in the center of the ceiling of the living room, and its viewing angle can cover the entire living room area. At the same time, the sound sensor continues to collect sound information to ensure the synchronous collection of sound and visual information.

[0095] When the user moves in the space, the camera captures the user's facial expression changes and gestures in real time. Facial expression recognition uses advanced deep learning algorithms, such as convolutional neural networks (CNN), which can recognize common facial expressions, such as smiles, frowns, surprise, etc. Gesture recognition is also based on deep learning models. For example, an algorithm based on skeleton key point detection is used to recognize user gestures, such as waving, fisting, pointing, etc. Sound sensors collect sounds made by users, such as talking, laughing, coughing, etc., as well as environmental sounds and smart home device sounds.

[0096] For the collected facial expression image data, the features such as the position and motion trajectory of the facial feature points are extracted. For example, the coordinates of the key points of the eyes, eyebrows, mouth and other parts are determined by the facial feature point detection algorithm, and the displacement and change speed of these key points within a certain period of time are calculated to form a facial expression feature vector. For gesture action data, the shape, movement direction, speed and other features of the gesture are extracted, such as constructing a three-dimensional model of the gesture through the skeleton key points, calculating the direction vector and speed of the gesture movement, and obtaining the gesture action feature vector. For sound information, the acoustic features such as the fundamental frequency, the spectrum centroid, the zero crossing rate, etc. are extracted according to the method in the above embodiment to form a sound feature vector.

[0097] The facial expression feature vector, gesture feature vector and sound feature vector are fused. A feature fusion algorithm is used, such as the weighted summation method or the feature concatenation method. Taking the weighted summation method as an example, weights are assigned according to the importance of different modal features to scene recognition. Assume that the weight of facial expression features is 0.4, the weight of gesture features is 0.3, and the weight of sound features is 0.3. Multiply the three feature vectors by the corresponding weights and add them together to obtain the fused multimodal feature vector, such as [facial expression feature vector*0.4+gesture feature vector*0.3+sound feature vector*0.3].

[0098] The system compares the fused multimodal feature vector with the multimodal association features of each library instruction stored in the database. The multimodal association features in the database are feature models obtained by pre-collecting and analyzing a large number of multimodal data in different scenarios. For example, in the "living room entertainment interaction" scenario, when the user smiles, waves and says "play music", the fusion features of his facial expression, gesture and sound information are stored in the database as the multimodal association features of the scenario.

[0099] A similarity measurement algorithm, such as the dynamic time warping (DTW) algorithm, is used to calculate the similarity between the multimodal feature vector and the multimodal associated features of each library instruction. The DTW algorithm can process feature sequences of different lengths and time offsets and is suitable for the comparison of multimodal data. If the similarity exceeds a set threshold (such as 0.85, which can be adjusted according to actual conditions), the target library instruction triggered by the multimodal information is determined. For example, after the similarity between the currently collected user multimodal feature vector and the multimodal associated features of the "living room entertainment interaction" scene is calculated, the threshold condition is met, and the target library instruction is determined to be to start the smart speaker to play music.

[0100] By integrating multimodal features of facial expressions, gestures, and sound information for scene recognition and library command triggering, the accuracy and richness of the smart home system's understanding of user intent are improved. Compared with single sound information recognition, multimodal information can more comprehensively reflect user needs and scene status. This makes the smart home system more intelligent and convenient, enhances the interactive experience between users and smart home devices, reduces misoperations caused by unclear or incomplete voice commands, and further improves the intelligence level of the smart home system and user satisfaction. At the same time, the fusion of multimodal features also improves the robustness of the system in complex environments (such as multiple people talking at the same time, background noise, etc.), and can more stably identify user intent and trigger the correct scene linkage.

[0101] By adopting the technical solution of the present application, linkage control of smart home devices can be performed based on user-specific sounds, environmental sounds, and even the sounds of home devices in multiple scenarios. The following embodiments will be used for specific explanation.

[0102] When the faucet is turned on during the zone time, the voice control recognition module wakes up the lights in the area, and links the smart mop to mop the floor, and links the nearby fan to air dry the floor.

[0103] When the toilet lid is closed, the touch sensor recognizes and turns on the self-circulating air setting; when the toilet is opened and flushed, the voice control system recognizes the area and delays the closing of the self-circulating control setting.

[0104] When the gas stove on the stove is turned on, the voice control system identifies the moment the gas stove switch knob is turned on. After successful recognition, it turns on the range hood and links the sweeper to clean the relevant area.

[0105] When someone at home coughs, the householder makes the preliminary settings by himself, and the voice control module recognizes the information and commands it to the central control. The central control determines and asks whether to turn on air self-circulation, temperature self-adjustment, air purification and sterilization, and scans and identifies the householder's temperature, and reports to the householder by voice whether the body surface temperature is normal. The carbon dioxide detector also synchronously detects the linkage of air indicators and other household smart products.

[0106] This application breaks the limitation that voice control of traditional smart home devices can only be based on specific voice commands. The sounds generated by user behavior, environmental sounds and even sounds generated by home devices can recognize specific application scenarios and trigger linkage commands of corresponding scenarios, making the linkage control of smart home devices more flexible and more in line with the actual needs of users, greatly improving the user experience, and solving the technical problem that the voice control of existing smart homes must rely on specific voice commands and has great limitations.

[0107] According to another aspect of the embodiment of the present application, Figure 3 As shown, a linkage control device for a smart home is provided, comprising:

[0108] A collection and extraction module 301 is used to extract acoustic features of the sound information when sound information is collected, wherein the sound information includes sound generated by at least one of a user, an environment, and a smart home device in a target space;

[0109] An instruction comparison module 303 is used to compare the acoustic feature with the associated features of each library instruction in the database to determine the target library instruction triggered by the sound information;

[0110] A determination module 305 is used to determine a scene linkage strategy and at least one target smart home device corresponding to the target library instruction;

[0111] The linkage control module 307 is used to send the target library instruction to the target smart home device to control the target smart home device to execute the scene linkage strategy according to the target library instruction.

[0112] It should be noted that the collection and extraction module 301 in this embodiment can be used to execute step S202 in the embodiment of the present application, the instruction comparison module 303 in this embodiment can be used to execute step S204 in the embodiment of the present application, the determination module 305 in this embodiment can be used to execute step S206 in the embodiment of the present application, and the linkage control module 307 in this embodiment can be used to execute step S208 in the embodiment of the present application.

[0113] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware.

[0114] Optionally, the linkage control device of the smart home also includes a library instruction establishment module, which is specifically used to: collect multiple sound samples of different scenes through a voice-controlled sensor; divide the multiple sound samples according to the scenes to obtain multiple sample sets corresponding to different scenes; extract features of the sound samples in each sample set, and take the average value of each feature as the standard feature value; splice multiple standard feature values ​​of multiple features in the same sample set to obtain a standard feature vector for each sample set; determine the scene linkage strategy corresponding to each scene, and generate the corresponding library instruction according to the scene linkage strategy, and associate the library instruction and the standard feature vector under the same scene to determine the standard feature vector as the associated feature of the library instruction.

[0115] Optionally, the linkage control device of the smart home also includes a scene linkage strategy determination module, which is specifically used to: obtain historical behavior data of the user's activities in the target space and historical operation data of multiple smart home devices in the target space; determine multiple scenes of the user's activities in the target space according to the historical behavior data, and determine target time periods corresponding to different scenes; determine the first smart home device with an instruction execution record in each of the target time periods according to the historical operation data; calculate the correlation between the instruction executed by the first smart home device in the target time period and the scene corresponding to the target time period; select a second smart home device whose correlation is greater than or equal to a set threshold and a target instruction executed by the second smart home device from the first smart home device; and determine the second smart home device and the target instruction under the same scene as the scene linkage strategy corresponding to the current scene.

[0116] Optionally, the library instruction establishment module is also used to: insert an identification code of the second smart home device that executes the target instruction into each of the target instructions in the same scenario to obtain multiple instruction segments; splice the multiple instruction segments to obtain the library instruction, wherein the identification code is used for the second smart home device to determine the target instruction to be executed according to the identification code when receiving the complete library instruction.

[0117] Optionally, the scene linkage strategy determination module is also used to: obtain video data of multiple time periods in the target space and historical operation data of multiple smart home devices in the target space in the same time period; after aligning the video data with the historical operation data in time, input them into a target large language model, so as to use the target large language model to analyze and summarize the user's behavior habits based on the video data, and determine the scenes corresponding to different behavior habits, and determine the smart home devices that have an associated relationship with different scenes; send a linkage strategy generation instruction to the target large language model, so that the target large language model generates the corresponding scene linkage strategy based on the user's behavior habits, the scenes, and the smart home devices that have an associated relationship with different scenes.

[0118] Optionally, the library instruction establishment module is also used to: send a library instruction generation instruction to the target large language model, so that the target large language model analyzes the linkage logic of multiple smart home devices in the same scene based on the scene linkage strategy, and determines the execution instructions and trigger conditions of each smart home device based on the linkage logic, and generates the library instructions corresponding to the scene linkage strategy based on the execution instructions and the trigger conditions.

[0119] Optionally, the determination module is also used to: capture the user's facial expressions and gestures in the target space through a camera, and extract multimodal features in combination with sound information; compare the multimodal features with the multimodal association features of each library instruction in the database to determine the target library instruction triggered by the sound information and the multimodal features.

[0120] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such as Figure 4 As shown, it includes a memory 401, a processor 403, a communication interface 405 and a communication bus 407. The memory 401 stores a computer program that can be run on the processor 403. The memory 401 and the processor 403 communicate through the communication interface 405 and the communication bus 407. When the processor 403 executes the computer program, the steps of the above method are implemented.

[0121] The memory and processor in the above electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0122] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0123] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0124] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any of the above embodiments.

[0125] Optionally, in an embodiment of the present application, the computer-readable medium is configured to store program codes for the processor to perform the following steps:

[0126] When sound information is collected, extracting acoustic features of the sound information, wherein the sound information includes sound generated by at least one of a user, an environment, and a smart home device in a target space;

[0127] Comparing the acoustic feature with the associated features of each library instruction in the database to determine the target library instruction triggered by the sound information;

[0128] Determine a scene linkage strategy and at least one target smart home device corresponding to the target library instruction;

[0129] The target library instruction is sent to the target smart home device to control the target smart home device to execute the scene linkage strategy according to the target library instruction.

[0130] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0131] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.

[0132] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0133] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0134] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0135] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0136] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0137] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0138] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0139] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0140] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A linkage control method for a smart home, characterized in that: include: When sound information is collected, extracting acoustic features of the sound information, wherein the sound information includes sound generated by at least one of a user, an environment, and a smart home device in a target space; Comparing the acoustic feature with the associated features of each library instruction in the database to determine the target library instruction triggered by the sound information; Determine a scene linkage strategy and at least one target smart home device corresponding to the target library instruction; The target library instruction is sent to the target smart home device to control the target smart home device to execute the scene linkage strategy according to the target library instruction.

2. The method according to claim 1, characterized in that The method further comprises pre-establishing the library instructions in the following manner: Collect multiple sound samples in different scenarios through voice-controlled sensors; Dividing the multiple sound samples according to scenes to obtain multiple sample sets corresponding to different scenes; Extract features from the sound samples in each sample set, and take the average value of each feature as the standard feature value; Concatenate multiple standard feature values ​​of multiple features in the same sample set to obtain a standard feature vector for each sample set; Determine the scene linkage strategy corresponding to each scene, generate the corresponding library instruction according to the scene linkage strategy, and associate the library instruction and the standard feature vector under the same scene to determine the standard feature vector as the association feature of the library instruction.

3. The method according to claim 2, characterized in that Determining the scene linkage strategy corresponding to each scene includes: Acquire historical behavior data of the user's activities in the target space and historical operation data of multiple smart home devices in the target space; Determine multiple scenes of the user's activities in the target space according to the historical behavior data, and determine target time periods corresponding to different scenes; Determine, according to the historical operation data, a first smart home device having a command execution record in each of the target time periods; Calculating a correlation between the instruction executed by the first smart home device in the target time period and the scene corresponding to the target time period; Selecting, from the first smart home devices, a second smart home device whose correlation degree is greater than or equal to a set threshold and a target instruction executed by the second smart home device; The second smart home device and the target instruction in the same scene are determined as the scene linkage strategy corresponding to the current scene.

4. The method according to claim 3, characterized in that Generating the corresponding library instruction according to the scene linkage strategy includes: Inserting the identification code of the second smart home device that executes the target instruction into each of the target instructions in the same scenario to obtain multiple instruction segments; The multiple instruction segments are spliced ​​together to obtain the library instruction, wherein the identification code is used for the second smart home device to determine the target instruction to be executed according to the identification code when receiving the complete library instruction.

5. The method according to claim 2, characterized in that: Determining the scene linkage strategy corresponding to each scene also includes: Acquire video data of the target space in multiple time periods and historical operation data of multiple smart home devices in the target space in the same time period; After aligning the video data with the historical operation data in time, the video data are input into a target large language model, so as to analyze and summarize the user's behavior habits according to the video data using the target large language model, determine the scenes corresponding to different behavior habits, and determine the smart home devices associated with different scenes; A linkage strategy generation instruction is sent to the target large language model, so that the target large language model generates the corresponding scene linkage strategy based on the user's behavior habits, the scenes, and smart home devices associated with different scenes.

6. The method according to claim 5, characterized in that Generating the corresponding library instruction according to the scene linkage strategy includes: Sending a library instruction generation instruction to the target large language model, so that the target large language model analyzes the linkage logic of multiple smart home devices in the same scene based on the scene linkage strategy, and determines the execution instruction and trigger condition of each smart home device based on the linkage logic, and generates the library instruction corresponding to the scene linkage strategy based on the execution instruction and the trigger condition.

7. The method according to any one of claims 1 to 6, characterized in that: The target library instruction for determining the sound information triggering further includes: Capture the user's facial expressions and gestures in the target space through the camera, combine them with sound information, and extract multimodal features; The multimodal feature is compared with the multimodal associated features of each library instruction in the database to determine the target library instruction triggered by the sound information and the multimodal feature.

8. A linkage control device for a smart home, characterized in that: include: A collection and extraction module, configured to extract acoustic features of the sound information when sound information is collected, wherein the sound information includes sound generated by at least one of a user, an environment, and a smart home device in a target space; An instruction comparison module, used to compare the acoustic feature with the associated features of each library instruction in the database to determine the target library instruction triggered by the sound information; A determination module, used to determine a scene linkage strategy and at least one target smart home device corresponding to the target library instruction; The linkage control module is used to send the target library instruction to the target smart home device to control the target smart home device to execute the scene linkage strategy according to the target library instruction.

9. An electronic device, comprising a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate through the communication bus and the communication interface, characterized in that: When the processor executes the computer program, the linkage control method of the smart home described in any one of claims 1 to 7 is implemented.

10. A computer readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the linkage control method of the smart home as described in any one of claims 1 to 7.

Images