Voice control method and device of smart home device, electronic device, medium and product

Abstract

The application discloses a voice control method and device of a smart home device, an electronic device, a medium and a product. The method comprises: collecting real-time multi-modal sensing data when a voice instruction is detected; generating a spatial binding result of a target person according to the space-time relationship between real-time sound source coordinates and real-time human radar position information; performing hierarchical memory analysis according to the spatial binding result and the voice instruction to obtain a structured task object and a long-term memory (LTM) retrieval result; generating an executable instruction queue according to a pre-constructed home knowledge graph, the structured task object and the long-term memory retrieval result; and performing a corresponding smart home device control operation according to the executable instruction queue. The method can realize collaborative and accurate execution of whole-house devices, complete a complete closed loop from multi-source sensing, spatial positioning, intent analysis to device control, and thus improve the accuracy and intelligent level of voice control.

Description

Technical Field

[0001] This application relates to the field of smart home technology, specifically to a voice control method, device, electronic device, readable storage medium, and computer program product for smart home devices. Background Technology

[0002] With the popularization of smart home technology, users' demands for voice control of home devices have evolved from simple command execution to more natural and intelligent interaction methods. In daily life, users often control home devices via voice within the same space. The presence of multiple family members and multiple rooms in a home necessitates that the system accurately identify the specific person issuing the command and their location within a complex acoustic environment. Existing smart home systems typically rely on cloud-based voice recognition and natural language processing capabilities to upload collected voice commands to the cloud for intent parsing and device mapping, before forwarding control commands to the corresponding devices via a gateway. During voice interaction, the system only relies on the voice signals collected by the microphone for intent recognition, failing to accurately associate voice commands with specific human targets in the current space. This results in the system being unable to determine the room and specific user from which the command originated, easily leading to mis-controls across rooms in multi-person home environments and affecting control accuracy. Summary of the Invention

[0003] In view of the above problems, this application provides a voice control method, device, electronic device, readable storage medium and computer program product for smart home devices, which can solve the problem of voice control in multi-person home environments that is prone to cross-room miscontrol.

[0004] Firstly, this application provides a voice control method for a smart home device, including: When a voice command is detected, real-time multimodal perception data is collected; wherein, the real-time multimodal perception data includes the voice command, the real-time sound source coordinates corresponding to the voice command, real-time human body radar position information, real-time environmental status information, and real-time device status information; Based on the spatiotemporal relationship between the real-time sound source coordinates and the real-time human body radar position information, a spatial binding result for the target person is generated; wherein, the spatial binding result includes room identifier, binding status, and target trajectory identifier; Based on the spatial binding results and the voice commands, hierarchical memory analysis is performed to obtain structured task objects and long-term memory retrieval results; An executable instruction queue is generated based on the pre-constructed family knowledge graph, the structured task object, and the long-term memory retrieval results; The corresponding smart home device control operations are executed according to the executable instruction queue.

[0005] In the above technical solution, this method can provide a unified data foundation for the spatiotemporal association between sound sources and human targets through the collection of real-time multimodal perception data; it can also generate spatial binding results through spatiotemporal relationships, accurately locating speech to specific individuals; it can also accurately understand user intentions through hierarchical memory analysis, combining short-term conversational memory and long-term vector memory, thereby improving the personalization of the analysis; it can also decompose tasks, verify and resolve conflicts through a family knowledge graph, thereby generating an ordered instruction queue; finally, by executing device control operations according to the instruction queue, it can achieve coordinated and precise execution of all devices in the house, thus completing a complete closed loop from perception to execution.

[0006] In some implementations, the acquisition of real-time multimodal sensing data includes: Voice commands and the original sound source coordinates of the voice commands are acquired through a microphone array; The original human body radar position information is collected by millimeter-wave radar; wherein, the original human body radar position information includes multiple original radar coordinates and multiple original motion trajectory identifiers; Raw environmental status information is collected through sensors; this raw environmental status information includes temperature and humidity, CO2 concentration, gas usage status, and door and window status. Obtain raw device power information; wherein, the raw device power information includes real-time power information reported by smart home devices and pre-configured device rated power; The original sound source coordinates, the original human body radar position information, the original environmental state information, and the original device power information are preprocessed to obtain real-time multimodal sensing data, which is then stored locally.

[0007] In the above technical solution, the method can acquire multiple modal sensing data from multiple sources, thereby achieving comprehensive acquisition of various sensing information in the smart home environment; at the same time, it can also obtain real-time multimodal sensing data through preprocessing.

[0008] In some implementations, generating a spatial binding result for the target person based on the spatiotemporal relationship between the real-time sound source coordinates and the real-time human body radar position information includes: Based on the spatiotemporal relationship between the real-time sound source coordinates and the real-time human body radar position information, the target person who issued the voice command and the target radar coordinates of the target person are determined. Based on the number of the target personnel, obtain the binding status corresponding to the target personnel, and obtain the target trajectory identifier of the target personnel based on the real-time human body radar location information; Based on the preset room boundary area, target sound source coordinates, and target radar coordinates, the room identifier of the current location of the target person is obtained; wherein, the target sound source coordinates are obtained by converting the real-time sound source coordinates to the home coordinate system; When the binding status is "bound", a spatial binding result is generated based on the target person's profile identifier, the room identifier, the binding status, and the target trajectory identifier; When there are multiple target personnel, the binding status is ambiguous. When the number of the target personnel is 1, the binding status is "bound". When the number of the target personnel is 0, the binding status is unbound.

[0009] In the above technical solution, the method can accurately determine the target person who issued the voice command and the target radar coordinates by the spatiotemporal relationship between the real-time sound source coordinates and the real-time human body radar position information, and determine the room where the target person is currently located by combining the room boundary area; at the same time, it can also accurately distinguish the binding status according to the number of target persons, thereby obtaining accurate spatial binding results.

[0010] In some embodiments, determining the target person issuing the voice command and the target radar coordinates of the target person based on the spatiotemporal relationship between the real-time sound source coordinates and the real-time human body radar position information includes: Based on the real-time human body radar position information, multiple candidate persons and multiple real-time radar coordinates corresponding to the multiple candidate persons are obtained. Based on the preset sensor installation parameters, the real-time sound source coordinates and the multiple real-time radar coordinates are transformed into the home coordinate system to obtain the target sound source coordinates and multiple candidate radar coordinates. Spatial distance calculations are performed on the coordinates of the target sound source and the coordinates of the multiple candidate radars to obtain multiple target spatial distances that correspond one-to-one with the multiple candidate radars. Spatial matching is performed based on the timestamp of the target sound source coordinates, the timestamps of the multiple candidate radar coordinates, and the spatial distances of the multiple targets to obtain the target radar coordinates that match the target sound source coordinates. The candidate personnel corresponding to the target radar coordinates are the target personnel who issued the voice command.

[0011] In the above technical solution, the method can uniformly transform the real-time sound source coordinates and multiple real-time radar coordinates to the home coordinate system, and perform spatial distance calculation and timestamp matching between the target sound source coordinates and multiple candidate radar coordinates, thereby accurately identifying the target person who issued the voice command among multiple candidate personnel.

[0012] In some implementations, the step of performing hierarchical memory analysis based on the spatial binding result and the voice command to obtain structured task objects and long-term memory retrieval results includes: The voice command is subjected to speech recognition to obtain the original voice command content; Using the original voice command content as input, call A large language model is used to extract control information; wherein, the control information includes device entities, action parameters, and preliminary intent semantics; Based on the real-time environmental status information and preset system constraint rules, environmental constraint conditions are generated. Based on the spatial binding results and the environmental constraints, generate the STM session key and LTM retrieval conditions; The STM context is obtained by searching the short-term session memory based on the STM session key, and the long-term vector memory is obtained by searching the long-term vector memory based on the LTM search conditions; wherein, the search window of the STM context is determined according to the instruction type of the voice command. Based on the original voice command content, the spatial binding result, the STM context, and the long-term memory retrieval result, the final intent is parsed to obtain a structured task object; The structured task object includes the room identifier, the STM session key, the device entity, the action parameters, the environmental constraints, and the preliminary intent semantics; and when the binding state is a bound state, the structured task object also includes the personnel profile identifier of the target personnel. When the binding state is unbound, no STM session key or LTM retrieval condition is generated.

[0013] In the above technical solution, the method can comprehensively analyze voice commands by combining environmental constraints and historical information in short-term conversational memory and long-term vector memory, thereby improving the accuracy of intent parsing and obtaining a structured task object that accurately reflects the user's intent.

[0014] In some implementations, generating the STM session key and LTM retrieval conditions based on the spatial binding result and the environmental constraints includes: When the binding status is "bound", an STM session key is generated based on the room identifier, the target trajectory identifier, and the current session identifier; and LTM search conditions are generated based on the room identifier, the personnel profile identifier, the preliminary intent semantics, and the environmental constraints. When the binding state is ambiguous, a room-level STM session key is generated based on the room identifier and the current session identifier; and room-level LTM search conditions are generated based on the room identifier, the preliminary intent semantics, and the environmental constraints.

[0015] In the above technical solution, the method can adaptively determine the granularity of memory retrieval based on the binding status, so as to achieve personalized memory retrieval for specific personnel by combining personnel profile and target trajectory when personnel have been confirmed, or to complete effective memory retrieval based on room identifiers when there is ambiguity in personnel.

[0016] In some implementations, generating an executable instruction queue based on a pre-constructed family knowledge graph, the structured task object, and the long-term memory retrieval results includes: The composite instructions and control requirements are obtained based on the structured task object; wherein, the control requirements include the expected equipment state or the expected environmental effect; The composite instruction is decomposed into multiple initial sub-tasks based on a pre-constructed family knowledge graph and preset task planning rules. The target device information corresponding to the composite instruction is obtained based on the real-time multimodal sensing data; wherein, the target device information includes device capability description information, action mutual exclusion relationship, device rated power, loop capacity and device real-time power; Based on the target device information, conflict checks are performed on the multiple initial subtasks to obtain the check results; Based on the verification results, the priority strategy in the preset task planning rules, and the personnel preference information in the long-term memory retrieval results, the multiple initial sub-tasks are solved to obtain multiple target sub-tasks; wherein, the priority strategy includes safety priority being greater than human comfort, and human comfort priority being greater than equipment energy consumption. An executable instruction queue is generated based on the multiple target sub-tasks; wherein, the executable instruction queue includes room identifier, STM session key, device entity, action parameters, control requirements, instruction execution sequence, instruction execution priority and timeout threshold, and when the binding status is a bound status, the executable instruction queue also includes the personnel profile identifier of the target personnel.

[0017] In the above technical solution, the method can generate a safe, orderly, and executable instruction queue that conforms to personnel preferences by decomposing tasks, performing conflict verification based on target device information, and solving tasks by combining personnel preferences and priority strategies.

[0018] In some embodiments, after performing the corresponding smart home device control operation according to the executable instruction queue, the method further includes: Receive confirmation messages from smart home devices; The system continuously collects the execution status information of the smart home devices; the execution status information includes the device operating status, current environmental status data, and personnel status. Feedback samples are generated based on the execution status information and the control requirements indicated by the voice commands; During system idle periods, obtain the set of feedback samples generated within a preset time period; The edge weights, preset task planning rules, and model parameters of the large language model of the family knowledge graph are optimized based on the feedback sample set. The model parameters of the large language model are encrypted and uploaded to the cloud for federated learning after user authorization. The large language model is deployed on a local edge gateway, which is divided into independent application sandboxes. Third-party applications run within the independent application sandboxes and only have temporary access to the device interface and the real-time multimodal perception data during the current session. The temporary access is revoked after the session ends.

[0019] In the above technical solution, the method can generate feedback samples based on the actual state of the device after execution, and optimize the edge weights, task planning rules and large language model parameters of the family knowledge graph by using the feedback sample set during the system idle period, so as to continuously improve the family knowledge graph, task planning rules and large language model.

[0020] In some implementations, generating feedback samples based on the execution state information and the control requirements indicated by the voice command includes: When it is determined from the execution status information that the control requirements indicated by the voice command have been met, a positive feedback sample is generated based on the STM session key and the room identifier. When it is determined from the execution status information that the control requirements indicated by the voice command have not been met, or when it is detected that the user has entered manual intervention information multiple times in a preset time window, the device control result and the manual intervention information are obtained, and a negative feedback sample is generated based on the device control result and the manual intervention information.

[0021] In the above technical solution, the method can generate positive or negative feedback samples that accurately correspond to the task execution results based on the degree of conformity between the execution status and the expected control requirements, as well as the user's manual intervention, thereby providing a reliable basis for system optimization.

[0022] Secondly, this application provides a voice control device for a smart home device, comprising: The acquisition unit is used to acquire real-time multimodal perception data when a voice command is detected; wherein, the real-time multimodal perception data includes the voice command, the real-time sound source coordinates corresponding to the voice command, real-time human body radar position information, real-time environmental status information, and real-time device status information; A spatial binding unit is used to generate a spatial binding result for a target person based on the spatiotemporal relationship between the real-time sound source coordinates and the real-time human body radar position information; wherein, the spatial binding result includes a room identifier, binding status, and target trajectory identifier; The analysis unit is used to perform hierarchical memory analysis based on the spatial binding result and the voice command to obtain structured task objects and long-term memory retrieval results; The instruction generation unit is used to generate an executable instruction queue based on the pre-constructed family knowledge graph, the structured task object, and the long-term memory retrieval results. An execution unit is used to perform corresponding smart home device control operations according to the executable instruction queue.

[0023] In the above technical solution, the device can provide a unified data foundation for the spatiotemporal association of sound sources and human targets through the collection of real-time multimodal perception data; it can also generate spatial binding results through spatiotemporal relationships, accurately locating speech to specific individuals; it can also accurately understand user intentions through hierarchical memory analysis, combining short-term conversational memory and long-term vector memory, thereby improving the personalization of the analysis; it can also decompose tasks, verify and solve conflicts through a family knowledge graph, thereby generating an ordered instruction queue; finally, by executing device control operations according to the instruction queue, it can achieve coordinated and precise execution of all devices in the house, thus completing a complete closed loop from perception to execution.

[0024] Thirdly, this application provides an electronic device including a memory and a processor, the memory storing a computer program, and the processor running the computer program to cause the electronic device to perform the voice control method of the smart home device described in any one of the first aspects.

[0025] Fourthly, this application provides a readable storage medium storing a computer program, which, when executed by a processor, performs the voice control method for a smart home device as described in any one of the first aspects.

[0026] Fifthly, this application provides a computer program product, which includes a computer program that, when executed by a processor, performs the voice control method for a smart home device as described in the first aspect.

[0027] The beneficial effects of this application are as follows: First, by determining personnel targets and room affiliations through spatiotemporal correlation under a unified coordinate system, and by combining room identifiers, target trajectory identifiers, and session identifiers for contextual management of room isolation, the accuracy of voice control and the utilization efficiency of edge computing power can be improved. Second, by decomposing tasks and verifying constraint conflicts through a family knowledge graph, execution schemes that meet device capabilities, action mutual exclusion, and loop power constraints can be generated, thereby improving task planning and constraint handling capabilities. Third, by combining message delivery confirmation with closed-loop observation of device, environment, and personnel status, instruction delivery and task completion can be distinguished, thereby improving the execution reliability of device control. Finally, by accumulating positive and negative reflection samples based on execution results and updating the knowledge graph edge weights, task planning rules, and local model parameters, the system can break free from dependence on fixed scene templates, thereby improving its adaptive capabilities. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating the voice control method for smart home devices in some embodiments of this application; Figure 2 This is a schematic diagram of the structure of the voice control device of a smart home device in some embodiments of this application; Figure 3 This is a schematic diagram of the structure of an electronic device in some embodiments of this application. Detailed Implementation

[0030] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0032] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more (including two), similarly, "multiple sets" refers to two or more sets (including two sets), and "multiple pieces" refers to two or more pieces (including two pieces), unless otherwise explicitly specified.

[0033] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0034] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0035] In existing smart home systems, users typically need to explicitly specify the device name when issuing a voice control request so that the platform can recognize the device name in the command and send the control command to the corresponding device. However, when users only issue commands without specifying a particular device (such as "turn on the light"), the system cannot determine which device to control.

[0036] To address the aforementioned technical issues, this application provides a voice control method for smart home devices. This method aims to: determine the speaker's spatial location and automatically match voice commands with devices within the space, thereby enabling users to accurately control devices within their space without having to speak the specific device name.

[0037] like Figure 1 As shown, some embodiments of this application provide a voice control method for a smart home device, which includes: S100. When a voice command is detected, real-time multimodal perception data is collected; wherein, the real-time multimodal perception data includes the voice command, the real-time sound source coordinates corresponding to the voice command, the real-time human body radar position information, the real-time environmental status information, and the real-time device status information.

[0038] In this embodiment, voice commands refer to voice control commands issued by the user, which carry the acquisition time.

[0039] In this embodiment, real-time sound source coordinates refer to the uniformly formatted sound source coordinates obtained after preprocessing the local coordinates of the sound source output by the microphone array.

[0040] In this embodiment, real-time human body radar location information refers to location information in a unified format obtained after preprocessing the raw human body radar location information collected by millimeter-wave radar.

[0041] In this embodiment, real-time environmental state information refers to environmental state information in a unified format obtained after preprocessing the raw environmental state information collected by the sensors.

[0042] In this embodiment, real-time device status information refers to device status information in a unified format obtained after preprocessing the original device power information.

[0043] In this embodiment, the voice control method for smart home devices is deployed on a whole-house edge smart gateway. The hardware of this whole-house edge smart gateway includes an edge computing main control chip, a message caching unit, a local storage unit, a multi-protocol communication module, and an encryption unit. The peripheral hardware of the whole-house edge smart gateway includes a distributed microphone pickup array, a millimeter-wave human body sensing radar, environmental sensors, security sensors, smart meters or circuit acquisition units, and various controlled home execution terminals.

[0044] S200. Based on the spatiotemporal relationship between the real-time sound source coordinates and the real-time human body radar position information, generate the spatial binding result of the target person; wherein, the spatial binding result includes room identifier, binding status and target trajectory identifier.

[0045] In this embodiment, the spatiotemporal relationship refers to the spatial distance and temporal correlation between the sound source coordinates and the human body radar coordinates.

[0046] In this embodiment, the target person refers to the person who issues the voice command.

[0047] In this embodiment, the spatial binding result refers to the result of associating the target person with their room, binding status, and target trajectory identifier.

[0048] In this embodiment, the binding status refers to the bound status, ambiguous status, or unbound status determined based on the number of target personnel.

[0049] In this embodiment, the target trajectory identifier refers to the identifier information used to identify the movement trajectory of the target person.

[0050] S300 performs hierarchical memory analysis based on spatial binding results and voice commands to obtain structured task objects and long-term memory retrieval results.

[0051] In this embodiment, hierarchical memory analysis refers to the intent analysis of voice commands by combining short-term conversational memory and long-term vector memory.

[0052] In this embodiment, a structured task object refers to task information organized in a structured form, including room identifiers, STM (Short-Term Memory) session keys, device entities, action parameters, environmental constraints, and preliminary intent semantics.

[0053] In this embodiment, long-term memory retrieval results refer to historical information retrieved from the long-term vector memory database based on LTM (Long-Term Memory) retrieval criteria.

[0054] S400 generates an executable instruction queue based on a pre-built family knowledge graph, structured task objects, and long-term memory retrieval results.

[0055] In this embodiment, the family knowledge graph refers to a knowledge graph constructed around people, space, equipment, environment, constraints, and loop capacity.

[0056] In this embodiment, the executable instruction queue refers to a sequence of instructions that includes information such as device entity, action parameters, control requirements, instruction execution timing, instruction execution priority, and timeout threshold.

[0057] S500 executes corresponding smart home device control operations according to the executable instruction queue.

[0058] In this embodiment, device control operation refers to the control actions performed on smart home devices according to the executable instruction queue.

[0059] In this embodiment, the method adopts an offline-first edge architecture, where perception, memory, planning, execution, observation, and reflection all run locally, so local functions can still continue to operate when the network is disconnected.

[0060] In this embodiment, the method can cache alarms, logs, and incremental model parameters during offline periods, and synchronously encrypt incremental model parameters after connecting to the network. It is important to emphasize that the raw perception data (including raw voice, image, human body, and health data) is stored on the local area network.

[0061] In this embodiment, when the original perception data participates in federated learning with user authorization, it can be uploaded online as encrypted model gradients or parameter increments to participate in global model aggregation. After downloading the updated global base model locally, it can be further fine-tuned. Users can clear their local memory data and reset the agent model with a single click, and the original user data is not stored in the cloud.

[0062] In this embodiment, the whole-house edge smart gateway can divide into independent application sandboxes, so that third-party plugins can only temporarily apply for access permissions to device interfaces, sensor data and audio and video data as needed, and automatically reclaim them after use, while retaining audit logs throughout the process.

[0063] In the above embodiments, this method can provide a unified data foundation for the spatiotemporal association of sound sources and human targets through the collection of real-time multimodal perception data; it can also generate spatial binding results through spatiotemporal relationships, accurately locating speech to specific individuals; it can also accurately understand user intentions through hierarchical memory analysis, combining short-term conversational memory and long-term vector memory, thereby improving the personalization of the analysis; it can also decompose tasks, verify and solve conflicts through a family knowledge graph, thereby generating an ordered instruction queue; finally, by executing device control operations according to the instruction queue, it can achieve coordinated and precise execution of all devices in the house, thus completing a complete closed loop from perception to execution.

[0064] In some embodiments, step S100 may include: S110: Acquire voice commands and the original sound source coordinates of the voice commands through a microphone array.

[0065] In this embodiment, the microphone array refers to a distributed microphone pickup array.

[0066] In this embodiment, voice commands refer to voice control commands issued by the user, which carry the acquisition time.

[0067] In this embodiment, the original sound source coordinates refer to the local sound source coordinates P_s_local=(X_s, Y_s, Z_s) output by the microphone array; Where X_s is the X-axis coordinate of the sound source in the local coordinate system of the microphone array; Y_s is the Y-axis coordinate of the sound source in the local coordinate system of the microphone array; Z_s is the Z-axis coordinate (height direction) of the sound source in the local coordinate system of the microphone array.

[0068] S120. Collect raw human body radar position information through millimeter-wave radar; wherein, the raw human body radar position information includes multiple raw radar coordinates and multiple raw motion trajectory identifiers.

[0069] In this embodiment, millimeter-wave radar refers to millimeter-wave human body perception radar.

[0070] In this embodiment, the original human body radar position information refers to the human body position information collected by millimeter-wave radar, including multiple original radar coordinates and multiple original motion trajectory markers, as well as the human body target's posture information.

[0071] In this embodiment, the original radar coordinates refer to the local coordinates of the human target, P_hi_local=(X_hi, Y_hi, Z_hi); Where X_hi is the X-axis coordinate of the human target in the local coordinate system of the millimeter-wave radar; Y_hi is the Y-axis coordinate of the human target in the local coordinate system of the millimeter-wave radar; Z_hi is the Z-axis coordinate (height direction) of the human target in the local coordinate system of the millimeter-wave radar. i represents the human goal of the i-th individual.

[0072] In this embodiment, the original motion trajectory identifier refers to the identifier information used to identify the motion trajectory of the human target.

[0073] S130. Collect raw environmental status information through sensors; the raw environmental status information includes temperature and humidity, CO2 concentration, gas usage status and door and window status.

[0074] In this embodiment, the sensors refer to environmental sensors and security sensors. The environmental sensors are used to collect temperature, humidity and CO2 concentration, while the security sensors are used to collect gas usage status and door and window status.

[0075] In this embodiment, the original environmental state information refers to the environmental state information collected by the sensor, including temperature and humidity, CO2 concentration, gas usage status, and door and window status.

[0076] S140. Obtain raw device power information; wherein, raw device power information includes real-time power information reported by smart home devices and pre-configured device rated power.

[0077] In this embodiment, real-time power information refers to the power information reported in real time by smart home devices. Real-time power is obtained from smart meters, circuit acquisition units, or device status reports.

[0078] In this embodiment, the rated power of the device refers to the pre-configured rated power of the device, which is provided by the device model or device file.

[0079] In this embodiment, the circuit capacity (the capacity that the power distribution circuit is allowed to carry) is provided by the power distribution circuit configuration.

[0080] S150. The original sound source coordinates, original human body radar position information, original environmental state information and original equipment power information are preprocessed to obtain real-time multimodal sensing data and stored locally.

[0081] In this embodiment, preprocessing refers to the noise reduction, spatiotemporal alignment, and structuring of the raw sensing data, and the unified recording of data source identifier, acquisition time, object identifier, quantity, unit, and quality status.

[0082] In this embodiment, weather and time-of-use electricity prices are optional external data accesses. When external data is unavailable, the most recent local cached value is used, or it is not used as a prerequisite for basic control.

[0083] In this embodiment, raw sensitive data such as voice, coordinates, and trajectory are only used locally for necessary spatiotemporal correlation. Data written into long-term memory, reflective samples, or data that allows external synchronization are subject to hierarchical desensitization or de-identification processing.

[0084] In the above embodiments, the method can acquire multiple modal sensing data from multiple sources, thereby achieving comprehensive acquisition of various sensing information in the smart home environment; at the same time, it can also obtain real-time multimodal sensing data through preprocessing.

[0085] In some embodiments, step S200 may include: S210. Based on the spatiotemporal relationship between the real-time sound source coordinates and the real-time human body radar position information, determine the target person who issued the voice command and the target radar coordinates of the target person.

[0086] In this embodiment, the target radar coordinates refer to the real-time radar coordinates that match the target personnel.

[0087] For example, this method can read radar snapshots of human targets within the time window corresponding to the moment of voice event acquisition in order to obtain real-time human radar location information.

[0088] In order to accurately identify the target person to issue the voice command among multiple candidates, step S210 may further include: S211. Obtain multiple candidate personnel and multiple real-time radar coordinates corresponding to each candidate personnel based on real-time human body radar position information.

[0089] In this embodiment, the candidate refers to the human target determined based on real-time human body radar location information, who is a possible person to issue voice commands.

[0090] In this embodiment, real-time radar coordinates refer to standardized radar coordinates in a unified format obtained after preprocessing the original radar coordinates.

[0091] In this embodiment, the method can combine time proximity and room boundary affiliation to screen candidates.

[0092] S212. Based on the preset sensor installation parameters, the real-time sound source coordinates and multiple real-time radar coordinates are transformed into the home coordinate system to obtain the target sound source coordinates and multiple candidate radar coordinates.

[0093] In this embodiment, the family coordinate system refers to a three-dimensional coordinate system established with a preset family space origin and configured according to the boundary areas of each room according to the apartment layout.

[0094] In this embodiment, the sensor installation parameters refer to the transformation parameters of each sensor's local coordinate system relative to the home coordinate system, which are obtained by recording the installation position and orientation of the microphone array and millimeter-wave radar during device installation or initial network configuration.

[0095] In this embodiment, the above conversion is performed according to P_home=R_sensor*P_local+T_sensor; Where P_home is the transformed coordinate, that is, the position of the target point in the home coordinate system (a three-dimensional coordinate system with the origin of the home space as the reference and the room boundaries configured according to the apartment layout); P_local refers to the local coordinates directly measured by the sensor, such as the local coordinates of the sound source output by the microphone array (P_s_local) and the local coordinates of the human target output by the millimeter-wave radar (P_hi_local). R_sensor is a rotation matrix, determined by the sensor's mounting orientation (how much the sensor coordinate system has rotated relative to the home coordinate system). T_sensor is a translation vector, determined by the sensor's installation location (where the sensor is installed in the home space).

[0096] In this embodiment, the target sound source coordinates refer to the sound source coordinates (X_s', Y_s', Z_s') obtained by transforming the real-time sound source coordinates to the home coordinate system.

[0097] In this embodiment, candidate radar coordinates refer to the radar coordinates (X_hi', Y_hi', Z_hi') obtained by transforming multiple real-time radar coordinates into the home coordinate system.

[0098] S213. Calculate the spatial distance between the target sound source coordinates and the coordinates of multiple candidate radars to obtain the spatial distances of multiple targets that correspond one-to-one with the multiple candidate radars.

[0099] In this embodiment, the formula for calculating spatial distance is: d_i=√[(X_s' X_hi') 2 +(Y_s' Y_hi') 2 +(Z_s' Z_hi') 2 ]; Where d_i is the spatial distance between the coordinates of the target sound source and the coordinates of the i-th candidate radar; X_s' is the X-axis coordinate of the target sound source in the home coordinate system; Y_s' is the Y-axis coordinate of the target sound source in the home coordinate system; Z_s' is the Z-axis coordinate (height direction) of the target sound source in the home coordinate system; X_hi' is the X-axis coordinate of the i-th candidate radar coordinate in the family coordinate system; Y_hi' is the Y-axis coordinate of the i-th candidate radar coordinate in the family coordinate system; Z_hi' is the Z-axis coordinate (height direction) of the i-th candidate radar coordinate in the family coordinate system; i represents the i-th candidate.

[0100] In this embodiment, the target spatial distance refers to the spatial distance calculated between the target sound source coordinates and the coordinates of each candidate radar.

[0101] S214. Based on the timestamp of the target sound source coordinates, the timestamps of multiple candidate radar coordinates, and the spatial distances of multiple targets, spatial matching is performed to obtain the target radar coordinates that match the target sound source coordinates. Among them, the candidate personnel corresponding to the target radar coordinates are the target personnel who issued the voice commands.

[0102] In this embodiment, spatial matching refers to matching based on temporal proximity, according to the timestamp of the target sound source coordinates, the timestamps of multiple candidate radar coordinates, and the spatial distances between multiple targets.

[0103] S220. Based on the number of target personnel, obtain the binding status of the target personnel and obtain the target trajectory identifier of the target personnel based on the real-time human body radar location information.

[0104] In this embodiment, when there are multiple target personnel, the binding status is ambiguous. When the number of target personnel is 1, the binding status is "bound". When the number of target personnel is 0, the binding status is unbound.

[0105] S230. Based on the preset room boundary area, target sound source coordinates and target radar coordinates, obtain the room identifier of the target person's current location; wherein, the target sound source coordinates are obtained by converting the real-time sound source coordinates to the home coordinate system.

[0106] In this embodiment, the room boundary area refers to the boundary area of ​​each room determined according to the apartment layout.

[0107] In this embodiment, room identification refers to the identification information used to identify the room where the target person is currently located.

[0108] S240. When the binding status is "bound", generate a spatial binding result based on the target person's profile identifier, room identifier, binding status, and target trajectory identifier.

[0109] In this embodiment, the personnel profile identifier refers to the identification information used to identify the personnel profile of the target personnel. The personnel profile includes identity, permission level and preference information.

[0110] In the above embodiments, the method can accurately determine the target person issuing the voice command and their target radar coordinates by the spatiotemporal relationship between the real-time sound source coordinates and the real-time human body radar position information, and determine the room where the target person is currently located by combining the room boundary area; at the same time, it can also accurately distinguish the binding status based on the number of target persons, thereby obtaining accurate spatial binding results.

[0111] In some embodiments, step S300 may include: S310. Perform speech recognition on the voice command to obtain the original voice command content.

[0112] In this embodiment, speech recognition refers to the recognition process that converts speech commands into the original speech command content.

[0113] In this embodiment, the original voice command content refers to the command text content obtained after voice recognition of the voice command.

[0114] S320: Using the original voice command content as input, call the large language model to extract control information; among which, the control information includes device entities, action parameters and preliminary intention semantics.

[0115] In this embodiment, the large language model refers to a lightweight large language model deployed on a local edge gateway or on a local or cloud server.

[0116] In this embodiment, control information refers to the device entity, action parameters, and preliminary intent semantics extracted from the original voice command content.

[0117] In this embodiment, the device entity refers to the smart home device indicated by the voice command.

[0118] In this embodiment, the action parameters refer to the actions and parameters performed on the device entity.

[0119] In this embodiment, preliminary intent semantics refers to the preliminary intent meaning parsed from the voice command.

[0120] S330. Generate environmental constraints based on real-time environmental status information and preset system constraint rules.

[0121] In this embodiment, system constraint rules refer to preset constraint rules related to system operation.

[0122] In this embodiment, environmental constraints refer to environmental constraint information generated based on real-time environmental state information and preset system constraint rules.

[0123] S340. Based on the spatial binding results and environmental constraints, generate the STM session key and LTM retrieval conditions.

[0124] As an optional implementation, when the binding state is unbound, STM session keys and LTM retrieval conditions are not generated, and instead general device semantic parsing or triggering an acknowledgment request is used.

[0125] In this embodiment, the STM session key refers to the key information used to retrieve short-term session memory and select the corresponding context cache space.

[0126] In this embodiment, LTM retrieval conditions refer to the conditional information used for retrieval in the long-term vector memory.

[0127] To achieve adaptive memory retrieval, step S340 may further include: S341. When the binding status is "bound", generate an STM session key based on the room identifier, target trajectory identifier, and current session identifier; and generate LTM search conditions based on the room identifier, personnel profile identifier, preliminary intent semantics, and environmental constraints.

[0128] In this embodiment, the current session identifier refers to the identifier information used to identify the current voice control session.

[0129] In this embodiment, the STM session key is used to retrieve the short-term session memory of the corresponding room.

[0130] S342. When the binding status is ambiguous, generate a room-level STM session key based on the room identifier and the current session identifier; and generate room-level LTM search conditions based on the room identifier, preliminary intent semantics, and environmental constraints.

[0131] In this embodiment, when the room has been confirmed but the personnel are ambiguous, only the short-term context of the room is retrieved, and the specific personnel profile is not used.

[0132] In this embodiment, the room-level STM session key refers to the STM session key generated solely using the room identifier and the current session identifier, which is used to retrieve the room-level short-term session memory.

[0133] In this embodiment, room-level LTM search criteria refer to LTM search criteria generated based on room identifiers, preliminary intent semantics, and environmental constraints, which do not depend on specific personnel profiles.

[0134] S350. Retrieve the STM context in the short-term session memory based on the STM session key, and retrieve the long-term vector memory based on the LTM retrieval conditions to obtain the long-term memory retrieval results.

[0135] In this embodiment, the short-term session memory bank refers to a short-term session memory bank that uses a single room as an independent isolation unit. The temporary cache is automatically archived and cleared after the session ends or the room is determined to be empty.

[0136] In this embodiment, the STM context refers to the short-term session context retrieved from the short-term session memory based on the STM session key.

[0137] In this embodiment, the long-term vector memory refers to a locally deployed lightweight vector database that uses similarity retrieval to match historical habit data.

[0138] In this embodiment, the long-term vector memory stores three types of structured vectors, including personnel profile vectors, historical interaction vectors, and environmental device vectors.

[0139] In this embodiment, the retrieval window of the STM context is dynamically adjusted according to the command type of the voice command: when the voice command is a single action command, the retrieval window only covers the previous round of conversation; when the voice command contains pronouns or referential statements, the retrieval window covers the recent rounds of conversation within the same room; when the voice command is a cross-room global command, the retrieval window covers the global conversation window.

[0140] S360 performs final intent parsing based on the original voice command content, spatial binding results, STM context, and long-term memory retrieval results to obtain a structured task object.

[0141] In this embodiment, final intent parsing refers to intent parsing that combines the original voice command content, spatial binding results, STM context, and long-term memory retrieval results.

[0142] In this embodiment, the structured task object is in JSON format; when the room is not confirmed, general device semantic parsing or a confirmation request is used.

[0143] In this embodiment, the structured task object includes room identifier, STM session key, device entity, action parameters, environmental constraints, and preliminary intent semantics; and when the binding state is "bound", the structured task object also includes the target person's profile identifier.

[0144] In the above embodiments, the method can comprehensively analyze voice commands by combining environmental constraints and historical information in short-term conversational memory and long-term vector memory, thereby improving the accuracy of intent parsing and obtaining a structured task object that accurately reflects the user's intent.

[0145] In some embodiments, step S400 may include: S410. Obtain composite instructions and control requirements based on the structured task object; wherein, the control requirements include the expected equipment status or the expected environmental effect.

[0146] In this embodiment, composite instructions refer to instruction information obtained from structured task objects.

[0147] In this embodiment, control requirements refer to the expected control effect on the equipment or environment, including the expected equipment state or the expected environmental effect.

[0148] In this embodiment, the expected device state refers to the state that the device should reach after execution.

[0149] In this embodiment, the expected environmental effect refers to the effect that the environment should achieve.

[0150] S420. Based on the pre-constructed family knowledge graph and preset task planning rules, the composite instruction is decomposed into multiple initial sub-tasks.

[0151] In this embodiment, the family knowledge graph refers to a knowledge graph constructed with people, space, equipment, environment, constraints and loop capacity as the core. The graph nodes are labeled with object attributes, and the edge relationships are labeled with mutual exclusion of actions, energy consumption cost, comfort weight and safety priority.

[0152] In this embodiment, the task planning rule refers to the preset planning rule used for task decomposition, conflict verification, and task solving.

[0153] In this embodiment, task decomposition refers to the process of splitting a composite instruction into a queue of serial / parallel subtasks.

[0154] In this embodiment, the initial subtask refers to the subtask obtained after decomposing the composite instruction into a task.

[0155] S430. Obtain target device information corresponding to the composite command based on real-time multimodal sensing data; wherein, the target device information includes device capability description information, action mutual exclusion relationship, device rated power, loop capacity and device real-time power.

[0156] In this embodiment, the target device information refers to the real-time device information corresponding to the composite instruction, including device capability description information, action mutual exclusion relationship, device rated power, loop capacity and device real-time power.

[0157] In this embodiment, the device capability description information refers to the description information of the control capabilities that the device possesses.

[0158] In this embodiment, the mutual exclusion relationship of actions refers to the mutually exclusive relationship between the actions of the devices.

[0159] In this embodiment, circuit capacity refers to the capacity that the power distribution circuit is allowed to carry.

[0160] In this embodiment, the real-time power of the device refers to the power reported by the device in real time.

[0161] S440. Perform conflict verification on multiple initial subtasks based on the target device information to obtain the verification result.

[0162] In this embodiment, conflict verification refers to the handling of conflicts between the initial subtasks based on device capabilities, mutual exclusion of actions, device rated power, loop capacity, and real-time power detection.

[0163] In this embodiment, the verification result refers to the result obtained by performing conflict verification on multiple initial subtasks.

[0164] S450. Based on the verification results, the priority strategy in the preset task planning rules, and the personnel preference information in the long-term memory retrieval results, multiple initial sub-tasks are solved to obtain multiple target sub-tasks; among them, the priority strategy includes safety priority being greater than human comfort, and human comfort priority being greater than equipment energy consumption.

[0165] In this embodiment, the priority strategy refers to the priority rules used when solving the task, with safety priority being higher than human comfort, and human comfort priority being higher than device energy consumption.

[0166] In this embodiment, user preference information refers to user preference information in the long-term memory retrieval results.

[0167] In this embodiment, task solving refers to solving the initial subtask based on the verification results, priority strategy, and personnel preference information to obtain the processing of the target subtask.

[0168] In this embodiment, the target subtask refers to the subtask obtained after solving multiple initial subtasks.

[0169] S460. Generate an executable instruction queue based on multiple target sub-tasks; wherein, the executable instruction queue includes room identifier, STM session key, device entity, action parameters, control requirements, instruction execution sequence, instruction execution priority and timeout threshold, and when the binding status is already bound, the executable instruction queue also includes the personnel profile identifier of the target personnel.

[0170] In this embodiment, the instruction execution timing refers to the execution order of each instruction in the executable instruction queue.

[0171] In this embodiment, instruction execution priority refers to the execution priority of each instruction in the executable instruction queue.

[0172] In this embodiment, the timeout threshold refers to the timeout threshold for instruction execution.

[0173] In the above embodiments, the method can generate a safe, orderly, and executable instruction queue that conforms to personnel preferences by decomposing tasks, performing conflict checks based on target device information, and solving tasks by combining personnel preferences and priority strategies.

[0174] In some embodiments, after step S500, the method may further include: S610: Receive confirmation messages from smart home devices.

[0175] In this embodiment, the method can convert the executable instruction queue into protocol messages such as Matter, Zigbee, and Wi-Fi through the gateway multi-protocol adaptation layer and then send them out.

[0176] In this embodiment, the message confirmation information refers to the ACK receipt returned by the device, which is only used to confirm that the message has been delivered and is not used as a basis for determining the completion of the action.

[0177] S620 continuously collects execution status information of smart home devices; among which, execution status information includes device operating status, current environmental status data, and personnel status.

[0178] In this embodiment, the execution status information refers to the continuously collected execution status information of smart home devices, including device operating status, current environmental status data, and personnel status.

[0179] In this embodiment, the method can continuously collect the equipment operating status, environmental data and personnel status after execution through closed-loop observation to determine whether the task has achieved the goal based on the expected equipment status or expected environmental effect; wherein, the execution status information carries the associated STM session key and room identifier.

[0180] S630: Generate feedback samples based on the execution status information and the control requirements indicated by the voice commands.

[0181] In this embodiment, feedback samples refer to sample information generated based on execution status information and control requirements indicated by voice commands.

[0182] In this embodiment, the method can combine device ACK, device status, environmental changes and personnel status to determine whether the task has been achieved.

[0183] In this embodiment, the execution failure result refers to the result information generated when a timeout occurs, no message confirmation information is received, or the observation result fails to meet the standard.

[0184] To provide a reliable basis for system optimization, step S630 may also include: S631. When it is determined from the execution status information that the control requirements indicated by the voice command have been met, a positive feedback sample is generated based on the STM session key and the room identifier.

[0185] In this embodiment, positive feedback samples refer to feedback samples generated when the control requirements indicated by the voice command are met.

[0186] In this embodiment, the method can accumulate positive feedback samples when the task is completed, which can serve as candidate content for the corresponding long-term memory.

[0187] S632. When it is determined from the execution status information that the control requirements indicated by the voice command have not been met, or when it is detected that the user has entered manual intervention information multiple times in a preset time window, the device control results and manual intervention information are obtained, and negative feedback samples are generated based on the device control results and manual intervention information.

[0188] In this embodiment, manual intervention information refers to intervention information where the user manually overturns the system's actions.

[0189] In this embodiment, negative feedback samples refer to feedback samples generated when it is determined that the control requirements indicated by the voice command have not been met, or when it is detected that the user has repeatedly entered manual intervention information within a preset time window. These negative feedback samples are used for strategy review and are not directly retrieved as long-term memories of user preferences.

[0190] In this embodiment, the device control result refers to the control result obtained after performing the device control operation.

[0191] S640. During the system's idle period, obtain the set of feedback samples generated within a preset time period.

[0192] In this embodiment, the system idle period refers to the period when the system load is low and can be used for batch processing.

[0193] In this embodiment, the feedback sample set refers to the set of feedback samples generated within a preset time period.

[0194] For example, this method can periodically take stock of positive and negative feedback samples in batches during system idle periods.

[0195] S650. Optimize the edge weights of the family knowledge graph, the preset task planning rules, and the model parameters of the large language model based on the feedback sample set.

[0196] In this embodiment, edge weight refers to the weight of edge relationships in the family knowledge graph.

[0197] In this embodiment, the model parameters refer to the parameters of the large language model.

[0198] In this embodiment, offline fine-tuning refers to the fine-tuning process of adjusting the edge weights of the knowledge graph, task planning rules, and local model parameters (such as LoRA).

[0199] In this embodiment, the model parameters of the large language model are encrypted and uploaded to the cloud for federated learning after user authorization. Specifically, after user authorization, the local edge gateway encrypts and uploads the incremental model parameters obtained after optimization of the large language model using feedback sample sets to the cloud for global model aggregation. The local edge gateway downloads the updated global base model from the cloud and performs secondary personalized fine-tuning locally to adapt to users' personalized habits while retaining general capabilities. Voice commands, sound source coordinates, and human radar location information are always retained within the local area network and are not transmitted to the cloud.

[0200] In this embodiment, the large language model is deployed on the local edge gateway, which is divided into independent application sandboxes. Third-party applications run in the independent application sandboxes and only have temporary access to the device interface and real-time multimodal perception data during the current session. The temporary access is automatically revoked after the session ends.

[0201] In the above embodiments, the method can generate feedback samples based on the actual state after the device is executed, and optimize the edge weights, task planning rules and large language model parameters of the family knowledge graph by using the feedback sample set during the system idle period, so that the family knowledge graph, task planning rules and large language model can be continuously improved.

[0202] The technical solution of this application will be illustrated by several specific application scenarios below.

[0203] The first example uses a home living environment as an application scenario. It illustrates, in detail, the complete implementation process of this application, from multi-source perception, spatial binding, intent parsing, knowledge graph planning, device execution, to autonomous reflection and iteration, using a complex voice command issued by a user in the living room area. The complete implementation process is as follows: (1) Perception, Binding and Retrieval After a voice event is triggered, the microphone array acquires the local coordinates of the sound source P_s_local=(X_s, Y_s, Z_s), and reads the local coordinates of the human target P_hi_local=(X_hi, Y_hi, Z_hi) output by the millimeter-wave radar within the corresponding time window.

[0204] The system converts both to a unified home coordinate system, combines spatial distance, temporal proximity, and living room / room boundaries to filter candidate targets, obtains bound human targets and living room / room identifiers, and generates associated STM session keys.

[0205] The system extracts preliminary intent semantics such as watching a movie, cleaning, boiling water, and closing windows from the speech content through a local edge big language model; it uses the living room room identifier, the confirmed person's profile identifier, the preliminary intent semantics, and environmental constraints to form LTM retrieval conditions, retrieves the short-term conversation context of the living room area, and retrieves the person's relevant historical habit data from the long-term vector memory.

[0206] (2) Final analysis and planning The system integrates voice content, spatial binding results, STM context, and long-term memory retrieval results to complete the final intent parsing, generating a structured task object containing personnel targets, living room room identifiers, STM session keys, device entities, action parameters, and environmental constraints, and decomposes it into four sub-tasks: a. Living room movie viewing mode (dimming, curtains closed); b. Robotic vacuum cleaner cleaning the living room; c. Electric kettle boiling water; d. Balcony windows closed.

[0207] The knowledge graph verification determines that there is no device mutual exclusion among the four tasks. Sweeping and boiling water can be executed in parallel, and the movie viewing mode and closing the window are arranged in the order required by the instructions, generating an executable instruction queue containing timing and priority.

[0208] (3) Execution and Observation The gateway converts the instruction queue into device protocol messages and sends them to each device; the ACK returned by the device is only used to confirm that the message has been delivered and is not used as a basis for determining the completion of the action.

[0209] The system continuously collects data on ambient brightness, equipment operating status, and door and window status after execution, and compares these data with the expected results to determine the achievement of each task.

[0210] (4) Memory sedimentation After the target is met, the task and action sequence of this round, which has been de-identified locally, will be stored as long-term positive memory candidates according to room identifier, personnel profile identifier at the time of confirmation, preliminary intent semantics and environmental constraints; the associated STM session key is only used for current session context management, providing historical reference for intent parsing and task planning of subsequent similar instructions.

[0211] The second example uses a home living environment as an application scenario, taking a user issuing a voice command with mutually exclusive device relationships as an example to illustrate the implementation process of this application, from conflict identification and task solving to closed-loop observation. The complete implementation process is as follows: (1) The user issues the instruction: "Open the windows for ventilation and turn on the air conditioner to cool to 26°C at the same time".

[0212] (2) Conflict identification: The knowledge graph identifies the two as strongly mutually exclusive actions (opening the window leads to the loss of cold air).

[0213] (3) Task solving: The final structured task object includes the living room room identifier, window opening and air conditioning cooling actions and environmental constraints; retrieve indoor CO2, outdoor temperature and humidity, daily electricity price or its most recent local cache value, and available user preferences retrieved by LTM, and generate the execution sequence according to the weight strategy of safety priority, human comfort and energy saving: after a short window opening to replace the air, the window is automatically closed and the air conditioning is started again.

[0214] (4) Closed-loop observation: After the equipment ACK confirmation message is delivered, the status of doors and windows, the operation status of air conditioning, and the changes in indoor temperature, humidity and CO2 are continuously observed until the expected comfort range is reached; after the task is completed, the relevant results will be used as long-term positive memory candidates.

[0215] The third example uses a home living environment as an application scenario, illustrating the implementation process of this application from negative sample labeling to adaptive adjustment by the user through continuous manual intervention in a pre-set automated task. The complete implementation process is as follows: (1) The system is set to automatically turn on fresh air exchange in the early morning; within the preset time window, the user manually turns off the fresh air N times in a row.

[0216] (2) Negative reflection sample marking: The reflection module records the corresponding execution results, associated STM session key, room identifier and manual intervention information, and marks the continuous manual intervention behavior as a negative reflection sample; this sample is used for subsequent strategy adjustment and is not directly retrieved as user preference.

[0217] (3) Adaptive adjustment: During the system's idle period, take stock of the negative reflection sample in batches and adjust the planning rules of the "early morning fresh air" related tasks in the knowledge graph or the comfort weight of the period.

[0218] (4) Model adjustment: If necessary, adjust the local environment control model parameters after offline inventory to adapt to this type of scenario.

[0219] The fourth example uses a home living environment as an application scenario, illustrating the implementation process of this application from local awareness reasoning to emergency response under the condition that the home's external network is completely disconnected. The complete implementation process is as follows: (1) The external network is completely disconnected. The millimeter-wave radar in the activity area of ​​the elderly at home detects that the equipment has been stationary for a long time and has not been operated, forming an abnormal state candidate.

[0220] (2) Local reasoning: The local agent judges the abnormal state candidates based on local perception data and triggers multi-sensor cross-verification.

[0221] (3) Emergency response: After cross-verification by multiple sensors, the gateway activates the whole house sound and light alarm, shuts off the gas valve, cuts off high-power equipment locally according to the security priority, and observes the equipment status to confirm the response results.

[0222] (4) Offline and privacy protection: Alarm records are cached locally during offline periods. After the network is restored, encrypted alarm summaries are uploaded to the cloud and pushed to emergency contacts according to alarm notification requirements; raw sensing data is stored in the local gateway.

[0223] like Figure 2 As shown, some embodiments of this application provide a structural schematic diagram of a voice control device for a smart home device. It should be understood that this device is related to... Figure 1 The method executed in the middle corresponds to the steps involved in the aforementioned method. The specific functions and effects of the device can be found in the description above. To avoid repetition, detailed descriptions are omitted here.

[0224] The voice control device of this smart home device includes: The acquisition unit 710 is used to acquire real-time multimodal perception data when a voice command is detected; wherein, the real-time multimodal perception data includes the voice command, the real-time sound source coordinates corresponding to the voice command, the real-time human body radar position information, the real-time environmental status information, and the real-time device status information; The spatial binding unit 720 is used to generate a spatial binding result for the target person based on the spatiotemporal relationship between the real-time sound source coordinates and the real-time human body radar position information; wherein, the spatial binding result includes room identifier, binding status and target trajectory identifier; Analysis unit 730 is used to perform hierarchical memory analysis based on spatial binding results and voice commands to obtain structured task objects and long-term memory retrieval results; The instruction generation unit 740 is used to generate an executable instruction queue based on a pre-built family knowledge graph, structured task objects, and long-term memory retrieval results. The execution unit 750 is used to execute corresponding smart home device control operations according to the executable instruction queue.

[0225] In this embodiment, the specific execution logic of each unit of the device corresponds one-to-one with the steps in the aforementioned method embodiment. The specific implementation method can be referred to the relevant description in the aforementioned method embodiment, and will not be repeated here.

[0226] like Figure 3As shown, this application provides an electronic device 800, which includes a processor 801 and a memory 802. The processor 801 and the memory 802 are interconnected and communicate with each other through a communication bus 803 and / or other forms of connection mechanism (not shown). The memory 802 stores a computer program that can be executed by the processor 801. When the computer program is run, the processor 801 executes the computer program to perform the method in any of the aforementioned optional implementations.

[0227] This application provides a readable storage medium storing a computer program, which, when executed by a processor, performs the method in any of the aforementioned optional implementations.

[0228] The readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0229] This application provides a computer program product, which includes a computer program that, when run by a processor, executes the method in any of the aforementioned optional implementations.

[0230] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application. In particular, as long as there is no conflict, the various technical features mentioned in the embodiments can be combined in any way. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A voice control method for a smart home device, characterized in that, include: When a voice command is detected, real-time multimodal perception data is collected; wherein, the real-time multimodal perception data includes the voice command, the real-time sound source coordinates corresponding to the voice command, real-time human body radar position information, real-time environmental status information, and real-time device status information; Based on the spatiotemporal relationship between the real-time sound source coordinates and the real-time human body radar position information, a spatial binding result for the target person is generated; wherein, the spatial binding result includes room identifier, binding status, and target trajectory identifier; Based on the spatial binding results and the voice commands, hierarchical memory analysis is performed to obtain structured task objects and long-term memory retrieval results; An executable instruction queue is generated based on the pre-constructed family knowledge graph, the structured task object, and the long-term memory retrieval results; The corresponding smart home device control operations are executed according to the executable instruction queue.

2. The voice control method for smart home devices according to claim 1, characterized in that, The acquisition of real-time multimodal sensing data includes: Voice commands and the original sound source coordinates of the voice commands are acquired through a microphone array; The original human body radar position information is collected by millimeter-wave radar; wherein, the original human body radar position information includes multiple original radar coordinates and multiple original motion trajectory identifiers; Raw environmental status information is collected through sensors; this raw environmental status information includes temperature and humidity, CO2 concentration, gas usage status, and door and window status. Obtain raw device power information; wherein, the raw device power information includes real-time power information reported by smart home devices and pre-configured device rated power; The original sound source coordinates, the original human body radar position information, the original environmental state information, and the original device power information are preprocessed to obtain real-time multimodal sensing data, which is then stored locally.

3. The voice control method for smart home devices according to claim 1, characterized in that, The step of generating a spatial binding result for the target person based on the spatiotemporal relationship between the real-time sound source coordinates and the real-time human body radar position information includes: Based on the spatiotemporal relationship between the real-time sound source coordinates and the real-time human body radar position information, the target person who issued the voice command and the target radar coordinates of the target person are determined. Based on the number of the target personnel, obtain the binding status corresponding to the target personnel, and obtain the target trajectory identifier of the target personnel based on the real-time human body radar location information; Based on the preset room boundary area, target sound source coordinates, and target radar coordinates, the room identifier of the current location of the target person is obtained; wherein, the target sound source coordinates are obtained by converting the real-time sound source coordinates to the home coordinate system; When the binding status is "bound", a spatial binding result is generated based on the target person's profile identifier, the room identifier, the binding status, and the target trajectory identifier; When there are multiple target personnel, the binding status is ambiguous. When the number of the target personnel is 1, the binding status is "bound". When the number of the target personnel is 0, the binding status is unbound.

4. The voice control method for smart home devices according to claim 3, characterized in that, Determining the target person issuing the voice command and the target radar coordinates of the target person based on the spatiotemporal relationship between the real-time sound source coordinates and the real-time human body radar position information includes: Based on the real-time human body radar position information, multiple candidate persons and multiple real-time radar coordinates corresponding to the multiple candidate persons are obtained. Based on the preset sensor installation parameters, the real-time sound source coordinates and the multiple real-time radar coordinates are transformed into the home coordinate system to obtain the target sound source coordinates and multiple candidate radar coordinates. Spatial distance calculations are performed on the coordinates of the target sound source and the coordinates of the multiple candidate radars to obtain multiple target spatial distances that correspond one-to-one with the multiple candidate radars. Spatial matching is performed based on the timestamp of the target sound source coordinates, the timestamps of the multiple candidate radar coordinates, and the spatial distances of the multiple targets to obtain the target radar coordinates that match the target sound source coordinates. The candidate personnel corresponding to the target radar coordinates are the target personnel who issued the voice command.

5. The voice control method for smart home devices according to claim 1, characterized in that, The hierarchical memory analysis based on the spatial binding result and the voice command, to obtain structured task objects and long-term memory retrieval results, includes: The voice command is subjected to speech recognition to obtain the original voice command content; Using the original voice command content as input, a large language model is invoked to extract control information; wherein, the control information includes device entity, action parameters, and preliminary intention semantics; Based on the real-time environmental status information and preset system constraint rules, environmental constraint conditions are generated. Based on the spatial binding results and the environmental constraints, generate the STM session key and LTM retrieval conditions; The STM context is obtained by searching the short-term session memory based on the STM session key, and the long-term vector memory is obtained by searching the long-term vector memory based on the LTM search conditions; wherein, the search window of the STM context is determined according to the instruction type of the voice command. Based on the original voice command content, the spatial binding result, the STM context, and the long-term memory retrieval result, the final intent is parsed to obtain a structured task object; The structured task object includes the room identifier, the STM session key, the device entity, the action parameters, the environmental constraints, and the preliminary intent semantics; and when the binding state is a bound state, the structured task object also includes the personnel profile identifier of the target personnel. When the binding state is unbound, no STM session key or LTM retrieval condition is generated.

6. The voice control method for smart home devices according to claim 5, characterized in that, The step of generating STM session keys and LTM retrieval conditions based on the spatial binding results and environmental constraints includes: When the binding status is "bound", an STM session key is generated based on the room identifier, the target trajectory identifier, and the current session identifier; and LTM search conditions are generated based on the room identifier, the personnel profile identifier, the preliminary intent semantics, and the environmental constraints. When the binding state is ambiguous, a room-level STM session key is generated based on the room identifier and the current session identifier; and room-level LTM search conditions are generated based on the room identifier, the preliminary intent semantics, and the environmental constraints.

7. The voice control method for smart home devices according to claim 1, characterized in that, The step of generating an executable instruction queue based on the pre-constructed family knowledge graph, the structured task object, and the long-term memory retrieval results includes: The composite instructions and control requirements are obtained based on the structured task object; wherein, the control requirements include the expected equipment state or the expected environmental effect; The composite instruction is decomposed into multiple initial sub-tasks based on a pre-constructed family knowledge graph and preset task planning rules. The target device information corresponding to the composite instruction is obtained based on the real-time multimodal sensing data; wherein, the target device information includes device capability description information, action mutual exclusion relationship, device rated power, loop capacity and device real-time power; Based on the target device information, conflict checks are performed on the multiple initial subtasks to obtain the check results; Based on the verification results, the priority strategy in the preset task planning rules, and the personnel preference information in the long-term memory retrieval results, the multiple initial sub-tasks are solved to obtain multiple target sub-tasks; wherein, the priority strategy includes safety priority being greater than human comfort, and human comfort priority being greater than equipment energy consumption. An executable instruction queue is generated based on the multiple target sub-tasks; wherein, the executable instruction queue includes room identifier, STM session key, device entity, action parameters, control requirements, instruction execution sequence, instruction execution priority and timeout threshold, and when the binding status is a bound status, the executable instruction queue also includes the personnel profile identifier of the target personnel.

8. The voice control method for smart home devices according to claim 1, characterized in that, After executing the corresponding smart home device control operation according to the executable instruction queue, the method further includes: Receive confirmation messages from smart home devices; The system continuously collects the execution status information of the smart home devices; the execution status information includes the device operating status, current environmental status data, and personnel status. Feedback samples are generated based on the execution status information and the control requirements indicated by the voice commands; During system idle periods, obtain the set of feedback samples generated within a preset time period; The edge weights, preset task planning rules, and model parameters of the large language model of the family knowledge graph are optimized based on the feedback sample set. The model parameters of the large language model are encrypted and uploaded to the cloud for federated learning after user authorization. The large language model is deployed on a local edge gateway, which is divided into independent application sandboxes. Third-party applications run within the independent application sandboxes and only have temporary access to the device interface and the real-time multimodal perception data during the current session. The temporary access is revoked after the session ends.

9. The voice control method for smart home devices according to claim 8, characterized in that, The step of generating feedback samples based on the execution status information and the control requirements indicated by the voice command includes: When it is determined from the execution status information that the control requirements indicated by the voice command have been met, a positive feedback sample is generated based on the STM session key and the room identifier. When it is determined from the execution status information that the control requirements indicated by the voice command have not been met, or when it is detected that the user has entered manual intervention information multiple times in a preset time window, the device control result and the manual intervention information are obtained, and a negative feedback sample is generated based on the device control result and the manual intervention information.

10. A voice control device for a smart home device, characterized in that, include: The acquisition unit is used to acquire real-time multimodal perception data when a voice command is detected; wherein, the real-time multimodal perception data includes the voice command, the real-time sound source coordinates corresponding to the voice command, real-time human body radar position information, real-time environmental status information, and real-time device status information; A spatial binding unit is used to generate a spatial binding result for a target person based on the spatiotemporal relationship between the real-time sound source coordinates and the real-time human body radar position information; wherein, the spatial binding result includes a room identifier, binding status, and target trajectory identifier; The analysis unit is used to perform hierarchical memory analysis based on the spatial binding result and the voice command to obtain structured task objects and long-term memory retrieval results; The instruction generation unit is used to generate an executable instruction queue based on the pre-constructed family knowledge graph, the structured task object, and the long-term memory retrieval results. An execution unit is used to perform corresponding smart home device control operations according to the executable instruction queue.

11. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the voice control method of the smart home device according to any one of claims 1 to 9.

12. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, performs the voice control method for the smart home device according to any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, performs the voice control method for the smart home device according to any one of claims 1 to 9.

Images