Personalized smart home device control method, electronic device, and program product

Abstract

The application provides a personalized smart home device control method, an electronic device and a program product. The method comprises: obtaining a voice instruction of a user and device attribute data corresponding to a smart home device; inputting the voice instruction and the device attribute data into a target large model, wherein the target large model comprises a cascaded first module and a second module, the first module has a first parameter group, the second module has a second parameter group, and the first parameter group and the second parameter group are obtained through end-to-end joint optimization; generating a multi-dimensional feature text according to the voice instruction through the first module, wherein the multi-dimensional feature text comprises a semantic feature text, a voiceprint feature text and an emotion feature text; outputting a target parameter according to the multi-dimensional feature text and the device attribute data through the second module; and generating an executable instruction of the smart home device according to the target parameter, wherein the executable instruction is used for controlling the smart home device to perform a corresponding operation.

Description

Technical Field

[0001] This application mainly relates to the field of Internet of Things (IoT) technology, and in particular to a personalized smart home device control method, electronic device, and program product. Background Technology

[0002] With the convergence of artificial intelligence (AI) and the Internet of Things (IoT) technologies, current smart home control methods can generate corresponding control commands based on recognized fuzzy natural language instructions. For example, recognizing a user's command to "generate an aurora scene" and controlling the corresponding smart lights to generate an aurora scene. This type of smart home control method significantly improves the convenience of smart home control.

[0003] However, these existing control methods generally employ segmented processing logic and rely solely on the semantic information of the transcribed text for control. In practical applications, they suffer from several shortcomings: First, they lack personalized user service capabilities, producing homogeneous control results when faced with the same commands from different users, making it difficult to adjust control strategies based on individual user preferences. Second, they lack user emotion perception capabilities, ignoring the emotional state of the user when issuing commands, resulting in control results that are difficult to match with the user's emotional state. Third, in existing methods, acoustic feature analysis typically only serves Automatic Speech Recognition (ASR) transcription, converting the content carried by speech into a visually readable string, lacking linkage with tasks such as sentiment analysis and user profile generation. Furthermore, the independent optimization of modules leads to bottlenecks in speech recognition performance.

[0004] Currently, existing technologies typically employ modular processing logic, with each module's parameters trained independently. For example, ASR transcription, emotion recognition, user profile generation, and control command generation modules are trained independently. This makes it difficult to achieve end-to-end joint optimization from voice signal input to user feature extraction and control command output, resulting in significant shortcomings in control response accuracy and personalized adaptation capabilities. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide a personalized smart home device control method, electronic device and program product, which can improve control response accuracy and personalization adaptability.

[0006] To address the aforementioned technical problems, this application provides a personalized smart home device control method, comprising: acquiring a user's voice command and device attribute data corresponding to the smart home device; inputting the voice command and the device attribute data into a target large model, wherein the target large model includes a cascaded first module and a second module, the first module having a first parameter set, the second module having a second parameter set, the first parameter set and the second parameter set being obtained through end-to-end joint optimization, wherein the training samples used for the end-to-end joint optimization include voice command examples, device attribute data examples, and executable command examples of the smart home device conforming to user preferences; generating multi-dimensional feature text based on the voice command through the first module, the multi-dimensional feature text including semantic feature text, voiceprint feature text, and emotional feature text; outputting target parameters through the second module based on the multi-dimensional feature text and the device attribute data; and generating executable commands for the smart home device based on the target parameters, the executable commands being used to control the smart home device to perform corresponding operations.

[0007] In one embodiment of this application, the semantic feature text includes the content of the voice command, the feature items of the voiceprint feature text include the user's identity, gender, age, tone and dialect, and the emotional feature text includes the emotion of the user when issuing the voice command and the corresponding emotional intensity.

[0008] In one embodiment of this application, the personalized smart home device control method further includes: maintaining several historical records in a user preference database, each historical record including processed multidimensional feature text and executed target parameters, wherein the executed target parameters are target parameters output by inputting the processed multidimensional feature text into the target large model; wherein the step of outputting the target parameters by the second module based on the multidimensional feature text and the device attribute data includes: searching the user's historical records in the user preference database; when a first record exists among the several historical records, and the processed multidimensional feature text in the first record reaches a preset matching degree with the multidimensional feature text, the executed target parameters in the first record are output as the target parameters; when the processed multidimensional feature text in each historical record does not reach a preset matching degree with the multidimensional feature text, the target parameters are output by the second module based on the several historical records, the multidimensional feature text, and the device attribute data.

[0009] In one embodiment of this application, the step of outputting a target parameter through the second module based on the plurality of historical records, the multidimensional feature text, and the device attribute data includes: filtering reference historical records from the plurality of historical records, wherein the similarity between the sentiment feature text in the reference historical records and the sentiment feature text in the multidimensional feature text is not lower than a preset threshold; and using the reference historical records, the multidimensional feature text, and the device attribute data as input, outputting the target parameter through the second module.

[0010] In one embodiment of this application, the preset rule base includes several constraint rules. Each constraint rule includes a constraint item and a target parameter threshold corresponding to the constraint item. The step of outputting the target parameter by the second module based on the multidimensional feature text and the device attribute data includes: searching for a constraint item in the constraint rules that is the same as a feature item in the voiceprint feature text; if no constraint item is found that is the same as a feature item in the voiceprint feature text, the initial target parameter output by the second module is output as the target parameter; if a constraint item is found that is the same as a feature item in the voiceprint feature text, the initial target parameter output by the second module is adjusted according to the corresponding target parameter threshold, and the adjusted initial target parameter is output as the target parameter.

[0011] In one embodiment of this application, the personalized smart home device control method further includes: acquiring a voiceprint entered by a user; generating a user profile text based on the voiceprint through the first module, the user profile text including user voiceprint feature text; storing the voiceprint and the user profile text as a user profile; wherein, generating multi-dimensional feature text based on the voice command through the first module includes: generating the voiceprint feature text in the multi-dimensional feature text based on the voice command and the user profile.

[0012] In one embodiment of this application, the smart home device includes at least three light sources with independently controllable light emission parameters. The target parameters include color value parameters and allocation ratios corresponding to the primary color, secondary color, and accent color, respectively. Generating executable instructions for the smart home device based on the target parameters includes: allocating the light sources to multiple gradient segments according to the number of light sources and the allocation ratios corresponding to the primary color, secondary color, and accent color, respectively. The multiple gradient segments include a first gradient segment transitioning from the primary color to the secondary color and a second gradient segment transitioning from the secondary color to the accent color; sampling the color value parameters corresponding to each light source in the color value space according to the number of light sources allocated to each gradient segment and the color value parameters corresponding to the primary color, secondary color, and accent color, respectively; and generating executable instructions for each light source based on the color value parameters of each light source.

[0013] In one embodiment of this application, the step of sampling the color value parameter corresponding to each light source in the color value space according to the number of light sources allocated to each gradient segment includes: calculating a first color vector from the color value parameter of the primary color to the color value parameter of the secondary color in the color value space; sampling the color value parameter corresponding to each light source in the first gradient segment according to the first color vector and the number of light sources in the first gradient segment; calculating a second color vector from the color value parameter of the secondary color to the color value parameter of the accent color in the color value space; and sampling the color value parameter corresponding to each light source in the second gradient segment according to the second color vector and the number of light sources in the second gradient segment.

[0014] In one embodiment of this application, the target parameter includes scene description text, and the step of outputting the target parameter by the second module based on the multidimensional feature text and the device attribute data includes: adjusting the scene description text by the second module based on the voiceprint feature text.

[0015] This application also proposes an electronic device comprising: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the method described above.

[0016] This application also proposes a computer program product including computer program code, which, when executed by one or more processors, implements the steps described above.

[0017] Compared with the prior art, this application has the following advantages: (1) The target big model of this application can directly generate personalized control commands based on the user's input voice commands and the device attribute data of smart home devices. There is no need to set up staged control logic. Moreover, by jointly optimizing the target big model end-to-end, it can achieve collaborative optimization of audio understanding and downstream control command generation, thereby avoiding the problem of independent module optimization getting stuck in local optima, and thus improving the control command generation effect. (2) The first module directly generates multi-dimensional feature text based on voice commands. There is no need to set up a sentiment analysis module and a user profile generation module separately. It can directly parse the full-dimensional information of the original voice input to obtain semantic feature text, voiceprint feature text and sentiment feature text, avoiding the result error and information loss caused by re-analysis based on ASR transcription content. (3) By inputting multidimensional feature text into the second module, the emotional feature text and the voiceprint feature text representing the user profile can be directly integrated into the control instruction generation logic, thereby adjusting the control result according to the user's personalized preferences and achieving the matching of the control result with the user's emotional state. Attached Figure Description

[0018] The accompanying drawings are included to provide a further understanding of this application; they are incorporated into and constitute a part of this application. The drawings illustrate embodiments of this application and, together with this specification, serve to explain the principles of this application. In the drawings: Figure 1 This is a flowchart illustrating a personalized smart home device control method according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a system for generating target parameters using a target large model according to an embodiment of this application; Figure 3 This is a schematic diagram of part of the operation flow of the second module according to an embodiment of this application; Figure 4 yes Figure 3 A detailed flowchart of step S143 in the second module's operation process is shown below; Figure 5 This is a schematic diagram of part of the operation flow of the second module according to an embodiment of this application; Figure 6 This is a flowchart illustrating the generation of executable instructions based on target parameters according to an embodiment of this application; Figure 7 This is a flowchart illustrating a method for sampling the color value parameters corresponding to each light source according to an embodiment of this application; Figure 8 This is a flowchart illustrating a method for establishing a user profile based on a user's voiceprint according to an embodiment of this application; Figure 9 This is a data flow diagram of the first module operating in voiceprint recognition mode according to an embodiment of this application; Figure 10 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application.

[0019] Reference numerals: target large model 200, first module 210, voiceprint encoding submodule 211, second module 220, device attribute database 300, user preference database 400, electronic device 500, internal communication bus 501, processor 502, read-only memory 503, random access memory 504, communication port 505, hard disk 506. Detailed Implementation

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this application. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0021] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0022] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0023] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, these terms have no special meaning and therefore should not be construed as limiting the scope of protection of this application. In addition, although the terminology used in this application is selected from commonly known and used terms, some terms mentioned in this application's specification may have been chosen by the applicant according to his or her judgment, and their detailed meanings are explained in the relevant sections of this description. Moreover, this application should be understood not only through the actual terms used, but also through the meaning implied by each term.

[0024] It should be understood that when a component is referred to as "on another component," "connected to another component," "coupled to another component," or "in contact with another component," it can be directly on, connected to, coupled to, or in contact with that other component, or there may be an intervening component. In contrast, when a component is referred to as "directly on another component," "directly connected to," "directly coupled to," or "directly in contact with" another component, there is no intervening component. Similarly, when a first component is referred to as "electrically contacting" or "electrically coupled to" a second component, there is an electrical path between the first and second components that allows current to flow. This electrical path may include capacitors, coupled inductors, and / or other components that allow current to flow, even if there is no direct contact between the conductive components.

[0025] This application proposes a personalized smart home device control method, which can generate personalized control commands based on user voice commands and device attribute data of smart home devices. The following specific embodiments illustrate this personalized smart home device control method.

[0026] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more steps may be removed from these processes.

[0027] refer to Figure 1 The flowchart shown is a schematic diagram of a personalized smart home device control method according to an embodiment. The control method includes... Figure 1 Steps S110 to S150 will be explained in detail below.

[0028] In step S110, the user's voice commands and the device attribute data corresponding to the smart home devices are obtained.

[0029] In practice, a voice acquisition module collects the user's voice commands and stores and transmits them as raw voice waveforms or spectrograms. This allows electronic components that perform subsequent processing steps to recognize the user's voice commands through the raw voice waveforms or spectrograms. The voice acquisition module can be built into smart home devices, such as a microphone within the device. Alternatively, it can be built into a user terminal, which may include, but is not limited to, smart home control panels, smartphones, and wearable devices.

[0030] The device attribute data corresponding to smart home devices includes performance data and operating area. In some embodiments, smart home devices include smart lighting fixtures. Taking a smart lighting fixture as an example, its device attribute data includes fixture type, fixture performance data, and fixture coverage area. Fixture performance data includes the numerical range of optical parameters, electrical parameters, control parameters, and functional dynamic parameters. In some embodiments, optical parameters include color value parameters of primary, secondary, and accent colors (such as the combination of the three primary colors RGB). In some embodiments, optical parameters also include color temperature parameters (in Kelvin K) and brightness parameters (in percentage). Electrical parameters include power, current, and voltage. Control parameters include dimming curves and flicker parameters. Functional dynamic parameters include projection direction and gradient speed, etc.

[0031] In some embodiments, reference Figure 2 The diagram shows the structure of a system that uses a target large model to generate target parameters. The target large model 200 is communicatively connected to the device attribute database 300. The device attribute database 300 pre-stores device attribute data corresponding to smart home devices in the user's space. In step S110, the target large model 200 can access and read the data stored in the device attribute database 300 to obtain the pre-stored device attribute data corresponding to smart home devices in the user's space. For example, after entering the house, the user issues a voice command to access the database and retrieve the pre-stored device attribute data corresponding to the smart home devices in that house.

[0032] In step S120, voice commands and device attribute data are input into the target large model. The target large model includes a cascaded first module and a second module. The first module has a first parameter set, and the second module has a second parameter set. The first parameter set and the second parameter set are obtained through end-to-end joint optimization. The training samples used for end-to-end joint optimization include voice command examples, device attribute data examples, and executable command examples of smart home devices that match user preferences.

[0033] like Figure 2 As shown, the target large model 200 includes a first module 210 and a second module 220 cascaded with the first module. The first module 210 has a first parameter set, and the second module 220 has a second parameter set. The input of the target large model 200 includes voice commands and device attribute data. Specifically, the input of the first module 210 includes voice commands, and the input of the second module 220 includes the output of the first module 210 and the device attribute data. The output of the second module 220 is the target parameters, which is also the output of the target large model 200. During the training phase of the target large model, the first parameter set and the second parameter set are learnable parameters, obtained through end-to-end joint optimization.

[0034] In the end-to-end joint optimization process, the first parameter set of the first module undergoes supervised fine-tuning (SFT) to adapt it to the joint output task of multi-dimensional feature text. The second parameter set of the second module undergoes supervised fine-tuning to adapt it to the task of generating target parameters based on multi-dimensional feature text and device attribute data. The first and second modules share loss parameters and jointly update the first and second parameter sets through backpropagation, thereby achieving end-to-end joint optimization.

[0035] In some embodiments, voice command examples and first device attribute data examples are used as input to the training samples, and executable command examples of smart home devices that conform to user preferences are used as output to the training samples. Each voice command example and first device attribute data example corresponds to one executable command example of a smart home device that conforms to user preferences. The training samples are then used to optimize the first parameter set and the second parameter set.

[0036] In some embodiments, after supervised fine-tuning of the first and second modules, reinforcement learning can be used to further optimize the first and second parameter sets. For example, Group Relative Policy Optimization (GRPO) can be employed.

[0037] In step S130, the first module generates multi-dimensional feature text based on the voice command. The multi-dimensional feature text includes semantic feature text, voiceprint feature text, and emotional feature text.

[0038] Specifically, the first module 210 integrates speech recognition, user profile generation, and sentiment analysis functions, and can directly generate multi-dimensional feature text based on the input voice command. Among them, the semantic feature text in the multi-dimensional feature text corresponds to the speech recognition result, the voiceprint feature text corresponds to the user profile generation result, and the sentiment feature text corresponds to the sentiment analysis result.

[0039] For example, the first module 210 can adopt a hierarchical architecture consisting of a backbone network, multiple output heads, and an output layer, with the multiple output heads connected in parallel between the backbone network and the output layer. The backbone network is used to extract shared features from the input speech commands. The multiple output heads are used to jointly perform speech recognition, user profile generation, and sentiment analysis tasks based on the shared features output by the backbone network. The output layer is used to control the multiple output heads to perform multiple tasks simultaneously, thereby outputting standardized multi-dimensional feature text. In this example, the parameters of the backbone network, multiple output heads, and the output layer constitute the first parameter group of the first module.

[0040] In some embodiments, during the training phase of the target large model and before end-to-end joint optimization, the first module 210 can be pre-trained. Specifically, the first module 210 is first pre-trained on large-scale speech data using multiple tasks, and then the pre-trained first module 210 is supervisedly fine-tuned on domain-annotated data. In some embodiments, during the training phase of the target large model and before end-to-end joint optimization, the pre-trained first module 210 can also be post-trained to adjust the first parameter set of the first module 210 according to the adaptability of the multi-dimensional feature text generation task. Specific methods for post-training can be found in related technologies and will not be elaborated here.

[0041] In some embodiments, the semantic feature text includes the content of the voice command, the voiceprint feature text includes features such as the user's identity, gender, age, tone, and dialect, and the emotional feature text includes the user's emotion when issuing the voice command and the corresponding emotional intensity. Specifically, the user's identity can be a family identity / role, the user's emotion when issuing the voice command can be happy, sad, or neutral, and the emotional intensity can be low, medium, or high. As an example, the first module 210 generates multi-dimensional feature text based on the user's voice command. This multi-dimensional feature text includes the semantic feature text "Generate aurora scene," the voiceprint feature text "Identity: Father; Gender: Male; Age: Middle-aged; Tone: Excited; Dialect: Mandarin Chinese," and the emotional feature text "Emotion: Happy; Emotion Intensity: High."

[0042] In step S140, the second module outputs the target parameters based on the multidimensional feature text and device attribute data.

[0043] Specifically, the second module 220 receives device attribute data and multi-dimensional feature text output by the first module 210 as input, and outputs target parameters accordingly. The target parameters include structured lighting parameters, scene name, and scene description. In some embodiments, the structured lighting parameters include color value parameters for primary, secondary, and accent colors. In some embodiments, the structured lighting parameters also include color temperature parameters, brightness parameters, and area allocation parameters, wherein the area allocation parameters are used to guide the coverage area of ​​the smart lighting fixtures. The scene name refers to the standardized text corresponding to the smart home device control scene output by the user's voice command, and the scene description refers to the explanatory text describing the smart home device control scene output by the user's voice command, used to provide feedback to the user on the scene control results. These target parameters can be used to generate executable instructions for the smart home device.

[0044] In some embodiments, the input to the second module 220 includes prompt words concatenated from the multidimensional feature text output by the first module 210 in the form of structured text. As an example, the prompt words concatenated from the multidimensional feature text output by the first module 210 in the form of structured text are as follows: [User Profile] Identity: Son, Age: Child, Emotion: Happy, Emotion Intensity: High, Tone: Excited; [Voice command] "I want to see the stars!"; [Device Attributes] Bedroom LED strip - Supports RGB, Ceiling spotlights - Supports RGB + color temperature; [Historical Preferences] Frequently uses blue and purple tones, and prefers shimmering effects.

[0045] This structured text format is a key step in generating personalized target parameters to enable personalized control of smart home devices.

[0046] In some embodiments, the second module 220 includes a pre-trained Large Language Model (LLM). For example, the pre-trained LLM in the second module 220 is a Transformer-based pre-trained LLM, such as the QWen series of open-source models or the Llama series of open-source models. Further, during the training phase of the target large model, a task is generated based on the target parameters, and the parameters of the pre-trained LLM in the second module 220 are fine-tuned to adapt to the target parameters.

[0047] In other embodiments, a lightweight deployment scheme can be used to deploy the pre-trained large language model. Specifically, the pre-trained large language model is first selected as the teacher model, and its knowledge is distilled to transfer the generalized knowledge it has learned to the student model, which has a significantly smaller number of parameters than the teacher model. In some embodiments, the number of parameters in the student model can be 30% to 60% of the number of parameters in the teacher model. Subsequently, the student model is post-trained using supervised training data manually annotated by the designer. This supervised training data includes voice command examples, corresponding multi-dimensional feature text and device attribute data, and the target parameters for the desired output.

[0048] In some embodiments, supervised training data is obtained through manual screening. Specifically, candidate target parameters are first generated based on candidate voice commands and device attribute data using a model with a larger parameter scale (e.g., 72B parameters). Then, high-quality samples are screened through manual evaluation to form thousands of supervised training data.

[0049] In some embodiments, the post-training process of the second module includes sequentially performing supervised fine-tuning and group relative policy optimization. Specifically, in the supervised fine-tuning phase, supervised fine-tuning is performed using supervised training data manually labeled by the designer, calculating the loss value for all assistant rounds. After the supervised fine-tuning phase, group relative policy optimization is used to perform reinforcement learning on the second module. Group relative policy optimization calculates the advantage function by the relative ranking of the sampling results within each group, thereby improving training efficiency.

[0050] In some embodiments, a reward model (RM) may also be employed during post-training to improve training efficiency. Specific implementation details of the above techniques can be found in related technologies and will not be elaborated further.

[0051] During the second module 220 operation phase, the student model in the second module 220 is driven to perform inference based on multidimensional feature text and device attribute data, and output target parameters based on the inference results.

[0052] Continue to refer to Figure 1 In step S150, executable instructions for the smart home device are generated based on the target parameters. These executable instructions are used to control the smart home device to perform corresponding operations. Taking a smart lamp as an example, the target parameters include structured lighting parameters, scene name, and scene description. The executable instructions generated based on the target parameters can, on the one hand, send executable instructions for lighting control to the smart lamp, and on the other hand, send audio playback instructions to the audio module of the smart lamp to play the corresponding scene description.

[0053] In some embodiments, the smart home device includes at least three light sources (such as LEDs) with independently controllable light emission parameters. The target parameters include the color value parameters and allocation ratios corresponding to the primary color, secondary color, and accent color, respectively. Based on the color value parameters and allocation ratios of the primary color, secondary color, and accent color, a spatial gradient calculation is performed to tile the calculated gradient color value parameters (RGB parameters) onto several LEDs of the smart home device. The range of the color value parameters is RGB (0,0,0) to RGB (255,255,255).

[0054] In some embodiments, the personalized smart home device control method of this application further includes: maintaining several historical records in a user preference database, each historical record including processed multidimensional feature text and executed target parameters, wherein the executed target parameters are the target parameters output by inputting the processed multidimensional feature text into a target large model. In specific implementations, such as... Figure 2As shown, the target large model 200 communicates with the user preference database 400 to maintain and retrieve historical records from the user preference database. The user preference database 400 is stored in tabular form, with each historical record being a record in the table. In some embodiments, a separate table can be created for each household to store the historical records generated by users within that household, thereby isolating information from different users and improving the security of user information.

[0055] Furthermore, in these embodiments, reference is made to Figure 3 The diagram shows a partial operation flow of the second module. Step S140 includes... Figure 3 Steps S141 to S143 are shown below, and steps S141 to S143 will be explained in detail below.

[0056] In step S141, the user's historical records are retrieved from the user preference database.

[0057] In practice, based on the identity revealed in the voiceprint feature text, historical records matching the user's identity are retrieved from the user preference database 400 and used as the user's generated historical records. The second module then determines the target parameters for the output based on the retrieved historical records. These historical records enable the second module to obtain the user's personalized preference information.

[0058] In step S142, when a first record exists among several historical records, and the processed multidimensional feature text in the first record reaches a preset matching degree with the multidimensional feature text, the executed target parameter in the first record is output as the target parameter.

[0059] In specific implementation, achieving the preset matching degree refers to the matching between key feature items in the processed multidimensional feature text and key feature items in the multidimensional feature text. Key feature items include semantic feature text, dialect in voiceprint feature text, and emotion in emotional feature text. For example, if the content of the voice command, the user's emotion, and the dialect in the processed multidimensional feature text all match the content of the voice command, the user's emotion, and the dialect in the multidimensional feature text output by the first module 210, then the processed multidimensional feature text is considered to have achieved the preset matching degree with the multidimensional feature text generated by the first module 210. This historical record is taken as the first record, and the executed target parameters in the first record are output as target parameters. In this way, there is no need to drive the model in the second module 220 to perform inference based on the multidimensional feature text and device attribute data, thereby avoiding redundant calculations and improving the output efficiency of the second module.

[0060] In step S143, when the processed multidimensional feature text in each historical record does not reach the preset matching degree with the multidimensional feature text, the target parameter is output by the second module based on several historical records, multidimensional feature text and device attribute data.

[0061] In practical implementation, "failure to achieve the preset matching degree" means that the key feature items in the processed multidimensional feature text do not completely match the key feature items of the multidimensional feature text. For example, if the processed multidimensional feature text in each historical record does not match any item in the content of the voice command, the user's emotion, or the dialect of the multidimensional feature text output by the first module 210, then it is considered that the processed multidimensional feature text and the multidimensional feature text generated by the first module 210 have not achieved the preset matching degree. At this time, based on several historical records generated by the user, the multidimensional feature text output by the first module 210, and the device attribute data, the target parameters are output by the second module.

[0062] In some embodiments, step S143 further includes Figure 4 Steps S1431 and S1432 are shown.

[0063] In step S1431, reference historical records are selected from several historical records. The similarity between the sentiment feature text in the reference historical records and the sentiment feature text in the multidimensional feature text is not lower than a preset threshold. Specifically, if the sentiment in the processed multidimensional feature text in the historical records matches the sentiment in the multidimensional feature text output by the first module, then the similarity between the sentiment feature text in these historical records and the sentiment feature text in the multidimensional feature text is considered to be not lower than the preset threshold. Historical records with a similarity to the sentiment feature text in the multidimensional feature text that is not lower than the preset threshold are selected from the user's several historical records and used as reference historical records.

[0064] In step S1432, the target parameters are output through the second module, taking the reference history, multidimensional feature text, and device attribute data as input. Specifically, the trained and optimized model in the second module is driven to perform inference using the reference history, device attribute data, and the multidimensional feature text output by the first module as input, and the inference result is output as the target parameters.

[0065] In some embodiments, the preset rule base includes several constraint rules, each including constraint terms and corresponding target parameter thresholds. Specifically, the constraint terms are pre-defined objects to be constrained, including at least one feature term from multi-dimensional feature text, such as the user's age. The target parameter thresholds are pre-defined constraint judgment conditions based on the objects to be constrained, including at least one parameter from the target parameters, such as the range of color temperature parameters. In these embodiments, when the trained and optimized model in the second module 220 performs inference, hard / soft constraints based on the preset rule base are automatically triggered to constrain the target parameters output by the second module 220. For example, the constraint rules include pre-defined safety constraint rules and aesthetic constraint rules based on the feature terms of voiceprint feature text. The safety constraint rules include: for the constraint term "elderly," setting the corresponding color temperature parameter range to 2700K to 4000K; for the constraint term "child," setting the corresponding flicker parameter to low frequency. The aesthetic constraint rules include: for the constraint term "excited," setting the gradient speed to slow.

[0066] Further, refer to Figure 5 The diagram illustrates a portion of the operational flow of the second module. In these embodiments, step S140 includes steps S144 to S146. In step S144, a constraint term identical to a feature term in the voiceprint feature text is searched from the constraint rules. In step S145, if no constraint term identical to a feature term in the voiceprint feature text is found, the initial target parameter output by the second module 220 is output as the target parameter. In step S146, if a constraint term identical to a feature term in the voiceprint feature text is found, the initial target parameter output by the second module 220 is adjusted according to the corresponding target parameter threshold, and the adjusted initial target parameter is output as the target parameter.

[0067] For example, the features in the voiceprint feature text include "Identity: Father; Gender: Male; Age: Middle-aged; Tone: Excited; Dialect: Mandarin Chinese". Based on this voiceprint feature text, constraints matching the identity, gender, age, tone, and dialect in the voiceprint feature text are searched from the constraint rules. The constraint "Tone: Excited" matching the tone in the voiceprint feature text is found, and the initial target parameters obtained from the model inference in the second module 220 are adjusted according to the corresponding gradual speed constraint. The adjusted initial target parameters are then output as the target parameters. In other examples, if no constraints matching the features in the voiceprint feature text are found, the initial target parameters obtained from the model inference in the second module are output as the target parameters.

[0068] In other embodiments, the target parameters include scene descriptions, and step S140 further includes adjusting the scene descriptions based on the voiceprint feature text via the second module 220. For example, if the age in the voiceprint feature text is "child," concrete and anthropomorphic words from a preset vocabulary, such as "magic," "elf," and "candy color," are used to adjust the scene descriptions. Another example is if the dialect in the voiceprint feature text is "Cantonese," the scene descriptions are in the corresponding dialect, making the scene descriptions more regionally relatable and evoking a sense of emotional connection.

[0069] In some embodiments, the smart home device includes at least three light sources with independently controllable light emission parameters, such as a light strip with multiple independently controllable LED beads. Accordingly, the target parameters output in step 140 include the allocation ratios of the primary color, secondary color, and accent color, respectively. (See reference...) Figure 6 The flowchart shown illustrates the generation of executable instructions based on target parameters. Step S150 further includes... Figure 6 Steps S151 to S153 are shown.

[0070] In step S151, based on the number of light sources and the corresponding allocation ratios of the main color, secondary color, and accent color, the light sources are allocated to multiple gradient segments. These multiple gradient segments include a first gradient segment transitioning from the main color to the secondary color and a second gradient segment transitioning from the secondary color to the accent color. As an example, the allocation ratio of the main color is 60%, the secondary color is 30%, and the accent color is 10%. The allocation ratio of the secondary color is evenly split, with half (15%) used to form a gradient transition with the main color and the other half (15%) used to form a gradient transition with the accent color. Assuming the light strip has 60 LEDs with independently controllable light emission parameters, the number of LEDs allocated to the first gradient segment transitioning from the main color to the secondary color is 60 × (60% + 15%) = 45, and the number of LEDs allocated to the second gradient segment transitioning from the secondary color to the accent color is 60 × (10% + 15%) = 15. It is understandable that the allocation ratio of secondary colors may not be equally divided, but rather divided according to proportions such as 40% / 60% or 70% / 30%.

[0071] In step S152, based on the number of light sources allocated to each gradient segment and the color value parameters corresponding to the main color, secondary color, and accent color, the color value parameters corresponding to each light source are sampled in the color value space. In specific implementation, the color value parameters corresponding to the main color, secondary color, and accent color are RGB parameters, and the color value space is the RGB space, that is, the color value space constructed from RGB (0,0,0) to RGB (255,255,255).

[0072] In some embodiments, such as Figure 7 As shown, step S152 further includes Figure 7The steps S1511 to S1514 are shown below. In step S1511, a first color vector is calculated in the color value space, from the color value parameter of the primary color to the color value parameter of the secondary color. In step S1512, based on the first color vector and the number of light sources in the first gradient segment, the color value parameter corresponding to each light source in the first gradient segment is sampled. In step S1513, a second color vector is calculated in the color value space, from the color value parameter of the secondary color to the color value parameter of the accent color. In step S1514, based on the second color vector and the number of light sources in the second gradient segment, the color value parameter corresponding to each light source in the second gradient segment is sampled.

[0073] Next, taking a light strip with multiple LEDs as an example, in the RGB space, a first color vector is calculated, starting with the RGB parameters of the primary color and ending with the RGB parameters of the secondary color. The first gradient segment is allocated 45 LEDs, and equidistant linear interpolation sampling is performed on the first color vector to obtain the RGB parameters corresponding to each of the 45 LEDs. In the RGB space, a second color vector is calculated, starting with the RGB parameters of the secondary color and ending with the RGB parameters of the accent color. The second gradient segment is allocated 15 LEDs, and equidistant linear interpolation sampling is performed on the second color vector to obtain the RGB parameters corresponding to each of the 15 LEDs.

[0074] In some embodiments, when the RGB spatial distance between the RGB parameters of the primary color and the secondary color is greater than a preset threshold, third-order Hermite spline interpolation sampling is performed on the first color vector to obtain the RGB parameters corresponding to each light source in the first gradient segment; when the RGB spatial distance between the RGB parameters of the secondary color and the accent color is greater than the preset threshold, third-order Hermite spline interpolation sampling is performed on the second color vector to obtain the RGB parameters corresponding to each light source in the second gradient segment. This makes the gradient transition between the first and second gradient segments smoother.

[0075] For example, for the first Each LED bead Its location is The color value parameters of the LED bead are sampled using third-order Hermitian spline interpolation. The specific calculation formula is as follows:

[0076] In the formula, This indicates the color value parameter of the LED chip. This represents the third-order Hermitian spline interpolation function. , and These represent the color value parameters for the primary color, secondary color, and accent color, respectively. This represents the gradient change rate. Specifically, it calculates the Euclidean distances between the color values ​​of the primary and secondary colors, the primary and accent colors, and the secondary and accent colors, and then calculates the weighted average of these three Euclidean distances to obtain the color contrast energy. The gradient change rate is adjusted based on the color contrast energy. For example, a preset reference gradient change rate is set, which is adjusted according to the ratio between color contrast energy and a preset energy threshold. The reference gradient change rate and the preset energy threshold can be set according to the type of luminaire.

[0077] In step S153, executable instructions for each light source are generated based on the color value parameters of each light source.

[0078] In some embodiments, the personalized smart home device control method of this application further includes establishing a user profile based on the user's voiceprint, with reference to... Figure 8 The flowchart shown is a method for building a user profile based on a user's voiceprint according to an embodiment. The method for building a user profile includes... Figure 8 Steps S160 to S180 are shown below. Steps S160 to S180 will be explained in detail below.

[0079] In step S160, the user's voiceprint is acquired. For new users, voiceprint recording can be performed by independently operating a user terminal (such as a smart home control panel, smartphone, or smart wearable device). For specific methods of voiceprint acquisition, please refer to the relevant content above.

[0080] In step S170, the first module generates user profile text based on the voiceprint, which includes user voiceprint feature text. Specifically, the user voiceprint feature text includes the user's identity, gender, age, and dialect. In some embodiments, the first module can be configured to operate in two modes: voiceprint recognition and command recognition. When a new user triggers voiceprint recording, such as... Figure 8 As shown, the first module 210 operates in voiceprint recognition mode and generates user profile text based on the voiceprint; when the user issues a voice command, such as... Figure 2 As shown, the first module 210 operates in command recognition mode and generates multi-dimensional feature text based on voice commands.

[0081] In some embodiments, such as Figure 9 As shown, the first module 210 also includes a voiceprint encoding submodule 211, which is used to extract voiceprint feature vectors from voice commands and match them with voiceprints stored in the user profile to output the user's identity.

[0082] In step S180, the voiceprint and user profile text are stored as a user profile. In some embodiments, the user can modify the generated user profile text before storing it as a user profile.

[0083] Since the user's voiceprint and user profile text are stored in the user profile, step S130 further includes generating voiceprint feature text in the multi-dimensional feature text based on the voice command and the user profile. Specifically, the first module can use the user profile text corresponding to the voiceprint with the highest similarity to guide the generation of voiceprint feature text based on the comparison results between the user's voice command and the voiceprint stored in the user profile, thereby improving the accuracy of the generated result.

[0084] An embodiment of this application also proposes a method such as Figure 10 The illustrated electronic device 500 can be deployed on a cloud server and communicate with smart home devices and user terminals. According to... Figure 9 The electronic device 500 may include an internal communication bus 501, a processor 502, a read-only memory (ROM) 503, a random access memory (RAM) 504, and a communication port 505. When used in a personal computer, the electronic device may also include a hard disk 506.

[0085] The internal communication bus 501 enables data communication between components of the electronic device 500. The processor 502 can perform judgments and issue prompts. In some embodiments, the processor 502 may consist of one or more processors. The communication port 505 enables data communication between the electronic device 500 and external devices. In some embodiments, the electronic device 500 can send and receive information and data from a network through the communication port 505.

[0086] Electronic device 500 may also include different forms of program storage units and data storage units, such as hard disk 506, read-only memory (ROM) 503, and random access memory (RAM) 504, capable of storing various data files used for computer processing and / or communication, as well as possible program instructions executed by processor 502. The processor executes these instructions to implement the main parts of the methods described above. The results of processor processing are transmitted to user equipment via a communication port and displayed on a user interface.

[0087] In addition, this application also proposes a computer program product, including computer program code, which, when executed by one or more processors, enables the implementation of the steps described above.

[0088] The basic concepts have been described above. Obviously, for those skilled in the art, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore remain within the spirit and scope of the exemplary embodiments of this application.

[0089] Furthermore, this application uses specific terms to describe embodiments of the application. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of the application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of the application can be appropriately combined.

[0090] Similarly, it should be noted that, in order to simplify the description of the present application and thus aid in the understanding of one or more embodiments, the foregoing description of the embodiments of the present application sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this disclosure method does not imply that the subject matter of the present application requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of the single embodiments disclosed above.

[0091] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of scope in some embodiments of this application are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0092] Although this application has been described with reference to specific embodiments, those skilled in the art should recognize that the above embodiments are only used to illustrate this application, and various equivalent changes or substitutions can be made without departing from the spirit of this application. Therefore, any changes or modifications to the above embodiments within the essential spirit of this application will fall within the scope of the claims of this application.

Claims

1. A personalized smart home device control method, characterized in that, include: Obtain the user's voice commands and the device attribute data corresponding to the smart home devices; The voice commands and device attribute data are input into a target large model, wherein the target large model includes a cascaded first module and a second module. The first module has a first parameter set, and the second module has a second parameter set. The first parameter set and the second parameter set are obtained through end-to-end joint optimization. The training samples used for the end-to-end joint optimization include voice command examples, device attribute data examples, and executable command examples of the smart home device that conform to user preferences. The first module generates multi-dimensional feature text based on the voice command. The multi-dimensional feature text includes semantic feature text, voiceprint feature text, and emotion feature text. The second module outputs target parameters based on the multidimensional feature text and the device attribute data; and The executable instructions for the smart home device are generated based on the target parameters. The executable instructions are used to control the smart home device to perform corresponding operations.

2. The personalized smart home device control method as described in claim 1, characterized in that, The semantic feature text includes the content of the voice command, the feature items of the voiceprint feature text include the user's identity, gender, age, tone and dialect, and the emotional feature text includes the emotion of the user when issuing the voice command and the corresponding emotional intensity.

3. The personalized smart home device control method as described in claim 1, characterized in that, Also includes: Several historical records are maintained in the user preference database. Each historical record includes processed multidimensional feature text and executed target parameters. The executed target parameters are the target parameters output by inputting the processed multidimensional feature text into the target large model. The step of outputting the target parameter by the second module based on the multidimensional feature text and the device attribute data includes: Retrieve the user's historical records from the user preference database; When a first record exists among the aforementioned historical records, and the processed multidimensional feature text in the first record reaches a preset matching degree with the multidimensional feature text, the executed target parameter in the first record is output as the target parameter. When the processed multidimensional feature text in each historical record does not reach a preset matching degree with the multidimensional feature text, the target parameter is output by the second module based on the several historical records, the multidimensional feature text and the device attribute data.

4. The personalized smart home device control method as described in claim 3, characterized in that, The step of outputting the target parameter through the second module based on the plurality of historical records, the multidimensional feature text, and the device attribute data includes: Reference historical records are obtained by filtering from the aforementioned historical records, and the similarity between the sentiment feature text in the reference historical records and the sentiment feature text in the multidimensional feature text is not lower than a preset threshold. Using the reference history, the multidimensional feature text, and the device attribute data as input, the second module outputs the target parameters.

5. The personalized smart home device control method as described in claim 1, characterized in that, The preset rule base includes several constraint rules, each constraint rule including a constraint term and a target parameter threshold corresponding to the constraint term. The step of outputting the target parameter through the second module based on the multidimensional feature text and the device attribute data includes: Find constraint terms from the constraint rules that are identical to the feature terms in the voiceprint feature text; If no constraint term matching the feature term in the voiceprint feature text is found, the initial target parameter output by the second module is output as the target parameter. If a constraint term identical to a feature term in the voiceprint feature text is found, the initial target parameter output by the second module is adjusted according to the corresponding target parameter threshold, and the adjusted initial target parameter is output as the target parameter.

6. The personalized smart home device control method as described in claim 1, characterized in that, Also includes: Obtain the user's recorded voiceprint; The first module generates user profile text based on the voiceprint, and the user profile text includes user voiceprint feature text. Store the voiceprint and the user profile text as a user profile; The process of generating multidimensional feature text by the first module based on the voice command includes: generating the voiceprint feature text in the multidimensional feature text based on the voice command and the user profile.

7. The personalized smart home device control method as described in claim 1, characterized in that, The smart home device includes at least three light sources with independently controllable light emission parameters. The target parameters include the color value parameters and allocation ratios corresponding to the primary color, secondary color, and accent color, respectively. Generating executable instructions for the smart home device based on the target parameters includes: Based on the number of light sources and the respective allocation ratios of the primary color, the secondary color, and the accent color, the light sources are allocated to multiple gradient segments, including a first gradient segment that transitions from the primary color to the secondary color and a second gradient segment that transitions from the secondary color to the accent color. Based on the number of light sources allocated to each gradient segment and the color value parameters corresponding to the main color, the secondary color, and the accent color, the color value parameters corresponding to each light source are sampled in the color value space. Generate executable instructions for each light source based on the color value parameters of each light source.

8. The personalized smart home device control method as described in claim 7, characterized in that, The step of sampling the color value parameter corresponding to each light source in the color value space according to the number of light sources allocated to each gradient segment includes: In the color value space, a first color vector is calculated from the color value parameters of the primary color to the color value parameters of the secondary color; Based on the first color vector and the number of light sources in the first gradient segment, the color value parameter corresponding to each light source in the first gradient segment is sampled. In the color value space, a second color vector is calculated from the color value parameters of the secondary color to the color value parameters of the accent color; Based on the second color vector and the number of light sources in the second gradient segment, the color value parameter corresponding to each light source in the second gradient segment is sampled.

9. The personalized smart home device control method as described in claim 1, characterized in that, The target parameters include scene descriptions, and the step of outputting the target parameters by the second module based on the multidimensional feature text and the device attribute data includes: The second module adjusts the scene description based on the voiceprint feature text.

10. An electronic device, comprising: Memory is used to store instructions that can be executed by the processor; as well as A processor for executing the instructions to implement the method as described in any one of claims 1-9.

11. A computer program product comprising computer program code, wherein when the computer program code is executed by one or more processors, the one or more processors implement the steps of the method as described in any one of claims 1-9.

Images