Control methods and devices for scene generation schemes, electronic equipment

Abstract

This application relates to the field of smart home technology and discloses a control method for a scene generation scheme, comprising: selecting policy parameters of a scene corresponding to user input information; linearly processing knowledge vectors in a knowledge vector library corresponding to the policy parameters to ensure consistent processing rules for different types of knowledge vectors; combining user behavior perception data, environmental perception data, policy parameters, and linearly processed knowledge vectors to obtain scene state description data of the current scene; and calculating a scene control scheme corresponding to the scene state description data based on a reinforcement learning algorithm; wherein the scene control scheme includes device information and device execution actions. This application can provide a scene control scheme for the current environment. The scene control scheme can adapt to changes in the environment and control home devices to automatically generate smart scenes. This application also discloses a control device and electronic device for a scene generation scheme.

Description

Technical Field

[0001] This application relates to the field of smart home technology, such as a control method and device for a scene generation scheme, and electronic equipment. Background Technology

[0002] Currently, with the development of artificial intelligence and the Internet of Things, smart scene services have become an important trend in life. However, current scene management is complex and requires manual settings by users, making it difficult for many users to effectively utilize these services.

[0003] To achieve the automatic generation of intelligent scenes, a method based on preset rules and a supervised learning model has been disclosed, relying on preset rules or simple algorithms for decision-making. A series of rules are defined to handle specific scenarios. A supervised learning model is used to train data to learn the mapping relationship between inputs and outputs.

[0004] In the process of implementing the embodiments of this disclosure, at least the following problems were found in the related art:

[0005] In related technologies, pre-defined rules need to be created and maintained manually, requiring users to spend a significant amount of time understanding and setting them, leading to difficulties in operation and a poor user experience for many. Furthermore, supervised learning models rely on large amounts of labeled data for training, and in practical applications, they cannot learn independently in the face of daily environmental changes, thus failing to meet the unique needs and preferences of each user.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0008] This disclosure provides a control method, apparatus, and electronic device for scene generation, which can learn automatically and generate intelligent scenes based on environmental changes.

[0009] In some embodiments, the control method of the scene generation scheme includes: selecting policy parameters of the scene corresponding to the user's input information; linearly processing the knowledge vectors in the knowledge vector library corresponding to the policy parameters to make the processing rules of different types of knowledge vectors consistent; combining user behavior perception data, environment perception data, policy parameters and linearly processed knowledge vectors to obtain scene state description data of the current scene; and calculating the scene control scheme corresponding to the scene state description data based on a reinforcement learning algorithm; wherein, the scene control scheme includes home appliances and the execution actions of the home appliances.

[0010] Optionally, the knowledge vectors in the knowledge vector base corresponding to the policy parameters are linearly processed, including: preprocessing the knowledge vectors in the knowledge vector base corresponding to the policy parameters; and extracting feature vectors from the preprocessed knowledge vectors.

[0011] Optionally, based on a reinforcement learning algorithm, a scenario control scheme corresponding to the scenario state description data is calculated, including: determining a reward function based on the knowledge vectors in the knowledge vector base corresponding to the policy parameters; calculating the expected reward values ​​of multiple scenario schemes through the reward function; wherein, multiple scenario schemes are determined according to different policy parameters; and selecting the scenario scheme corresponding to the largest expected reward value as the scenario control scheme.

[0012] Optionally, multiple scenario solutions can be obtained as follows: calculate the objective function of the update policy; update the policy parameters according to the gradient ascent method to obtain new policy parameters so as to maximize the objective function; obtain a new scenario solution based on the new policy parameters until the current update state meets the stopping criterion.

[0013] Optionally, the objective function of the update strategy is calculated, including calculating the objective function according to the following formula:

[0014] L(θ')=Et[π(at|st; θ') / π(at|st; θ)×At(θ)]

[0015] Where L(θ') is the objective function, Et is the expectation at time step t, θ is the initial policy parameter, θ' is the new policy parameter, π(at|st;θ) is the initial scheme, π(at|st;θ') is the new scheme, and At(θ) is the advantage function obtained at time step t according to the initial policy parameter θ.

[0016] Optionally, the policy parameters are updated according to the gradient ascent method to obtain new policy parameters, including obtaining new policy parameters according to the following formula:

[0017]

[0018] Where θ' is the new policy parameter, and θ is the initial policy parameter. Let θ be the gradient of the objective function with respect to θ, and α be the learning rate.

[0019] Optionally, updating the policy parameters according to the gradient ascent method to obtain new policy parameters further includes: determining the Kulebeck-Leibler KL divergence between the new policy parameters and the previous policy parameters; and controlling the KL divergence between the new policy parameters and the previous policy parameters to be lower than or equal to a preset KL divergence threshold.

[0020] Optionally, select strategy parameters for a scenario corresponding to the user's input information, including: classifying the user's input information to obtain classification results; and selecting strategy parameters for a scenario corresponding to the classification results.

[0021] In some embodiments, the control device for the scene generation scheme includes a processor and a memory storing program instructions, wherein the processor is configured to execute the control method for the scene generation scheme as described above when the program instructions are executed.

[0022] In some embodiments, the electronic device includes: an electronic device body; and a control device, as described above for the scene generation scheme, installed on the electronic device body.

[0023] The control method, apparatus, and electronic device for scene generation provided in this disclosure can achieve the following technical effects:

[0024] In this embodiment, policy parameters are selected based on user input information. These parameters allow for a preliminary understanding of the user's information, facilitating the selection of relevant knowledge from a large pool of information, thus ensuring subsequent operations are more closely aligned with the user's input. Linear processing of the knowledge vectors corresponding to the policy parameters in the knowledge vector library enables different types of knowledge to be processed and analyzed under the same rules. Combining user behavior perception data, environmental perception data, policy parameters, and linearly processed knowledge vectors allows for the integration of current environmental information into a complete current state description, represented by scene state description data. Using this scene state description data, reinforcement learning algorithms can provide a scene control scheme for the current environment. This scene control scheme can adapt to environmental changes and control home appliances to automatically generate intelligent scenes.

[0025] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0026] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein:

[0027] Figure 1 This is a schematic diagram of the hardware environment of a control method for a scene generation scheme provided in an embodiment of this disclosure;

[0028] Figure 2 This is a schematic diagram of a control method for a scene generation scheme provided in an embodiment of this disclosure;

[0029] Figure 3 This is a schematic diagram of a control method for another scene generation scheme provided in this embodiment of the disclosure;

[0030] Figure 4 This is a schematic diagram of a control method for another scene generation scheme provided in this embodiment of the disclosure;

[0031] Figure 5 This is a schematic diagram of a control method for another scene generation scheme provided in this embodiment of the disclosure;

[0032] Figure 6 This is a schematic diagram of a control device for a scene generation scheme provided in an embodiment of this disclosure. Detailed Implementation

[0033] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0034] The terms "first," "second," etc., used in the technical solutions described in this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0035] Unless otherwise stated, the term "multiple" means two or more.

[0036] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0037] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0038] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0039] Current smart home environments need to be able to self-adjust to adapt to new situations in various environments and maintain efficient and stable operation at all times. Reinforcement learning algorithms have significant advantages in terms of adaptability, ease of use, and stability.

[0040] According to one aspect of the embodiments of this application, a control method for a scene generation scheme is provided. This control method for a scene generation scheme is widely used in whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligence house ecosystems. Optionally, in this embodiment, the above-mentioned control method for a scene generation scheme can be applied to, for example... Figure 1 The hardware environment shown consists of smart home device 102 and server 104. Figure 1 As shown, server 104 is connected to smart home device 102 via a network and can be used to provide services (such as application services) to terminals or clients installed on terminals. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0041] The aforementioned networks may include, but are not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. Smart home devices 102 may not be limited to PCs, mobile phones, tablets, smart air conditioners, smart range hoods, smart refrigerators, smart ovens, smart stoves, smart washing machines, smart water heaters, smart washing equipment, smart dishwashers, smart projectors, smart TVs, smart clothes racks, smart curtains, smart audio-visual equipment, smart sockets, smart speakers, smart speakers, smart fresh air systems, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaners, smart window cleaning robots, smart mopping robots, smart air purifiers, smart steam ovens, smart microwave ovens, smart water heaters, smart air purifiers, smart water dispensers, smart door locks, etc.

[0042] It should be understood that Figure 1 The number of smart home devices and servers shown is merely illustrative. Depending on actual needs, there can be any number of smart home devices and servers. For example, one server can correspond to multiple smart home devices.

[0043] Combination Figure 2 As shown, this disclosure provides a control method for a scene generation scheme, including:

[0044] S201, the server selects the strategy parameters for the scenario corresponding to the user's input information.

[0045] S202, the server performs linear processing on the knowledge vectors in the knowledge vector library that correspond to the policy parameters, so that the processing rules for different types of knowledge vectors are consistent.

[0046] S203, the server combines user behavior perception data, environment perception data, policy parameters, and linearly processed knowledge vectors to obtain scene state description data for the current scene.

[0047] S204, the server uses a reinforcement learning algorithm to calculate a scene control scheme corresponding to the scene state description data; the scene control scheme includes smart home devices and the actions performed by the smart home devices.

[0048] The method provided in this disclosure selects policy parameters based on user input information. These parameters allow for a preliminary understanding of the user's information, facilitating the selection of relevant knowledge from a large pool of information, thus ensuring subsequent operations are more closely aligned with the user's input. Linear processing of the knowledge vectors corresponding to the policy parameters in the knowledge vector library enables different types of knowledge to be processed and analyzed under the same rules. Combining user behavior perception data, environmental perception data, policy parameters, and linearly processed knowledge vectors allows for the integration of current environmental information into a complete current state description, represented by scene state description data. Using this scene state description data, reinforcement learning algorithms can provide a scene control scheme for the current environment. This scene control scheme can adapt to environmental changes and control home appliances to automatically generate intelligent scenes.

[0049] Optionally, the knowledge vectors in the knowledge vector base corresponding to the policy parameters are linearly processed, including: preprocessing the knowledge vectors in the knowledge vector base corresponding to the policy parameters; and extracting feature vectors from the preprocessed knowledge vectors.

[0050] Combination Figure 3 As shown, this disclosure provides another control method for scene generation, including:

[0051] S301, the server selects the strategy parameters for the scenario corresponding to the user's input information.

[0052] S302, the server performs data preprocessing on the knowledge vectors in the knowledge vector base that correspond to the policy parameters.

[0053] S303, the server extracts feature vectors from the knowledge vectors that have undergone data preprocessing.

[0054] S304, the server combines user behavior perception data, environment perception data, policy parameters, and linearly processed knowledge vectors to obtain scene state description data for the current scene.

[0055] S305, the server uses a reinforcement learning algorithm to calculate a scene control scheme corresponding to the scene state description data; the scene control scheme includes smart home devices and the actions performed by the smart home devices.

[0056] The knowledge vector library stores various expert knowledge, represented in vector form. Upon receiving policy parameters, it linearly processes the relevant knowledge. The knowledge vector library is a crucial resource, containing a large number of industry-specific design rules. These rules can be viewed as predefined strategies based on expert knowledge and experience. For example, various "recipes" represent optimal parameter settings for a specific environment or requirement; these "recipes" are derived from scientific research and industry practice. "Recipes" can be directly used to generate specific actions or decisions, or served as initial values ​​or references for learning algorithms, helping the system find effective solutions more quickly. Simultaneously, utilizing these professional design rules allows for rapid and accurate responses to various environments and requirements, enabling the entire system to possess both expert-level expertise and the ability to learn and optimize itself.

[0057] In this embodiment of the disclosure, knowledge vectors from a knowledge vector base are received and linearly processed. This allows various types of knowledge to be processed and analyzed under the same rules. Data preprocessing includes cleaning the data and removing or imputing missing values. Data preprocessing also includes standardizing or normalizing the data to give different features the same scale.

[0058] Extracting feature vectors from knowledge vectors after data preprocessing can be achieved through PCA (Principal Component Analysis). PCA can reveal the linear relationships between data features by calculating the covariance matrix or correlation coefficient matrix. Eigenvalue decomposition of the covariance matrix or correlation coefficient matrix yields eigenvalues ​​and eigenvectors.

[0059] Optionally, based on a reinforcement learning algorithm, a scenario control scheme corresponding to the scenario state description data is calculated, including: determining a reward function based on the knowledge vectors in the knowledge vector base corresponding to the policy parameters; calculating the expected reward values ​​of multiple scenario schemes through the reward function; wherein, multiple scenario schemes are determined according to different policy parameters; and selecting the scenario scheme corresponding to the largest expected reward value as the scenario control scheme.

[0060] Combination Figure 4 As shown, this disclosure provides another control method for scene generation, including:

[0061] S401, the server selects the strategy parameters for the scenario corresponding to the user's input information.

[0062] S402, the server performs linear processing on the knowledge vectors in the knowledge vector library that correspond to the policy parameters, so that the processing rules for different types of knowledge vectors are consistent.

[0063] S403, the server combines user behavior perception data, environment perception data, policy parameters, and linearly processed knowledge vectors to obtain scene state description data for the current scene.

[0064] S404, the server determines the reward function based on the knowledge vectors in the knowledge vector base that correspond to the policy parameters.

[0065] S405, the server calculates the expected reward value of multiple scenario schemes through a reward function; where multiple scenario schemes are determined according to different policy parameters.

[0066] S406, the server will use the scenario scheme corresponding to the highest expected reward value as the scenario control scheme.

[0067] In this embodiment, the knowledge vector base contains different knowledge vectors, such as the effects of different lighting methods, angles, power, positions, color temperatures, brightness, and color rendering indices on visual acuity, mood, and viewing effects. A reward function is generated based on the knowledge vectors corresponding to the policy parameters in the expert knowledge base. This reward function can be used by PPO (Proximal Policy Optimization, a reinforcement learning algorithm) to evaluate different scenario solutions.

[0068] This embodiment first initializes the strategy parameters and sets an initial scenario plan. Then, based on the execution effect of the initial scenario plan, multiple different scenario plans are calculated. The execution effect can be obtained by sampling user reactions in various situations or historical data. The expected reward values ​​of the multiple scenario plans are calculated, and the scenario plan corresponding to the highest expected reward value is selected as the scenario control plan.

[0069] Optionally, multiple scenario solutions can be obtained as follows: calculate the objective function of the update policy; update the policy parameters according to the gradient ascent method to obtain new policy parameters so as to maximize the objective function; obtain a new scenario solution based on the new policy parameters until the current update state meets the stopping criterion.

[0070] In this embodiment, based on the execution effect of the scenario plan, an objective function can be calculated and the policy parameters corresponding to the scenario plan can be updated using the gradient ascent method. The goal of the PPO algorithm is to find new policy parameters that maximize the objective function. Typically, multiple iterations are required to calculate the objective function to find the optimal scenario plan. After the scenario plan is implemented and user feedback is obtained, the feedback results are used to update and optimize the policy parameters, completing one reinforcement learning cycle. For example, if user feedback indicates that the current lighting is too dim, the policy parameters are adjusted accordingly to provide brighter lighting in similar situations. The iterative process is repeated until a stopping criterion is met, such as reaching the maximum number of iterations, a small policy change, or an average reward exceeding a threshold.

[0071] Optionally, the objective function of the update strategy is calculated, including calculating the objective function according to the following formula:

[0072] L(θ')=Et[π(at|st; θ') / π(at|st; θ)×At(θ)]

[0073] Where L(θ') is the objective function, Et is the expectation at time step t, θ is the initial policy parameter, θ' is the new policy parameter, π(at|st;θ) is the initial scheme, π(at|st;θ') is the new scheme, and At(θ) is the advantage function obtained at time step t according to the initial policy parameter θ.

[0074] This disclosure defines a state space s and an action space a. State s represents the user's current environment and needs, such as current lighting conditions and the user's brightness requirements. Action a represents a scene scheme obtained by configuring various parameters of the device, such as the number of lights, lighting method, light emission angle, power, position, color temperature, brightness, and color rendering index.

[0075] The objective function L(θ') is calculated using a formula, and after multiple iterations, the maximum value of the objective function L(θ') is calculated using new policy parameters. The scenario scheme obtained using the policy parameters corresponding to the maximum value is used as the scenario control scheme.

[0076] Optionally, the policy parameters are updated according to the gradient ascent method to obtain new policy parameters, including obtaining new policy parameters according to the following formula:

[0077]

[0078] Where θ' is the new policy parameter, and θ is the initial policy parameter. Let θ be the gradient of the objective function with respect to θ, and α be the learning rate.

[0079] In this embodiment of the disclosure, gradient ascent is used to update the policy parameter θ to maximize the objective function L(θ').

[0080] In practical applications, when a user selects the eye health protection mode, it is necessary to find a set of strategy parameters θ' that maximizes the objective function L(θ').

[0081] First, define the state space s and the action space a. State s represents the current lighting conditions, the user's brightness requirements, etc. Action a represents the number of lights, lighting method, light emission angle, power, position, color temperature, brightness, color rendering index, etc.

[0082] Second, initialize the policy π(a|s; θ), where θ is the policy parameter representing the probability of selecting each action in a given state. For example, θ can be initialized as a uniform distribution, meaning each action has the same probability of being selected. Sample the current policy π(a|s; θ) and state s to obtain action a and reward r.

[0083] Third, calculate the objective function L(θ') based on the sampling results and the current policy, and use the gradient ascent method to update the policy parameter θ to maximize the objective function L(θ').

[0084] Fourth, after multiple iterative calculations, several updated strategy parameters are obtained, resulting in lighting schemes such as those shown in Table 1:

[0085] Table 1

[0086]

[0087]

[0088] Fifth, calculate the expected reward values ​​for various lighting schemes using Table 2. Table 2 is the reward function table for eye-protection lighting.

[0089] Table 2

[0090]

[0091] By combining the reward / penalty rules in Table 2, the expected reward values ​​for various lighting schemes can be calculated:

[0092] R1=1000-(|1-1|+|90-90|+|30-30|+|100-100|+|4500-4500|+|450-450|+|98-98|)=1000.

[0093] R2=1000-(|1-2|+|90-75|×10+|30-25|×5+|100-120|×10+|4500-4200|×100+|450-430| / 10+|98-92|×2)=886.

[0094] R2=1000-(|1-3|+|90-60|×10+|30-20|×5+|100-150|×10+|4500-4000|×100+|450-410| / 10+|98-85|×2)=770.

[0095] Where R1 is the expected reward value of Option 1, R2 is the expected reward value of Option 2, and R3 is the expected reward value of Option 3.

[0096] Thus, Option 1 will receive the highest reward, and the PPO algorithm will select Option 1's lighting scheme as the scene control scheme.

[0097] Optionally, updating the policy parameters according to the gradient ascent method to obtain new policy parameters further includes: determining the KL (Kullback-Leibler divergence) between the new policy parameters and the previous policy parameters; and controlling the KL divergence between the new policy parameters and the previous policy parameters to be lower than or equal to a preset KL divergence threshold.

[0098] In this embodiment of the disclosure, if the policy changes too drastically in each iteration during policy iteration, it may lead to instability in the learning process. To prevent instability caused by excessive differences between the old and new policy parameters, a constraint is added. The KL divergence between the new and previous policy parameters is controlled to be lower than or equal to a preset KL divergence threshold. KL divergence is a measure of the difference between two probability distributions and is commonly used to describe the difference between policy distributions. By limiting the KL divergence, it can be ensured that the new policy does not deviate too far from the old policy, which helps to avoid drastic policy changes and thus reduces potential instability and risk.

[0099] Optionally, select strategy parameters for a scenario corresponding to the user's input information, including: classifying the user's input information to obtain classification results; and selecting strategy parameters for a scenario corresponding to the classification results.

[0100] Combination Figure 5 As shown, this disclosure provides another control method for scene generation, including:

[0101] S501, the server classifies the user's input information and obtains the classification results.

[0102] S502, the server selects the strategy parameters for the scenario corresponding to the classification result.

[0103] S503: The server performs linear processing on the knowledge vectors in the knowledge vector library that correspond to the policy parameters, so that the processing rules for different types of knowledge vectors are consistent.

[0104] S504, the server combines user behavior perception data, environment perception data, policy parameters, and linearly processed knowledge vectors to obtain scene state description data for the current scene.

[0105] The S505 server uses a reinforcement learning algorithm to calculate a scene control scheme corresponding to the scene state description data; the scene control scheme includes smart home devices and the actions performed by the smart home devices.

[0106] In this embodiment, user input can be received in various forms, such as voice, text, or a graphical interface. Parsing user input, tracking user actions, or predicting user intent allows for the capture and understanding of user behavior, thus obtaining user input information. The user input information is then categorized to obtain classification results. Selecting appropriate strategy parameters based on the classification results ensures that the calculated strategy parameters better meet the user's needs. For example, if the input information is "avoid collision," then strategy parameters related to safe driving might be selected.

[0107] Combination Figure 6 As shown, this embodiment of the disclosure provides a control device 60 for a scene generation scheme, including a processor 600 and a memory 601. Optionally, the device 60 may further include a communication interface 602 and a bus 603. The processor 600, communication interface 602, and memory 601 can communicate with each other via the bus 603. The communication interface 602 can be used for information transmission. The processor 600 can call logical instructions in the memory 601 to execute the control method of the scene generation scheme described in the above embodiment.

[0108] Furthermore, the logic instructions in the aforementioned memory 601 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0109] The memory 601, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 600 executes functional applications and data processing by running the program instructions / modules stored in the memory 601, thereby implementing the control method of the scene generation scheme in the above embodiments.

[0110] The memory 601 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 601 may include high-speed random access memory and may also include non-volatile memory.

[0111] This disclosure provides an electronic device, including: an electronic device body, and a control device 60 for the scene generation scheme described above. The control device 60 for the scene generation scheme is mounted on the electronic device body. The mounting relationship described herein is not limited to placement inside the electronic device body, but also includes mounting connections with other components of the electronic device, including but not limited to physical connections, electrical connections, or signal transmission connections. Those skilled in the art will understand that the control device 60 for the scene generation scheme can be adapted to feasible electronic device bodies to achieve other feasible embodiments.

[0112] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., and other media capable of storing program code.

[0113] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0114] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0115] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to implement this embodiment according to actual needs. Furthermore, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A control method of a scenario generation scheme, characterized by, The method comprises: selecting a strategy parameter corresponding to a scene of the input information of the user; linearly processing a knowledge vector corresponding to the strategy parameter in a knowledge vector library, so that processing rules of different types of knowledge vectors are consistent; combining user behavior perception data, environment perception data, the strategy parameter, and the knowledge vector after linear processing to obtain scene state description data of the current scene; determining a reward function according to the knowledge vector corresponding to the strategy parameter in the knowledge vector library; calculating expected reward values of multiple scene schemes through the reward function; wherein the multiple scene schemes are determined according to different strategy parameters; taking a scene scheme corresponding to the maximum expected reward value as a scene control scheme; wherein the scene control scheme comprises device information and device execution actions.

2. The control method of a scenario generation scheme according to claim 1, characterized by, The linear processing of the knowledge vector corresponding to the strategy parameter in the knowledge vector library comprises: data preprocessing of the knowledge vector corresponding to the strategy parameter in the knowledge vector library; extracting a feature vector from the knowledge vector after data preprocessing.

3. The control method of a scenario generation scheme according to claim 1, characterized by, The multiple scene schemes are obtained in the following manner: calculating a target function of an updating strategy; updating the strategy parameter in a gradient ascent manner to obtain a new strategy parameter, so as to maximize the target function; obtaining a new scene scheme according to the new strategy parameter until a current updating state meets a stopping criterion.

4. The control method of a scenario generation scheme according to claim 3, characterized by, The calculation of the target function of the updating strategy comprises calculation of the target function according to the following formula: L(θ') = E t [π(a t |s t ;θ') / π(a t |s t ;θ) × A t (θ)] where L(θ') is the objective function, E t is the expectation at time step t, θ is the initial policy parameter, θ' is the new policy parameter, π(a t |s t is the initial policy, π(a t |s t is the new policy, A t (θ) is the advantage function obtained at time step t according to the initial policy parameter θ.

5. The control method of a scenario generation scheme according to claim 3, wherein The updating of the strategy parameter in the gradient ascent manner to obtain the new strategy parameter comprises obtaining the new strategy parameter according to the following formula: θ' = θ + a x θ L(θ') where θ' is the new policy parameter, θ is the initial policy parameter, θ L(θ') is the gradient of the objective function with respect to θ, and α is the learning rate.

6. The control method of a scenario generation scheme according to claim 5, wherein The updating of the strategy parameter in the gradient ascent manner to obtain the new strategy parameter further comprises: determining a Kullback-Leibler (KL) divergence between the new strategy parameter and a previous strategy parameter; controlling the KL divergence between the new strategy parameter and the previous strategy parameter to be lower than or equal to a preset KL divergence threshold.

7. The control method of a scenario generation scheme according to any one of claims 1 to 6, characterized by, The selection of the strategy parameter corresponding to the scene of the input information of the user comprises: classifying the input information of the user to obtain a classification result; selecting a strategy parameter corresponding to the scene of the classification result.

8. A control device of a scenario generation scheme, comprising a processor and a memory having stored program instructions, characterized in that, The processor is configured to execute the control method of the scene generation scheme according to any one of claims 1 to 7 when running the program instructions.

9. An electronic device, comprising: The device comprises: an electronic device body; the control device of the scene generation scheme according to claim 8 is installed in the electronic device body.

Images