Scene generation scheme control method and apparatus, and electronic device
Through the reinforcement learning algorithm combined with user input and environmental data, an intelligent scene control solution is generated, which solves the problem of user complex operations and supervised learning models in the existing technology that cannot learn by themselves, and realizes automatic generation and personalized adaptation of intelligent scenes.
Patent Information
- Application Number
- CN202411051642.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-01
AI Technical Summary
The existing intelligent scenario management technology requires users to manually set rules, which are complex and difficult to meet personalized needs, and the supervised learning model cannot learn by itself to adapt to environmental changes.
The reinforcement learning algorithm is adopted to generate scene control schemes through user input information, environment perception data and knowledge vector library, and automatically adjust home equipment to adapt to environmental changes to realize automatic generation of intelligent scenes.
It simplifies user operations, improves the adaptability of intelligent scenarios, can automatically adjust according to environmental changes, meets personalized needs, and improves user experience and adaptability.
Smart Images

Figure CN119002261A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of smart home technology, for example, to a control method and device for a scene generation solution, and an electronic device. Background Art
[0002] At present, with the development of artificial intelligence and the Internet of Things, intelligent scene services have become an important trend in life. However, the current scene management is complicated and requires users to set it manually, which makes it difficult for many users to effectively use these services.
[0003] In order to realize the automatic generation of intelligent scenes, the related technology discloses a method based on preset rules and supervised learning models, which relies on preset rules or simple algorithms to make decisions. A series of rules are defined to handle specific scenes. The mapping relationship between input and output is learned through supervised learning model training data.
[0004] In the process of implementing the embodiments of the present disclosure, it is found that there are at least the following problems in the related art:
[0005] In related technologies, preset rules need to be manually created and maintained, and users need to spend a lot of time to understand and set rules, which makes it difficult for many users to operate and has a poor user experience. In addition, supervised learning models rely on a large amount of labeled data for training. In actual applications, they cannot self-learn when faced with daily environmental changes, and thus cannot meet the unique needs and preferences of each user.
[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0007] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.
[0008] The embodiments of the present disclosure provide a control method and device for a scene generation solution, and an electronic device that can self-learn and automatically generate intelligent scenes according to environmental changes.
[0009] In some embodiments, the control method of the scene generation scheme includes: selecting strategy parameters of the scene corresponding to the user's input information; linearly processing the knowledge vectors corresponding to the strategy parameters in the knowledge vector library to make the processing rules of different types of knowledge vectors consistent; combining user behavior perception data, environmental perception data, strategy parameters and linearly processed knowledge vectors to obtain scene state description data of the current scene; based on a reinforcement learning algorithm, calculating a scene control scheme corresponding to the scene state description data; wherein the scene control scheme includes home appliances and execution actions of home appliances.
[0010] Optionally, linear processing is performed on the knowledge vectors corresponding to the strategy parameters in the knowledge vector library, including: data preprocessing is performed on the knowledge vectors corresponding to the strategy parameters in the knowledge vector library; and feature vectors are extracted from the knowledge vectors that have undergone data preprocessing.
[0011] Optionally, based on a reinforcement learning algorithm, a scene control scheme corresponding to the scene state description data is calculated, including: determining a reward function based on a knowledge vector corresponding to the strategy parameters in a knowledge vector library; calculating expected reward values of multiple scene schemes through the reward function; wherein the multiple scene schemes are determined based on different strategy parameters; and taking the scene scheme corresponding to the maximum expected reward value as the scene control scheme.
[0012] Optionally, multiple scenario solutions are obtained in the following manner: calculate the objective function of the update strategy; update the strategy parameters in a gradient ascent manner to obtain new strategy parameters to maximize the objective function; obtain new scenario solutions based on the new strategy parameters until the current update state meets the stopping criteria.
[0013] Optionally, calculating the objective function of the update strategy includes calculating the objective function according to the following formula:
[0014] L(θ')=Et[π(at|st; θ') / π(at|st; θ)×At(θ)]
[0015] Among them, L(θ') is the objective function, Et is the expectation at time step t, θ is the initial strategy parameter, θ' is the new strategy parameter, π(at|st;θ) is the initial plan, π(at|st;θ') is the new plan, and At(θ) is the advantage function obtained according to the initial strategy parameter θ at time step t.
[0016] Optionally, updating the policy parameters in a gradient ascent manner to obtain new policy parameters includes obtaining the new policy parameters in accordance with the following formula:
[0017]
[0018] Among them, θ' is the new strategy parameter, θ is the initial strategy parameter, is the gradient of the objective function with respect to θ, and α is the learning rate.
[0019] Optionally, updating the strategy parameters in a gradient ascent manner to obtain new strategy parameters also includes: determining the Kulibec-Leibler (KL) divergence between the new strategy parameters and the previous strategy parameters; and controlling the KL divergence between the new strategy parameters and the previous strategy parameters to be lower than or equal to a preset KL divergence threshold.
[0020] Optionally, selecting a strategy parameter for a scenario corresponding to the user's input information includes: classifying the user's input information to obtain a classification result; and selecting a strategy parameter for the scenario corresponding to the classification result.
[0021] In some embodiments, the control device of the scene generation scheme includes: a processor and a memory storing program instructions, and the processor is configured to execute the control method of the scene generation scheme as described above when running the program instructions.
[0022] In some embodiments, the electronic device includes: an electronic device body; and a control device for the scene generation solution as described above, installed on the electronic device body.
[0023] The control method and device of the scene generation solution and the electronic device provided by the embodiments of the present disclosure can achieve the following technical effects:
[0024] In the disclosed embodiment, the policy parameters are selected according to the user's input information, and the user's information can be initially understood through the policy parameters, so that the knowledge related to the user's input can be selected from a large amount of knowledge, so that the subsequent operations are more matched with the user's input. Linear processing of the knowledge vectors corresponding to the policy parameters in the knowledge vector library can enable different types of knowledge to be processed and analyzed under the same rules. After combining the user behavior perception data, the environmental perception data, the policy parameters and the linearly processed knowledge vectors, the current environmental information can be combined to form a complete current state description, which is represented by the scene state description data. Using the scene state description data, the reinforcement learning algorithm can provide a scene control solution for the current environment. The scene control solution can adapt to changes in the environment and control home appliances to automatically generate smart scenes.
[0025] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:
[0027] Figure 1 It is a hardware environment diagram of a control method of a scene generation solution provided by an embodiment of the present disclosure;
[0028] Figure 2 is a schematic diagram of a control method for a scene generation solution provided by an embodiment of the present disclosure;
[0029] Figure 3 is a schematic diagram of a control method of another scene generation solution provided by an embodiment of the present disclosure;
[0030] Figure 4 is a schematic diagram of a control method of another scene generation solution provided by an embodiment of the present disclosure;
[0031] Figure 5 is a schematic diagram of a control method of another scene generation solution provided by an embodiment of the present disclosure;
[0032] Figure 6 It is a schematic diagram of a control device of a scene generation solution provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0033] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.
[0034] The terms "first", "second", etc. in the technical solutions described in the embodiments of the present disclosure are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present disclosure described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.
[0035] Unless otherwise stated, the term "plurality" means two or more.
[0036] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.
[0037] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.
[0038] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.
[0039] The current smart home environment needs to be able to self-adjust to adapt to new situations in various environments and always maintain efficient and stable operation. Reinforcement learning algorithms have significant advantages in adaptability, ease of use, and stability.
[0040] According to one aspect of an embodiment of the present application, a control method for a scene generation scheme is provided. The control method for the scene generation scheme is widely used in smart home (Smart Home), smart home, smart home device ecology, smart residential (Intelligence House) ecology and other whole-house intelligent digital control application scenarios. Optionally, in this embodiment, the control method of the above scene generation scheme can be applied to Figure 1 In the hardware environment composed of the smart home device 102 and the server 104 shown in FIG. Figure 1 As shown, the server 104 is connected to the smart home device 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or the client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.
[0041] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The smart home device 102 may be but is not limited to a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing equipment, a smart dishwasher, a smart projection equipment, a smart TV, a smart clothes drying rack, a smart curtain, a smart audio and video, a smart socket, a smart speaker, a smart speaker, a smart fresh air equipment, a smart kitchen and bathroom equipment, a smart bathroom equipment, a smart sweeping robot, a smart window cleaning robot, a smart mopping robot, a smart air purification equipment, a smart steamer, a smart microwave oven, a smart kitchen treasure, a smart purifier, a smart water dispenser, a smart door lock, etc.
[0042] It should be understood that Figure 1 The number of smart home devices and servers in the figure is only for illustration. According to actual needs, there may be any number of smart home devices and servers. For example, one server may correspond to multiple smart home devices.
[0043] Combination Figure 2 As shown, the present disclosure provides a control method for a scene generation scheme, including:
[0044] S201: The server selects a policy parameter of a scenario corresponding to user input information.
[0045] S202: The server performs linear processing on the knowledge vectors corresponding to the policy parameters in the knowledge vector library to make the processing rules of different types of knowledge vectors consistent.
[0046] S203, the server combines the user behavior perception data, the environment perception data, the strategy parameters and the linearly processed knowledge vector to obtain the scene state description data of the current scene.
[0047] S204, the server calculates a scene control solution corresponding to the scene state description data based on a reinforcement learning algorithm; wherein the scene control solution includes smart home devices and execution actions of the smart home devices.
[0048] By adopting the method provided by the embodiment of the present disclosure, the policy parameters are selected according to the user's input information, and a preliminary understanding of the user's information can be made through the policy parameters, so as to facilitate the selection of knowledge related to the user's input from a large amount of knowledge, so that subsequent operations are more matched with the user's input. Linear processing of the knowledge vectors corresponding to the policy parameters in the knowledge vector library can enable different types of knowledge to be processed and analyzed under the same rules. After combining the user behavior perception data, the environmental perception data, the policy parameters and the linearly processed knowledge vectors, the current environmental information can be combined to form a complete current state description, which is represented by the scene state description data. Using the scene state description data, the reinforcement learning algorithm can provide a scene control solution for the current environment. The scene control solution can adapt to changes in the environment and control home appliances to automatically generate smart scenes.
[0049] Optionally, linear processing is performed on the knowledge vectors corresponding to the strategy parameters in the knowledge vector library, including: data preprocessing is performed on the knowledge vectors corresponding to the strategy parameters in the knowledge vector library; and feature vectors are extracted from the knowledge vectors that have undergone data preprocessing.
[0050] Combination Figure 3 As shown, the embodiment of the present disclosure provides another control method for a scene generation solution, including:
[0051] S301: The server selects a policy parameter of a scenario corresponding to user input information.
[0052] S302, the server performs data preprocessing on the knowledge vectors corresponding to the strategy parameters in the knowledge vector library.
[0053] S303, the server extracts a feature vector from the knowledge vector that has undergone data preprocessing.
[0054] S304, the server combines the user behavior perception data, the environment perception data, the strategy parameters and the linearly processed knowledge vector to obtain the scene state description data of the current scene.
[0055] S305, the server calculates a scene control solution corresponding to the scene state description data based on a reinforcement learning algorithm; wherein the scene control solution includes smart home devices and execution actions of the smart home devices.
[0056] The knowledge vector library stores various expert knowledge in the form of vectors. When the policy parameters are received, the relevant knowledge will be processed linearly. The knowledge vector library is a key resource that contains a large number of industry-specific design rules, which can be regarded as predefined strategies based on expert knowledge and experience. For example, various "recipes" are the best parameter settings for a specific environment or demand. The "recipes" are based on scientific research and industry practice. "Recipes" can be used directly to generate specific actions or decisions, or as initial values or references for learning algorithms to help the system find effective solutions faster. At the same time, using these professional design rules, it is possible to respond quickly and accurately to a variety of different environments and needs, so that the entire system has both expert-level expertise and the ability to self-learn and optimize.
[0057] In the disclosed embodiment, the knowledge vector transmitted by the knowledge vector library is received and linearly processed. In this way, different types of knowledge can be processed and analyzed under the same rules. Data preprocessing includes cleaning data and removing or filling missing values. Data preprocessing also includes standardizing or normalizing data so that different features have the same scale.
[0058] Extracting eigenvectors from knowledge vectors that have undergone data preprocessing can be achieved through PCA (Principal Component Analysis). PCA can understand the linear relationship between data features by calculating the covariance matrix or the correlation coefficient matrix. Performing eigenvalue decomposition on the covariance matrix or the correlation coefficient matrix can obtain eigenvalues and eigenvectors.
[0059] Optionally, based on a reinforcement learning algorithm, a scene control scheme corresponding to the scene state description data is calculated, including: determining a reward function based on a knowledge vector corresponding to the strategy parameters in a knowledge vector library; calculating expected reward values of multiple scene schemes through the reward function; wherein the multiple scene schemes are determined based on different strategy parameters; and taking the scene scheme corresponding to the maximum expected reward value as the scene control scheme.
[0060] Combination Figure 4 As shown, the embodiment of the present disclosure provides another control method for a scene generation solution, including:
[0061] S401: The server selects a policy parameter of a scenario corresponding to the user's input information.
[0062] S402: The server performs linear processing on the knowledge vectors corresponding to the policy parameters in the knowledge vector library to make the processing rules of different types of knowledge vectors consistent.
[0063] S403, the server combines the user behavior perception data, the environment perception data, the strategy parameters and the linearly processed knowledge vector to obtain the scene state description data of the current scene.
[0064] S404: The server determines a reward function according to the knowledge vector corresponding to the strategy parameter in the knowledge vector library.
[0065] S405, the server calculates the expected reward values of multiple scenario solutions through a reward function; wherein the multiple scenario solutions are determined according to different strategy parameters.
[0066] S406: The server uses the scenario solution corresponding to the maximum expected reward value as the scenario control solution.
[0067] In the disclosed embodiment, the knowledge vector library contains different knowledge vectors, for example, the effects of different lighting methods, angles, power, positions, color temperatures, brightness, and color rendering index on vision, emotions, and viewing effects. A reward function is generated based on the knowledge vectors corresponding to the policy parameters in the expert knowledge library. The reward function can be used by PPO (Proximal Policy Optimization, reinforcement learning algorithm) to evaluate different scenario solutions.
[0068] The disclosed embodiment first initializes the policy parameters and sets an initial scenario. Then, multiple different scenario plans are calculated based on the execution effect of the initial scenario. The execution effect can be obtained by sampling the user's reactions or historical data in various situations. The expected reward values of multiple scenario plans are calculated, and the scenario plan corresponding to the largest expected reward value is used as the scenario control plan.
[0069] Optionally, multiple scenario solutions are obtained in the following manner: calculate the objective function of the update strategy; update the strategy parameters in a gradient ascent manner to obtain new strategy parameters to maximize the objective function; obtain new scenario solutions based on the new strategy parameters until the current update state meets the stopping criteria.
[0070] In the disclosed embodiment, according to the execution effect of the scenario solution, the objective function can be calculated and the policy parameters corresponding to the scenario solution can be updated using the gradient ascent method. The goal of the PPO algorithm is to find new policy parameters to maximize the objective function. Usually, multiple iterations are required to calculate the objective function to find the optimal scenario solution. After the scenario solution is implemented and user feedback is obtained, the feedback results are used to update and optimize the policy parameters to complete a reinforcement learning. For example, if user feedback indicates that the current lighting is too dim, the policy parameters are adjusted accordingly to provide brighter lighting in similar situations. The iterative process is repeated until the stopping criteria are met, such as reaching the maximum number of iterations, the policy change is small, or the average reward exceeds a threshold.
[0071] Optionally, calculating the objective function of the update strategy includes calculating the objective function according to the following formula:
[0072] L(θ')=Et[π(at|st; θ') / π(at|st; θ)×At(θ)]
[0073] Among them, L(θ') is the objective function, Et is the expectation at time step t, θ is the initial strategy parameter, θ' is the new strategy parameter, π(at|st;θ) is the initial plan, π(at|st;θ') is the new plan, and At(θ) is the advantage function obtained according to the initial strategy parameter θ at time step t.
[0074] The disclosed embodiment defines a state space s and an action space a. State s is the user's current environment and needs, such as the current lighting conditions, the user's needs for brightness, etc. Action a is a scene solution obtained by configuring various parameters of the device, such as the number of lights, lighting mode, light angle, power, position, color temperature, brightness, color rendering index, etc.
[0075] The objective function L(θ') is calculated using the formula, and after multiple iterations, the maximum value of the objective function L(θ') is calculated using the new strategy parameters. The scenario solution obtained by the strategy parameters corresponding to the maximum value is used as the scenario control solution.
[0076] Optionally, updating the policy parameters in a gradient ascent manner to obtain new policy parameters includes obtaining the new policy parameters in accordance with the following formula:
[0077]
[0078] Among them, θ' is the new strategy parameter, θ is the initial strategy parameter, is the gradient of the objective function with respect to θ, and α is the learning rate.
[0079] In the disclosed embodiment, a gradient ascent method is used to update the policy parameter θ to maximize the objective function L(θ').
[0080] In practical applications, when the user selects the healthy eye protection mode, a set of strategy parameters θ' needs to be found to maximize the objective function L(θ').
[0081] First, define the state space s and action space a. State s refers to the current lighting conditions, user requirements for brightness, etc. Action a refers to the number of lights, lighting mode, lighting angle, power, position, color temperature, brightness, color rendering index, etc.
[0082] Second, initialize the policy π(a|s; θ), where θ is the policy parameter, which is the probability of selecting each action in a given state. For example, θ can be initialized to a uniform distribution, that is, each action has the same probability of being selected. Sampling is performed based on the current policy π(a|s; θ) and state s to obtain action a and reward r.
[0083] Third, the objective function L(θ') is calculated based on the sampling results and the current strategy, and the gradient ascent method is used to update the strategy parameters θ to maximize the objective function L(θ').
[0084] Fourth, after multiple iterative calculations, multiple updated strategy parameters are obtained, thereby obtaining a lighting solution such as that shown in Table 1:
[0085] Table 1
[0086]
[0087]
[0088] Fifth, the expected reward values of various lighting schemes are calculated through Table 2. Table 2 is a light eye protection reward function table.
[0089] Table 2
[0090]
[0091] Combining the reward / penalty rules in Table 2, the expected reward values for various lighting schemes can be calculated:
[0092] R1=1000-(|1-1|+|90-90|+|30-30|+|100-100|+|4500-4500|+|450-450|+|98-98|)=1000.
[0093] R2=1000-(|1-2|+|90-75|×10+|30-25|×5+|100-120|×10+|4500-4200|×100+|450-430| / 10+|98-92|×2)=886.
[0094] R2=1000-(|1-3|+|90-60|×10+|30-20|×5+|100-150|×10+|4500-4000|×100+|450-410| / 10+|98-85|×2)=770.
[0095] Among them, R1 is the expected reward value of plan one, R2 is the expected reward value of plan two, and R3 is the expected reward value of plan three.
[0096] In this way, scheme 1 will get the highest reward, and the PPO algorithm will select the lighting scheme of scheme 1 as the scene control scheme.
[0097] Optionally, updating the strategy parameters in a gradient ascent manner to obtain new strategy parameters also includes: determining a KL (Kullback-Leibler divergence) divergence between the new strategy parameters and the previous strategy parameters; and controlling the KL divergence between the new strategy parameters and the previous strategy parameters to be lower than or equal to a preset KL divergence threshold.
[0098] In the disclosed embodiment, during the policy iteration process, if the policy changes in each iteration are too large, the learning process may become unstable. In order to prevent the learning process from being unstable due to the large difference between the new and old policy parameters, a constraint condition needs to be added. The KL divergence between the new policy parameters and the previous policy parameters is controlled to be lower than or equal to the preset KL divergence threshold. KL divergence is a measure of the difference between two probability distributions, and is typically used to describe the difference between policy distributions. By limiting the KL divergence, it can be ensured that the new policy does not deviate too far from the old policy, which helps to avoid drastic changes in policy, thereby reducing potential instability and risk.
[0099] Optionally, selecting a strategy parameter for a scenario corresponding to the user's input information includes: classifying the user's input information to obtain a classification result; and selecting a strategy parameter for the scenario corresponding to the classification result.
[0100] Combination Figure 5 As shown, the embodiment of the present disclosure provides another control method for a scene generation solution, including:
[0101] S501, the server classifies the user's input information and obtains classification results.
[0102] S502: The server selects a policy parameter of a scenario corresponding to the classification result.
[0103] S503: The server performs linear processing on the knowledge vectors corresponding to the policy parameters in the knowledge vector library to make the processing rules of different types of knowledge vectors consistent.
[0104] S504, the server combines the user behavior perception data, the environment perception data, the strategy parameters and the linearly processed knowledge vector to obtain the scene state description data of the current scene.
[0105] S505, the server calculates a scene control solution corresponding to the scene state description data based on a reinforcement learning algorithm; wherein the scene control solution includes smart home devices and execution actions of the smart home devices.
[0106] In the embodiments of the present disclosure, various forms such as voice, text or graphical interface can be adopted to receive user input. Parsing user input, tracking user actions or predicting user intentions can capture and understand user behavior and obtain user input information. Classify the user's input information to obtain classification results. Selecting corresponding strategy parameters based on the classification results can make the strategy parameters used for calculation more in line with user needs. For example, if the input information is "avoid collision", some strategy parameters related to safe driving may be selected.
[0107] Combination Figure 6 As shown, the embodiment of the present disclosure provides a control device 60 for a scene generation scheme, including a processor (processor) 600 and a memory (memory) 601. Optionally, the device 60 may also include a communication interface (Communication Interface) 602 and a bus 603. Among them, the processor 600, the communication interface 602, and the memory 601 can communicate with each other through the bus 603. The communication interface 602 can be used for information transmission. The processor 600 can call the logic instructions in the memory 601 to execute the control method of the scene generation scheme of the above embodiment.
[0108] In addition, the logic instructions in the memory 601 described above can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.
[0109] The memory 601 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 600 executes the function application and data processing by running the program instructions / modules stored in the memory 601, that is, the control method of the scene generation solution in the above embodiment is implemented.
[0110] The memory 601 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 601 may include a high-speed random access memory and may also include a non-volatile memory.
[0111] The disclosed embodiment provides an electronic device, including: an electronic device body, and a control device 60 of the above-mentioned scene generation scheme. The control device 60 of the scene generation scheme is installed in the electronic device body. The installation relationship described here is not limited to placement inside the electronic device body, but also includes installation connections with other components of the electronic device, including but not limited to physical connections, electrical connections or signal transmission connections, etc. Those skilled in the art can understand that the control device 60 of the scene generation scheme can be adapted to a feasible electronic device body, thereby realizing other feasible embodiments.
[0112] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes.
[0113] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.
[0114] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.
[0115] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.
[0116] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.
Claims
1. A control method for a scene generation scheme, characterized in that: include: Selecting a strategy parameter for a scenario corresponding to the user's input information; The knowledge vectors corresponding to the strategy parameters in the knowledge vector library are processed linearly to make the processing rules of different types of knowledge vectors consistent; Combine user behavior perception data, environment perception data, strategy parameters and linearly processed knowledge vectors to obtain scene state description data of the current scene; Based on a reinforcement learning algorithm, a scene control solution corresponding to the scene state description data is calculated; wherein the scene control solution includes device information and device execution actions.
2. The control method of the scene generation scheme according to claim 1, characterized in that: The knowledge vectors corresponding to the strategy parameters in the knowledge vector library are linearly processed, including: Perform data preprocessing on the knowledge vectors corresponding to the strategy parameters in the knowledge vector library; Extract feature vectors from the knowledge vector after data preprocessing.
3. The control method of the scene generation scheme according to claim 1, characterized in that: Based on the reinforcement learning algorithm, the scene control scheme corresponding to the scene state description data is calculated, including: Determine the reward function according to the knowledge vector corresponding to the strategy parameters in the knowledge vector library; Calculate the expected reward values of multiple scenario solutions through the reward function; wherein the multiple scenario solutions are determined according to different strategy parameters; The scenario solution corresponding to the maximum expected reward value is used as the scenario control solution.
4. The control method of the scene generation scheme according to claim 3, characterized in that: Get multiple scenarios as follows: Calculate the objective function of the update strategy; Update the policy parameters according to the gradient ascent method to obtain new policy parameters to maximize the objective function; Get a new scenario plan based on the new strategy parameters until the current update state meets the stopping criteria.
5. The control method of the scene generation scheme according to claim 4, characterized in that: Calculate the objective function of the update strategy, including calculating the objective function according to the following formula: L(θ')=E t [π(a t |s t ;θ') / π(a t |s t ;θ)×A t (i)] Among them, L(θ') is the objective function, E t is the expectation at time step t, θ is the initial policy parameter, θ' is the new policy parameter, π(a t |s t ;θ) is the initial solution, π(a t |s t ; θ') is the new solution, A t (θ) is the advantage function obtained according to the initial strategy parameter θ at time step t.
6. The control method of the scene generation scheme according to claim 4, characterized in that: Update the policy parameters in a gradient ascent manner to obtain new policy parameters, including obtaining new policy parameters according to the following formula: Among them, θ' is the new strategy parameter, θ is the initial strategy parameter, is the gradient of the objective function with respect to θ, and α is the learning rate.
7. The control method of the scene generation scheme according to claim 6, characterized in that: Update the policy parameters in a gradient ascent manner to obtain new policy parameters, including: Determine the Kulibeck-Leibler KL divergence of the new strategy parameters with the previous strategy parameters; Control the KL divergence between the new strategy parameters and the previous strategy parameters to be lower than or equal to the preset KL divergence threshold.
8. The control method of the scene generation scheme according to any one of claims 1 to 7, characterized in that: Select the policy parameters for the scenario corresponding to the user's input information, including: Classify the user's input information and obtain classification results; Select the strategy parameters for the scenario corresponding to the classification result.
9. A control device for a scene generation scheme, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the control method of the scene generation solution according to any one of claims 1 to 8 when running the program instructions.
10. An electronic device, characterized in that: include: Electronic device body; The control device of the scene generation scheme as described in claim 9 is installed on the electronic device body.
Citation Information
Patent Citations
Method for selecting reward function in adversarial imitation learning
CN111401556A
Scene graph skeleton construction method under reinforcement learning framework
CN116524316A
Navigation decision-making method combining curiosity mechanism and self-imitation learning
CN116892932A
Automatic driving test scene generation method and system, and storage medium
CN116894395A
Robot agent skill learning and generalization method based on case reasoning
CN116992964A
