Parameter determination method and device, equipment and storage medium

Abstract

The present application discloses a method, apparatus, device, and storage medium for determining parameters, belonging to the field of artificial intelligence. In an embodiment of the present application, on the one hand, a virtual environment is generated for the real environment to simulate the real environment with the virtual environment and generate simulated environment data, and the determination parameters are updated based on the simulated environment data, avoiding the number of data interactions and the amount of data with hardware devices such as sensors and controllers, thereby greatly improving the determination efficiency of the determination parameters. On the other hand, the real environment is simulated by combining at least two virtual environments. When executing an action control instruction through the virtual environment subsequently, simulated environment data is obtained by randomly selecting a virtual environment, which can capture the uncertainty in the real environment, improve the expression ability of the virtual environment, reduce the difference between the virtual environment and the real environment, make the virtual environment more conform to the real environment, and further improve the accuracy of the determined determination parameters.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, device and storage medium for determining parameters. Background Technology

[0002] Reinforcement learning is a field within machine learning. It emphasizes how to act based on the environment to maximize expected benefits. In recent years, reinforcement learning techniques have been widely applied across various fields.

[0003] Currently, the method for determining the parameters of motion control commands typically involves real-time acquisition of real-world environmental data via sensors in a real-world scenario. The controller then processes this data based on the parameters for determining the motion control command to determine the next motion control command. This process is repeated, with real-time acquisition of real-world environmental data used to update the parameters for determining the next action control command, and so on, until the target parameters are finally determined.

[0004] In this approach, the current learning and parameter determination device needs to interact extensively with hardware devices such as sensors and controllers, requiring a lot of hardware manipulation, which causes significant wear and tear and results in high costs. Furthermore, because these hardware devices are inefficient in various working scenarios, the reinforcement learning process is limited by their efficiency, leading to low efficiency in determining the parameters for action control commands. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for determining parameters, which can improve the efficiency and accuracy of parameter determination. The technical solution is as follows:

[0006] On the one hand, a method for determining parameters of a motion control command is provided, the method comprising:

[0007] Obtain real-world environmental data from a real environment;

[0008] Based on at least two sets of data from the real environment data, generate at least two virtual environments corresponding to the real environment.

[0009] From the at least two virtual environments, a target virtual environment is randomly selected to execute the first action control command, thereby obtaining simulated environment data;

[0010] Based on the simulated environment data, the determination parameters of the motion control command are updated to determine the second motion control command.

[0011] Continuing based on the second action control instruction, a target virtual environment is randomly selected to execute the second action control instruction and update the determined parameters until the first target condition is met, thus obtaining the target determined parameters.

[0012] In some embodiments, the first objective condition is that the expected value converges; or the first objective condition is that the number of iterations reaches a target number.

[0013] On the one hand, a device for determining parameters of motion control commands is provided, the device comprising:

[0014] The acquisition module is used to acquire real-world environmental data.

[0015] The generation module is used to generate at least two virtual environments corresponding to the real environment based on at least two sets of data in the real environment data.

[0016] The acquisition module is further configured to randomly select a target virtual environment from the at least two virtual environments to execute a first action control command and obtain simulated environment data;

[0017] The update module is used to update the determination parameters of the motion control command based on the simulated environment data, and determine the second motion control command.

[0018] The acquisition module and the update module are further configured to continue to randomly select a target virtual environment based on the second action control instruction, execute the second action control instruction and update the determined parameters until the first target condition is met, thereby obtaining the target determined parameters.

[0019] In some embodiments, the generation module is used for:

[0020] The real-world environmental data is grouped to obtain at least two sets of data;

[0021] Based on the at least two sets of data, at least two neural networks are trained to obtain at least two virtual environments corresponding to the real environment, wherein one virtual environment is obtained by training one neural network based on one set of data.

[0022] In some embodiments, the generation module is used for:

[0023] The real environment data is randomly sampled at least twice to obtain at least two sets of data.

[0024] In some embodiments, the real-world environment data includes the environmental state of the target and the action control commands executed by the target;

[0025] In some embodiments, the generation module is used for:

[0026] The at least two sets of data are respectively input into the at least two neural networks;

[0027] For one of the at least two neural networks, the neural network determines the candidate environmental state of the target based on any environmental state in which the target is located and the action control instructions executed by the target in the environmental state;

[0028] Based on the candidate environment state and the target environment state in the real environment data, the error of the candidate environment state is obtained;

[0029] Based on the error, the network parameters of the neural network are adjusted until the second target condition is met.

[0030] In some embodiments, the acquisition module is used to:

[0031] Obtain the first action control command;

[0032] From the at least two virtual environments, a virtual environment is randomly selected as the target virtual environment;

[0033] Based on the environmental state of the target in the target virtual environment and the first action control command, the next environmental state of the target is obtained, and the next environmental state of the target is used as the candidate environmental data.

[0034] In some embodiments, the target virtual environment is implemented based on a neural network;

[0035] The acquisition module is used to input the first action control command into the target neural network, and the target neural network determines the environmental state change information of the target based on the environmental state of the target and the first action control command, and outputs the next environmental state of the target based on the environmental state change information.

[0036] In some embodiments, the update module is used to:

[0037] The simulated environment data is filtered based on the target filtering rules;

[0038] Based on the filtered simulated environment data, the determination parameters of the motion control command are updated.

[0039] In some embodiments, the update module is used to:

[0040] The simulated environment data generated during the execution of the first action control command in the target virtual environment is written into the cache in batches.

[0041] In response to the data volume in the cache reaching a threshold, the simulated environment data in the cache is filtered based on the target filtering rules;

[0042] Clear the cache and continue executing the batch writing and filtering steps until the third target condition is met, then stop to obtain the filtered simulation environment data.

[0043] In some embodiments, the update module is configured to perform any of the following:

[0044] Based on the simulated environment data in the cache, a target threshold is determined; data in the simulated environment data that is higher than the target threshold is removed to obtain filtered simulated environment data.

[0045] The simulated environment data in the cache is analyzed, and behavioral data related to the target event is removed to obtain the filtered simulated environment data.

[0046] In some embodiments, the update module is used to:

[0047] Based on the simulated environment data and the first action control command, the expected value is obtained;

[0048] Obtain the gradient of the expected value;

[0049] Based on the gradient, the determination parameters of the motion control command are updated;

[0050] Based on the updated determined parameters, the simulated environment data, and the first action control command, a second action control command is determined.

[0051] In some embodiments, the acquisition module is used to collect the environmental state of the target in a real environment and the action control commands executed by the target in the environmental state based on sensors, so as to obtain the real environment data.

[0052] In some embodiments, the apparatus further includes:

[0053] The determination module is used to determine the target environment state after the target executes the second action control command in the real environment, based on the real environment data and the second action control command;

[0054] The update module is further configured to update the target virtual environment based on the target environment state and the simulated environment data obtained by executing the second action control command in the target virtual environment.

[0055] On one hand, an electronic device is provided, the electronic device including one or more processors and one or more memories, the one or more memories storing at least one computer program, the at least one computer program being loaded and executed by the one or more processors to implement various optional implementations of the method for determining the determination parameters of the above-mentioned motion control instructions.

[0056] On the one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the storage medium, and the at least one computer program is loaded and executed by a processor to implement various optional implementations of the method for determining the determination parameters of the above-mentioned action control instructions.

[0057] In one aspect, a computer program product or computer program is provided, the computer program product or computer program comprising one or more lines of program code stored in a computer-readable storage medium. One or more processors of an electronic device are capable of reading the one or more lines of program code from the computer-readable storage medium, and the one or more processors execute the one or more lines of program code, enabling the electronic device to execute the method for determining the determination parameters of motion control instructions according to any of the above possible embodiments.

[0058] In this embodiment, on the one hand, a virtual environment is generated for the real environment to simulate the real environment, and simulated environment data is generated. Parameters are updated based on this simulated environment data, avoiding the number of data interactions and the amount of data exchanged with hardware devices such as sensors and controllers, thereby significantly improving the efficiency of parameter determination. On the other hand, by combining at least two virtual environments to simulate the real environment, when executing action control commands through the virtual environment, simulated environment data is obtained by randomly selecting one virtual environment. This captures uncertainties in the real environment, improves the expressive power of the virtual environment, reduces the difference between the virtual and real environments, and makes the virtual environment more closely resemble the real environment, thus improving the accuracy of the determined parameters. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram of the implementation environment of a method for determining parameters of a motion control command provided in an embodiment of this application;

[0061] Figure 2This is a flowchart of a method for determining parameters of a motion control command provided in an embodiment of this application;

[0062] Figure 3 This is a flowchart of a method for determining parameters of a motion control command provided in an embodiment of this application;

[0063] Figure 4 This is a schematic diagram of the structure of a neuron model provided in an embodiment of this application;

[0064] Figure 5 This is a schematic diagram of the basic structure of a neural network provided in an embodiment of this application;

[0065] Figure 6 This is a schematic diagram of a simulation environment module provided in an embodiment of this application;

[0066] Figure 7 This is a schematic diagram of a data filtering module provided in an embodiment of this application;

[0067] Figure 8 This is a schematic diagram of a strategy optimization module provided in an embodiment of this application;

[0068] Figure 9 This is a schematic diagram of the overall architecture of a method for determining the parameters of a motion control command provided in an embodiment of this application;

[0069] Figure 10 This is a schematic diagram of the structure of a device for determining parameters of an action control command provided in an embodiment of this application;

[0070] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0071] Figure 12 This is a structural block diagram of a terminal provided in an embodiment of this application;

[0072] Figure 13 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0074] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have substantially the same function and purpose. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the scope of various examples, a first image can be referred to as a second image, and similarly, a second image can be referred to as a first image. Both the first image and the second image can be images, and in some cases, they can be separate and distinct images.

[0075] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple data packets means two or more data packets.

[0076] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing the particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0077] It should also be understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. The term "and / or" describes an association between related targets, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this application generally indicates that the preceding and following related targets are in an "or" relationship.

[0078] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0079] It should also be understood that determining B based on A does not mean determining B solely based on A; it is also possible to determine B based on A and / or other information.

[0080] It should also be understood that the term “comprising” (also referred to as “includes”, “including”, “comprises” and / or “comprising”) as used in this specification specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0081] It should also be understood that the term "if" can be interpreted as meaning "when" or "upon" or "in response to determination" or "in response to detection." Similarly, depending on the context, the phrases "if determination..." or "if detection [the stated condition or event]" can be interpreted as meaning "when determination..." or "in response to determination..." or "when detection [the stated condition or event]" or "in response to detection [the stated condition or event]."

[0082] The following is an explanation of the terms used in this application.

[0083] The method provided in this application involves artificial intelligence, and the relevant technologies of artificial intelligence are described below.

[0084] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0085] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0086] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and further processes images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0087] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0088] Autonomous driving technology typically includes high-precision mapping, environmental perception, behavior decision-making, path planning, and motion control. Autonomous driving technology has broad application prospects.

[0089] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0090] Furthermore, the method provided in this application relates to reinforcement learning, a field within machine learning that emphasizes how to act based on the environment to maximize expected benefits. The difference between reinforcement learning and standard supervised learning is that it does not require correct input / output pairs or precise correction of suboptimal behavior. Reinforcement learning focuses more on online planning, requiring a balance between exploration (in unknown domains) and conformity (existing knowledge). It is inspired by behaviorist theory in psychology, which describes how organisms, under the stimulation of rewards or punishments from the environment, gradually form expectations of stimuli, producing habitual behaviors that maximize benefits.

[0091] Reinforcement learning is universally applicable and has been applied in many fields, such as game theory, cybernetics, operations research, information theory, simulation optimization, multi-agent system learning, swarm intelligence, statistics, and genetic algorithms. In the context of operations research and control theory, reinforcement learning is called "approximate dynamic programming (ADP)." This problem is also studied in optimal control theory, although most research focuses on the existence and characteristics of optimal solutions rather than learning or approximation. In economics and game theory, reinforcement learning is used to explain how equilibrium arises under conditions of bounded rationality.

[0092] In the method for determining parameters for motion control commands provided in this application embodiment, the motion control commands can be used to instruct the behavior control of a target. The determined parameters can be used to determine motion control commands based on the environment. This process of determining motion control commands based on the environment can be understood as an intelligent decision-making process, and the determined parameters can also be called intelligent decision-making strategies. This method can be applied to multiple projects and product applications, including intelligent robot automatic control, autonomous driving of automobiles, and IoT resource optimization. It enables various types of devices to learn intelligent decision-making strategies at a controllable cost, automatically judging and executing reasonable and safe operations when facing different states, achieving the effect of unmanned automatic control.

[0093] The solutions provided in this application involve technologies such as computer vision technology in artificial intelligence, reinforcement learning in machine learning, and autonomous driving, which are specifically illustrated through the following embodiments.

[0094] The implementation environment of this application is described below.

[0095] Figure 1This is a schematic diagram of the implementation environment for a method for determining parameters of a motion control command according to an embodiment of this application. The implementation environment includes a terminal 101, or it includes a terminal 101 and a platform 102 for determining parameters of the motion control command. The terminal 101 is connected to the platform 102 for determining parameters of the motion control command via a wireless network or a wired network.

[0096] Terminal 101 can be at least one of the following: smartphone, game console, desktop computer, tablet computer, e-book reader, MP3 (Moving Picture Experts Group Audio Layer III) player or MP4 (Moving Picture Experts Group Audio Layer IV) player, laptop computer, or automated control device. This automated control device can be an intelligent robot, self-service payment device, autonomous vehicle, etc., or other automated control devices such as smart home devices.

[0097] Exemplarily, the terminal 101 can complete this task independently, or it can be provided with data services by the motion control command determination parameter determination platform 102. This application embodiment does not limit this aspect. Exemplarily, the motion control command determination parameter determination platform 102 can complete this task independently. The motion control command determination parameter determination platform 102 can collect real-world environmental data through the terminal 101, thereby independently completing the parameter determination process.

[0098] The action control command parameter determination platform 102 includes at least one of a server, multiple servers, a cloud computing platform, and a virtualization center. The action control command parameter determination platform 102 provides background services for applications that support the determination of action control command parameters. Optionally, the action control command parameter determination platform 102 undertakes the primary processing work, and the terminal 101 undertakes the secondary processing work; or, the action control command parameter determination platform 102 undertakes the secondary processing work, and the terminal 101 undertakes the primary processing work; or, the action control command parameter determination platform 102 or the terminal 101 can each independently undertake processing work. Alternatively, the action control command parameter determination platform 102 and the terminal 101 collaborate using a distributed computing architecture.

[0099] Optionally, the parameter determination platform 102 for determining the motion control command includes at least one server 1021 and a database 1022. The database 1022 is used to store data. In this embodiment, the database 1022 can store sample data, which can be real environment data, to provide data services to at least one server 1021.

[0100] Servers can be standalone physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminals can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, etc., but are not limited to these.

[0101] Those skilled in the art will understand that the number of terminals 101 and servers 1021 can be more or less. For example, there may be only one terminal 101 or server 1021, or there may be dozens or hundreds of terminals 101 or servers 1021, or even more. The embodiments of this application do not limit the number or type of terminals or servers.

[0102] Figure 2 This is a flowchart illustrating a method for determining parameters of an action control command, as provided in an embodiment of this application. This method is applied to an electronic device, which may be a terminal or a server. (See also...) Figure 2 The method includes the following steps.

[0103] 201. Electronic devices acquire real-world environmental data from the real environment.

[0104] The environment refers to the space where the target is located and the various natural factors within it that can directly or indirectly affect the target's behavior. The real environment refers to the environment that actually exists in the real world.

[0105] In some embodiments, the real-world environment data may include the environmental state of the target and the action control commands executed by the target. The target refers to the controlled device, which may be an automated control device, such as an intelligent robot or an autonomous vehicle. For example, the target may be the electronic device or other controlled electronic devices.

[0106] Motion control commands are instructions that control the behavior of a target. For example, controlling the target to move forward at its designated speed; controlling the target to turn left; or controlling the target to raise its robotic arm.

[0107] The motion control command may differ in different application scenarios. This command can be set by relevant technical personnel according to requirements, and this application embodiment does not limit its implementation.

[0108] 202. The electronic device generates at least two virtual environments corresponding to the real environment based on at least two sets of data from the real environment data.

[0109] A virtual environment refers to a virtual environment displayed (or provided) by an application while it is running on a device, used to simulate a real environment. This virtual environment can be a simulation of the real world, a semi-simulated / semi-fictional virtual environment, or a purely fictional virtual environment.

[0110] Electronic devices generate multiple virtual environments by dividing real-world environmental data into multiple sets of data. These multiple virtual environments are then combined to simulate the real-world environment. Since these multiple virtual environments are generated based on multiple sets of different data, they may focus on different parts of the real-world environment and be able to capture the uncertainties in the real-world environment.

[0111] 203. The electronic device randomly selects a target virtual environment from the at least two virtual environments to execute the first action control command and obtain simulated environment data.

[0112] The simulated environment data may include the environmental state of the virtual target within the virtual environment. In some embodiments, the simulated environment data may also include the action control commands executed by the virtual target within the virtual environment. Different virtual environments may simulate the real environment from different perspectives, and the same action control command may produce different feedback, thus reflecting the uncertainty of the real environment, adding randomness to the results, and obtaining more realistic simulated environment data.

[0113] 204. Based on the simulated environment data, the electronic device updates the determination parameters of the motion control command and determines the second motion control command.

[0114] These parameters can be used to determine action control commands based on the environment. Electronic devices can process simulated environment data using these parameters to determine the next action control command to be executed. This process of determining action control commands based on the environment can be understood as an intelligent decision-making process, and these parameters can also be called intelligent decision-making strategies. For example, when an electronic device acquires simulated environment data, it processes the data using these parameters to determine the action control command that the target needs to execute in the current environmental state. The role of these parameters is to determine the next action control command. If the process of processing simulated environment data to determine action control commands is understood as being implemented through a decision model, then these parameters are the model parameters of the decision model.

[0115] By using the simulated environment data described above, electronic devices can learn more adaptive, realistic, and accurate decision-making parameters (intelligent decision-making strategies). These learned parameters enable them to make more accurate decisions based on data from real-world environments.

[0116] 205. The electronic device continues to randomly select a target virtual environment based on the second action control instruction, execute the second action control instruction and update the determined parameters until the first target condition is met, and obtain the target determined parameters.

[0117] The process of updating parameters based on each action control command can be considered an iterative process. The electronic device can repeat the iterative steps based on the next action control command determined at the current time. Through multiple iterations, it can learn a more accurate decision-making strategy based on the feedback of each action control command in the virtual environment, so as to respond efficiently and accurately to various states and situations in the real environment.

[0118] In this embodiment, on the one hand, a virtual environment is generated for the real environment to simulate the real environment, and simulated environment data is generated. Parameters are updated based on this simulated environment data, avoiding the number of data interactions and the amount of data exchanged with hardware devices such as sensors and controllers, thereby significantly improving the efficiency of parameter determination. On the other hand, by combining at least two virtual environments to simulate the real environment, when executing action control commands through the virtual environment, simulated environment data is obtained by randomly selecting one virtual environment. This captures uncertainties in the real environment, improves the expressive power of the virtual environment, reduces the difference between the virtual and real environments, and makes the virtual environment more closely resemble the real environment, thus improving the accuracy of the determined parameters.

[0119] Figure 3 This is a flowchart of a method for determining parameters of a motion control command according to an embodiment of this application. See also... Figure 3 The method includes the following steps.

[0120] 301. Electronic devices acquire real-world environmental data from the real environment.

[0121] In the embodiments of this application, the electronic device can generate a corresponding virtual environment for the real environment. This virtual environment is used to simulate the real environment. In this way, when making a certain action control command based on the real environment, it can be determined based on the virtual environment. This reduces the dependence on the real environment, eliminates the need for a large amount of data interaction with sensors, and effectively improves the efficiency of determining the parameters.

[0122] This real-world environmental data can be obtained in various ways. In some embodiments, it can be acquired based on sensor data. Specifically, an electronic device can acquire the environmental state of a target in a real-world environment, as well as the action control commands executed by the target in that environmental state, based on sensors, to obtain the real-world environmental data.

[0123] The sensor can be a data acquisition device outside the electronic device or a component on the electronic device; this application embodiment does not limit this.

[0124] For example, this method can be applied to the automatic control of intelligent robots / robots, by manipulating the robot to perform actions in a real environment, and then collecting data on the real environment based on sensors to obtain real environment data.

[0125] In other embodiments, the real-world environmental data is data that has been pre-collected and stored in a database. Accordingly, in step 301, the electronic device can obtain the real-world environmental data from the database.

[0126] It should be noted that the embodiments in this application do not limit this.

[0127] 302. The electronic device groups the real-world environmental data to obtain at least two sets of data.

[0128] After acquiring real-world environmental data, electronic devices can use this data as a representation of the real environment to generate a corresponding virtual environment, thus creating a virtual environment that closely matches the real environment.

[0129] In this embodiment, a real environment is simulated by combining at least two virtual environments. "At least two virtual environments" refers to two or more virtual environments, i.e., multiple virtual environments. These at least two virtual environments can be generated from different real environment data, thus creating distinct virtual environments that can simulate the real environment using different data.

[0130] In some embodiments, the electronic device performs at least two random samplings of the real-world environment data to obtain at least two sets of data. By using random sampling, the grouping process becomes random and not based on fixed grouping rules. This results in stronger randomness in the at least two sets of data obtained, making it easier for the generated virtual environment to capture the uncertainties of the real environment, reducing the difference between the virtual and real environments, and improving the realism of the virtual environment.

[0131] In other embodiments, the grouping process can be implemented based on grouping rules. Specifically, the electronic device groups the real-world environmental data based on a target grouping rule to obtain at least two groups of data. The target grouping rule can be set by relevant technical personnel according to requirements. For example, the target grouping rule can be to evenly distribute the data into at least two groups based on the amount of data, or to limit the amount of data in each group, etc. The embodiments of this application do not limit this.

[0132] For at least two sets of data obtained by grouping, each set of data can also be called a dataset. When an electronic device groups real-world environmental data, it obtains N sets of data, which means that the electronic device groups real-world environmental data and obtains N datasets.

[0133] 303. The electronic device trains at least two neural networks based on the at least two sets of data to obtain at least two virtual environments corresponding to the real environment, wherein one virtual environment is obtained by training one neural network based on a set of data.

[0134] Electronic devices can generate virtual environments using neural networks. These virtual environments simulate real-world environments and are also called simulated environments. This virtual environment can be the trained neural network, also known as a simulator, used to simulate real-world environments. The trained neural network, through training with real-world data, gains the ability to process similar or identical data, thus simulating real-world environments.

[0135] In some embodiments, the process of training a neural network by an electronic device can be achieved through the following steps one through four.

[0136] Step 1: The electronic device can input the at least two sets of data into the at least two neural networks respectively.

[0137] The basic building block of a neural network is the neuron model, which consists of multiple neurons. For example... Figure 4 As shown, each neuron receives an input signal (or input data), performs a weighted sum of the received signals or data, and then outputs the result after processing by an activation function. For a single neuron, let the neuron's input vector be... The weight vector is The output of the neuron is obtained by a nonlinear activation function after linear weighted summation, as shown in Formula 1 below.

[0138] Formula 1

[0139] in, This serves as an identifier for the input data in the input vector or the weights in the weight vector. This is the output of the neuron. In this embodiment, the input vector can be real-world environmental data, and the output can be a virtual environment. These are the weights of the neurons in this layer.

[0140] The basic structure of a neural network is as follows: Figure 5 As shown, a neural network is composed of multiple layers of neurons. In its basic structure, the neural network is divided into three parts: the first part is the input layer 501, the second part is the hidden layer 502, and the third part is the output layer 503. Figure 5 This embodiment only uses a single neuron per layer as an example; however, each layer can also contain multiple neurons, and this application does not limit this. Input data is passed from the input layer to the hidden layer, processed layer by layer by neurons, and finally output by the output layer. Activation functions are set between layers to increase non-linear expressive power. The process by which the neural network processes the input data and outputs the result is called forward propagation. In this embodiment, when training the neural network, real-world data can be used as input data and input to the input layer 501 of the neural network. The input layer 501 processes the real-world data and passes it to the hidden layer 502. Each hidden layer 502 can also further process the data after receiving the data output from the input layer and pass it to the output layer 503 for output. Finally, the output layer 503 outputs the result.

[0141] In addition to the forward propagation mentioned above, the training process also involves backpropagation. The electronic device can compare the data output by the output layer 503 of the neural network with the real-world data to determine the accuracy of the data determined by the neural network. This allows for updating the network parameters. Through continuous updates, the network parameters are optimized, enabling the processing of input data to produce data that mirrors the environmental state changes in the real environment. This trained neural network can be referred to as a virtual environment of the real environment, or a simulator. When a determined first action control command is subsequently input into the neural network, it can determine, based on the trained network parameters, how the environmental state changes after executing the first action control command in the current virtual environment, thus obtaining simulated environmental data.

[0142] like Figure 6As shown, in a specific example, the generation process of the virtual environment can be implemented by the simulation environment module 600. After obtaining the real environment data 601, the real environment data 601 can be input into the simulation environment module 600. The simulation environment module 600 can divide the collected real environment data 601 into N datasets 602. This data division process is also the grouping process in step 302 above. The N datasets 602 are at least two groups of data in step 302, but again, N groups are used to indicate at least two groups, with each dataset 602 being a group of data obtained from the grouping. Then, N neural networks 603 are trained using the N datasets 602 respectively to obtain N simulators 604. These N simulators are the N simulation environments 605 (virtual environments). Based on a dataset 602, one neural network 603 can be trained to obtain one simulator 604. The process of training N simulators 604 is the same as the process in step 303 above of training at least two neural networks based on at least two sets of data. In this process, the N simulators 604 are trained on different datasets. This allows for the random selection of one simulator from the N simulators 604 each time a simulator is needed, increasing the uncertainty in acquiring simulated environment data. This effectively captures the uncertainty in the real environment. Combining the N simulators 604 to simulate the real environment results in a more realistic simulation. When the N simulated environments 605 are used, one can be randomly selected from the N simulated environments 605 for processing based on the input command 606, and feedback 607 will be output. Taking the input command 606 as the first action control command as an example, randomly selecting one simulated environment 605 from the N simulated environments 605 and executing the first action control command in the first simulated environment 605 yields simulated environment data. This simulated environment data then indicates the feedback of the simulated environment 605 to the input command 606.

[0143] In this first step, the electronic device inputs real-world environmental data into the neural network. The neural network processes this real-world environmental data and outputs the result.

[0144] Step 2: For one of the at least two neural networks, the electronic device determines the candidate environmental state of the target based on the target's environmental state and the action control command executed by the target in that environmental state.

[0145] The process of each neural network processing each set of data is similar; here, we will only use one neural network processing one set of data as an example. After an electronic device inputs a set of data into a neural network, the neural network can process the input set of data through the aforementioned forward propagation process. For each layer of the neural network, each neuron can perform weighted processing on the input data and obtain the sum of the weighted processing result and the weights of that layer. This weighted processing result and the sum of the weights are used as the output data and input into the next layer of neurons. The next layer of neurons can then perform similar processing based on the input data. The data output by the neurons of the last layer is the output of the neural network.

[0146] In the above processing, the data input to the neural network can include any environmental state of the target and the action control command executed by the target in that environmental state. After the weighted processing of each neuron layer is performed and the weights are summed, the neural network can output a final result, which is a candidate environmental state. That is, the environmental state after the target executes the action control command in the current environmental state. In some embodiments, the neural network can also output a candidate action control command that the target will execute in the candidate environmental state.

[0147] The training method for neural networks is the gradient backpropagation algorithm. Backpropagation is a training method based on model fitting error, which effectively saves computational resources. Its main steps are: after obtaining the output via forward propagation, a custom loss function is used to calculate the error between the model output and the true value. Then, chain rule differentiation is used to calculate the gradient of this error with respect to each neuron, layer by layer from the output layer to the hidden layer and then to the input layer. The gradient is used to update the network parameters of the neurons, which can be the neuron weights. This iterative process of adjusting neuron weights continues until convergence, making the model output as close as possible to the actual value, thus achieving the goal of training the neural network. This backpropagation algorithm corresponds to steps three and four below.

[0148] Step 3: The electronic device can obtain the error of the candidate environment state based on the candidate environment state and the target environment state in the real environment data.

[0149] After obtaining the output from the neural network, the result can be compared with the true value to determine the error. The neural network output is the predicted value obtained by processing the input, which is the candidate environment state, while the true value is the target environment state in the aforementioned real environment data. The error of this candidate environment state reflects its accuracy.

[0150] Step 4: Based on this error, the electronic device adjusts the network parameters of the neural network until the second target condition is met.

[0151] After an electronic device receives an error, the purpose of training a neural network based on that error is to improve the accuracy of the predicted values ​​and enhance the processing power of the neural network. By adjusting the network parameters, the neural network can process different output data based on the same input data, making the output data closer to the true value. In this embodiment, this means making the output candidate environment state closer to the target environment state in the real environment data.

[0152] In some embodiments, the electronic device may obtain the gradient based on the error, adjust the network parameters of the neural network based on the gradient, and then repeat steps one to four above based on the adjusted network parameters until the second target condition is met.

[0153] The second objective condition can be error convergence or the number of iterations reaching a target number, which can be set by relevant technical personnel according to their needs. This application embodiment does not limit this.

[0154] For example, taking a three-layer fully connected neural network as an example, the update process of neuron parameters (i.e., the network parameters of the neural network) can be achieved using Equations 2 and 3 below. Equation 2 calculates the gradient, and then Equation 3 updates the network parameters. This method of updating network parameters can also be called the chain rule.

[0155] Formula 2

[0156] Formula 3

[0157] in, represents the weight of each neuron in the neural network. Let be the loss function, which is the error mentioned above. The first layer of the output layer of the neural network The output of each neuron. The parameter used to control the learning speed is a hyperparameter.

[0158] Steps 302 and 303 involve generating at least two virtual environments corresponding to the real environment based on at least two sets of data from the real environment data. In this process, the virtual environments are implemented using a neural network. The above description uses the neural network as a neuron model, with each neuron model being a fully connected network. The neural network is a combination of multilayer perceptrons. Other types of neural networks can also be used. In some embodiments, the neural network can be a convolutional neural network (CNN), which is suitable for image processing-focused problems. In other embodiments, the neural network can be a recurrent neural network (RNN), which is suitable for time-series processing problems. In still other embodiments, the neural network can be a Bayesian neural network (BNN), which is used to more effectively simulate real environments with high levels of interference. This application does not limit the specific neural network structure used.

[0159] 304. The electronic device acquires the first action control command.

[0160] After obtaining a virtual environment, electronic devices can train and determine parameters based on the virtual environment. Then, based on the determined parameters, they can determine the changes in the virtual environment or the changes in the target's state when executing a certain action control command in the virtual environment. In turn, they can determine which action control command to execute in the changed environmental state of the target.

[0161] The first motion control command can be a motion control command from real scene data. The electronic device determines candidate environment data in the virtual environment based on the motion control command, rather than in the real environment. This can reduce data interaction with sensors and improve efficiency.

[0162] 305. The electronic device randomly selects one of the at least two virtual environments as the target virtual environment.

[0163] When determining candidate environment data, electronic devices can randomly select one from multiple virtual environments to execute the first action control command. This random selection method can increase the randomness and uncertainty of the determined candidate environment data, which can better capture the uncertainty of the real environment, reduce the difference between the virtual environment and the real environment, improve the expressive power of the neural network, and avoid the neural network overfitting the input data.

[0164] like Figure 6As shown, the simulation environment module 600 uses a multi-neural network integration method. Multiple different networks are trained by sampling data to construct a combined simulation environment 605 composed of multiple simulators 604. In the combined simulation environment 605, one simulator is randomly selected to execute the control commands input by the controller, i.e., the input commands 606, and the generated simulation data (i.e., simulation environment data) is used as output feedback 607. The complex simulation environment generated by the simulation environment module 600 can reduce the difference between the simulator and the real environment, improve security, and significantly improve actual deployment performance.

[0165] It should be noted that this explanation only illustrates the processing of a single action control command (the first action control command). The same processing procedure can be performed whenever an action control command is obtained, and each time a virtual environment is randomly selected as the target virtual environment for action control.

[0166] The virtual environment can be the neural network trained above. Accordingly, in step 305, the electronic device randomly selects a target neural network from multiple trained neural networks.

[0167] 306. The electronic device obtains the next environmental state of the target based on the environmental state of the target in the target virtual environment and the first action control command, and uses the next environmental state of the target as candidate environmental data.

[0168] Step 306 is similar to step two in step 303 above, and can be the process of the neural network performing forward propagation on the input data. It will not be elaborated on here.

[0169] Steps 304 to 306 above involve randomly selecting a target virtual environment from the at least two virtual environments to execute a first action control command and obtain simulated environment data. The virtual environment can be the trained neural network described above. Accordingly, in step 306, the electronic device can input the first action control command into the target neural network. The target neural network then determines the environmental state change information of the target based on the environmental state of the target and the first action control command, and outputs the next environmental state of the target based on this environmental state change information.

[0170] 307. Based on the simulated environment data, the electronic device updates the determination parameters of the motion control command and determines the second motion control command.

[0171] After obtaining the simulated environment data, the electronic device can update the parameters for determining the action control command based on the simulated environment data. After the update, if the first target condition is met, the training can end; if the first target condition is not met, a second action control command can be determined, and steps 304 to 307 above can be repeated to update the parameters again. The first target condition is the condition for ending the parameter update.

[0172] It should be noted that this determined parameter is used to determine the action control command. In simple terms, this determined parameter can be used to decide the next action based on the environment, and therefore can be called an intelligent decision-making strategy. Step 307 is also the step of optimizing and updating the intelligent decision-making strategy based on simulated environment data. Through optimization and updating, the intelligent decision-making strategy can make more accurate action control commands that are more in line with the real environment.

[0173] In some embodiments, before determining parameter updates, the electronic device can first filter the simulated environment data, and then perform the parameter update step based on the filtered data. Specifically, the electronic device can filter the simulated environment data based on target filtering rules, and then update the parameters of the action control command based on the filtered simulated environment data. By adding a data filtering step, the problem of reinforcement learning algorithms easily converging to local optima can be effectively avoided, improving the adaptability to complex data and enhancing the generalization performance of reinforcement learning algorithms on simulators.

[0174] In some embodiments, the electronic device can write simulated environment data generated during the execution of the first action control command in the target virtual environment into a cache in batches. In response to the data volume in the cache reaching a threshold, the simulated environment data in the cache is filtered based on the target filtering rules. The cache is then cleared, and the batch writing and filtering steps continue until a third target condition is met, at which point the filtered simulated environment data is obtained. The third target condition can be convergence of the deterministic parameters or reaching a target number of iterations. This method, by reasonably caching and filtering training data, actively excludes data with excessively high feedback values ​​in the cache, thereby suppressing the convergence of the decision strategy (deterministic parameters) to a local optimum. This improves the adaptability of the decision strategy to complex scenarios and enhances the generalization performance of reinforcement learning algorithms on simulators.

[0175] In one possible implementation, the data volume in the cache reaching a threshold can be defined as the data volume in the cache reaching the maximum data volume of the cache, that is, the threshold can be the maximum data volume of the cache, and the data volume in the cache reaching the threshold can be considered as filling the cache. Of course, other situations are also possible, for example, the data volume in the cache reaching half of the maximum data volume, and this application embodiment does not limit this.

[0176] Different application scenarios may require different target filtering rules, which can be set by relevant technical personnel according to business needs.

[0177] In some embodiments, the target filtering rule can be a data threshold-based filtering rule. In this case, the electronic device can determine the target threshold based on the simulated environment data in the cache, and remove data in the simulated environment data that is higher than the target threshold to obtain the filtered simulated environment data.

[0178] The target threshold can also be called the cached data threshold. Its determination method can be determined by relevant technical personnel according to business needs. For example, the target threshold can be the average or quantile of each data in the cached data, or the confidence level is the threshold of the target confidence level, etc. The target threshold can also be a threshold preset by relevant technical personnel. This application embodiment does not limit this.

[0179] like Figure 7 As shown, simulated data generated in the simulation environment (i.e., simulation environment data) is filled into the cache in batches. The cache is checked to see if it is full. If not, more data is added to the cache. If it is full, a cache data threshold is calculated, and data is filtered using this threshold before being output. The output data serves as the data for subsequent updates to the intelligent decision-making strategy. In the cache filling step, after filtering the data, it can be determined whether the loop count has been reached. If the loop count has not been reached, the cache is cleared, and the next loop begins, filling in new data. If the loop count has been reached, the cache filling step ends.

[0180] In other embodiments, the electronic device can analyze the cached simulated environment data, removing behavioral data related to the target event to obtain filtered simulated environment data. For example, in an autonomous driving scenario, behavioral data that leads to an accident can be used as the target filtering rule; in a robot control scenario, behavioral data that leads to robot damage can be used as the target filtering rule, and so on.

[0181] In some embodiments, the above data filtering step can be implemented by a data filtering module, and the parameter update determination step, which is also the strategy optimization step, can be implemented by a strategy optimization module.

[0182] like Figure 8As shown, the strategy optimization module processes the simulated data filtered by the data filtering module. Specifically, it calculates the data value, optimizes the decision strategy using the strategy optimization formula, determines a new decision strategy, and then determines whether the strategy converges. If converged, training is complete, and the final strategy (objective parameters) is obtained. If not converged, a second action control command needs to be determined based on the current decision strategy, and the controller then sends this second action control command to the simulation environment for execution.

[0183] When updating the determined parameters, the electronic device can obtain the desired value based on the simulated environment data and the first action control command, then obtain the gradient of the desired value, and update the determined parameters of the action control command based on the gradient. The electronic device can then determine the second action control command based on the updated determined parameters, the simulated environment data, and the first action control command.

[0184] For example, the above-described parameter optimization process can be achieved through a strategy optimization formula (i.e., Figure 8 The strategy optimization formulas in the text are implemented as follows: Formula 4 and Formula 5.

[0185] Formula 4

[0186] Formula 5

[0187] in, This represents the expected value calculated based on simulated environment data. Used to calculate the expected value. This represents the data cache pool in the data filtering module. The reduction factor in the reinforcement learning algorithm is a fixed constant value. express The environmental state at any given time and the motion control commands input into the environment. This represents feedback values ​​from the environment regarding its state and commands, used to indicate changes in environmental state or motion control commands. Parameters for controlling learning speed, express gradient, These are the decision strategy function at the current moment and the decision strategy function after optimization and update, respectively.

[0188] 308. The electronic device continues to randomly select a target virtual environment based on the second action control instruction, execute the second action control instruction and update the determined parameters until the first target condition is met, and obtain the target determined parameters.

[0189] After the electronic device determines the second action control command, it can repeat steps 305 to 307 above, except that the action control command changes from the first action control command to the second action control command. By continuously making decisions on action control commands based on the environmental state, and then updating the determined parameters (intelligent decision-making strategy) accordingly, the determined parameters can be continuously optimized to obtain the target determined parameters.

[0190] In some embodiments, after the electronic device determines the second action control command, it can execute the second action control command not only through the virtual environment, but also send it to the real environment. In this way, the real feedback on the second action control command obtained through the real environment can be used to correct the virtual feedback of the virtual environment on the second action control command, so that the difference between the virtual environment and the real environment becomes smaller and smaller, ensuring that the final target determination parameters are more accurate.

[0191] Specifically, the electronic device can determine the target environment state after the target executes the second action control command in the real environment based on the real environment data and the second action control command, and update the target virtual environment based on the target environment state and the simulated environment data obtained by executing the second action control command in the target virtual environment.

[0192] In some embodiments, the first objective condition is the convergence of the expected value in step 307; or the first objective condition is that the number of iterations reaches a target number. This application does not limit this.

[0193] The following is through Figure 9 This application explains the overall architecture of the determination parameters used to determine motion control commands. For example... Figure 9 As shown, this invention consists of multiple core modules. First, data from the real environment is collected via sensors. A simulation environment module then constructs a near-real environment. The generated simulated data is filtered by a data filtering module to select data more suitable for generalization learning, which is then provided to the policy optimization module. A reinforcement learning algorithm optimizes the decision policy for the current iteration, and a controller manages the policy in both the real and simulated environments. Once the convergence criterion is met, training stops, and the final intelligent decision policy is output.

[0194] Specifically, in this overall architecture, sensors collect data from the real environment to obtain environmental states and action feedback values, which are then sent to a simulation environment module. This module generates a corresponding simulated environment based on the real environment. Through action control commands, it generates simulated data. After the simulated data is filtered by a data filtering module, the strategy optimization module optimizes the decision-making strategy. The controller then issues the next action control command to both the real and simulated environments. For the real environment, this action control command is called an action signal. For the simulated environment, the action control command is called an analog signal. The strategy optimization module can determine whether the strategy has converged in each iteration. If it has converged, the optimal intelligent decision-making strategy has been obtained.

[0195] The technical solution of this application constructs a more complex simulation environment by combining multiple neural networks and adding a data filtering module. This not only utilizes the high efficiency of the simulator but also improves the adaptability to complex data, enhancing the training effect of the reinforcement learning algorithm in the simulation environment. Ultimately, it yields a reinforcement learning algorithm that is not limited by the working efficiency of hardware devices such as sensors and controllers, achieving low cost, high efficiency, and high generalization performance.

[0196] In this embodiment, on the one hand, a virtual environment is generated for the real environment to simulate the real environment, and simulated environment data is generated. Parameters are updated based on this simulated environment data, avoiding the number of data interactions and the amount of data exchanged with hardware devices such as sensors and controllers, thereby significantly improving the efficiency of parameter determination. On the other hand, by combining at least two virtual environments to simulate the real environment, when executing action control commands through the virtual environment, simulated environment data is obtained by randomly selecting one virtual environment. This captures uncertainties in the real environment, improves the expressive power of the virtual environment, reduces the difference between the virtual and real environments, and makes the virtual environment more closely resemble the real environment, thus improving the accuracy of the determined parameters.

[0197] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0198] Figure 10 This is a schematic diagram of a device for determining parameters of an action control command according to an embodiment of this application. See also... Figure 10 The device includes:

[0199] Module 1001 is used to acquire real-world environmental data.

[0200] The generation module 1002 is used to generate at least two virtual environments corresponding to the real environment based on at least two sets of data in the real environment data.

[0201] The acquisition module 1001 is also used to randomly select a target virtual environment from the at least two virtual environments to execute a first action control command and obtain simulated environment data;

[0202] The update module 1003 is used to update the determination parameters of the motion control command based on the simulation environment data, and determine the second motion control command.

[0203] The acquisition module 1001 and the update module 1003 are further configured to continue to randomly select a target virtual environment based on the second action control instruction, execute the second action control instruction and update the determined parameters until the first target condition is met, thereby obtaining the target determined parameters.

[0204] In some embodiments, the generation module 1002 is used for:

[0205] The real-world data is grouped to obtain at least two sets of data;

[0206] Based on the at least two sets of data, at least two neural networks are trained to obtain at least two virtual environments corresponding to the real environment, wherein one virtual environment is obtained by training a neural network based on a set of data.

[0207] In some embodiments, the generation module 1002 is used for:

[0208] At least two random samples were taken from the real-world environmental data to obtain at least two sets of data.

[0209] In some embodiments, the real-world environment data includes the environmental state of the target and the action control commands executed by the target;

[0210] In some embodiments, the generation module 1002 is used for:

[0211] Input the at least two sets of data into the at least two neural networks respectively;

[0212] For one of the at least two neural networks, the neural network determines the candidate environmental state of the target based on any environmental state in which the target is located and the action control command executed by the target in that environmental state;

[0213] Based on the candidate environment state and the target environment state in the real environment data, the error of the candidate environment state is obtained;

[0214] Based on this error, the network parameters of the neural network are adjusted until the second objective condition is met.

[0215] In some embodiments, the acquisition module 1001 is used to:

[0216] Obtain the first action control command;

[0217] From the at least two virtual environments, randomly select one virtual environment as the target virtual environment;

[0218] Based on the environmental state of the target in the target virtual environment and the first action control command, the next environmental state of the target is obtained, and the next environmental state of the target is used as the candidate environmental data.

[0219] In some embodiments, the target virtual environment is implemented based on a neural network;

[0220] The acquisition module 1001 is used to input the first action control command into the target neural network, and the target neural network determines the environmental state change information of the target based on the environmental state of the target and the first action control command, and outputs the next environmental state of the target based on the environmental state change information.

[0221] In some embodiments, the update module 1003 is used to:

[0222] Based on the target selection rules, the simulated environment data is filtered;

[0223] Based on the filtered simulation environment data, the determination parameters of the motion control command are updated.

[0224] In some embodiments, the update module 1003 is used to:

[0225] The simulated environment data generated during the execution of the first action control command in the target virtual environment is written to the cache in batches.

[0226] In response to the data volume in the cache reaching a threshold, the simulated environment data in the cache is filtered based on the target filtering rules;

[0227] Clear the cache and continue executing the batch writing and filtering steps until the third target condition is met, then stop to obtain the filtered simulation environment data.

[0228] In some embodiments, the update module 1003 is configured to perform any of the following:

[0229] Based on the simulated environment data in the cache, a target threshold is determined; data in the simulated environment data that exceeds the target threshold is removed to obtain filtered simulated environment data.

[0230] The simulated environment data in the cache is analyzed, and behavioral data related to the target event is removed to obtain the filtered simulated environment data.

[0231] In some embodiments, the update module 1003 is used to:

[0232] Based on the simulated environment data and the first motion control command, the expected value is obtained;

[0233] Obtain the gradient of the expected value;

[0234] Based on this gradient, the determination parameters of the motion control command are updated;

[0235] Based on the updated parameters, the simulation environment data, and the first action control command, a second action control command is determined.

[0236] In some embodiments, the acquisition module 1001 is used to acquire the environmental state of the target in a real environment and the action control commands executed by the target in that environmental state based on sensors, thereby obtaining the real environment data.

[0237] In some embodiments, the device further includes:

[0238] The determination module is used to determine the target environment state after the target executes the second action control command in the real environment, based on the real environment data and the second action control command;

[0239] The update module 1003 is also used to update the target virtual environment based on the target environment state and the simulated environment data obtained by executing the second action control command in the target virtual environment.

[0240] The apparatus provided in this application, on the one hand, generates a virtual environment for the real environment, uses the virtual environment to simulate the real environment, and generates simulated environment data. Based on this simulated environment data, it updates and determines parameters, avoiding the number of data interactions and the amount of data exchanged with hardware devices such as sensors and controllers, thereby greatly improving the efficiency of parameter determination. On the other hand, by combining at least two virtual environments to simulate the real environment, when executing action control commands through the virtual environment, the simulated environment data is obtained by randomly selecting one virtual environment. This captures the uncertainties in the real environment, improves the expressive power of the virtual environment, reduces the difference between the virtual and real environments, and makes the virtual environment more closely resemble the real environment, thereby improving the accuracy of the determined parameters.

[0241] It should be noted that the motion control command determination parameter determination device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the motion control command determination parameter determination device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the motion control command determination parameter determination device and the motion control command determination parameter determination method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0242] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 1100 can vary significantly due to differences in configuration or performance. It can include one or more Central Processing Units (CPUs) 1101 and one or more memories 1102. The memories 1102 store at least one computer program, which is loaded and executed by the processor 1101 to implement the parameter determination method for determining motion control instructions provided in the various method embodiments described above. The electronic device can also include other components for implementing device functions. For example, the electronic device can also have wired or wireless network interfaces and input / output interfaces for input and output. Further details are not elaborated upon in this embodiment.

[0243] The electronic device in the above method embodiments can be implemented as a terminal. For example, Figure 12 This is a structural block diagram of a terminal provided in an embodiment of this application. The terminal 1200 can be a portable mobile terminal, such as a smartphone, tablet computer, MP3 (Moving Picture Experts Group Audio Layer III) player, MP4 (Moving Picture Experts Group Audio Layer IV) player, laptop computer, or desktop computer. The terminal 1200 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0244] Typically, terminal 1200 includes a processor 1201 and a memory 1202.

[0245] Processor 1201 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1201 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1201 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1201 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1201 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0246] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1202 are used to store at least one instruction, which is executed by the processor 1201 to implement the motion control instruction determination parameter determination method provided in the method embodiments of this application.

[0247] In some embodiments, the terminal 1200 may also optionally include a peripheral device interface 1203 and at least one peripheral device. The processor 1201, memory 1202, and peripheral device interface 1203 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1203 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: radio frequency circuitry 1204, display screen 1205, camera assembly 1206, audio circuitry 1207, and power supply 1209.

[0248] Peripheral device interface 1203 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1201 and memory 1202. In some embodiments, processor 1201, memory 1202 and peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1201, memory 1202 and peripheral device interface 1203 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0249] The radio frequency (RF) circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1204 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1204 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1204 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0250] Display screen 1205 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1205 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1201 for processing. In this case, display screen 1205 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1205, disposed on the front panel of terminal 1200; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1200 or in a folded design; in still other embodiments, display screen 1205 may be a flexible display screen, disposed on a curved or folded surface of terminal 1200. Furthermore, display screen 1205 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1205 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0251] The camera assembly 1206 is used to acquire images or videos. Optionally, the camera assembly 1206 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1206 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0252] The audio circuit 1207 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1201 for processing, or input to the radio frequency circuit 1204 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1200. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1201 or the radio frequency circuit 1204 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1207 may also include a headphone jack.

[0253] Power supply 1209 is used to power the various components in terminal 1200. Power supply 1209 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1209 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0254] In some embodiments, the terminal 1200 further includes one or more sensors 1210. The one or more sensors 1210 include, but are not limited to: an acceleration sensor 1211, a gyroscope sensor 1212, a pressure sensor 1213, an optical sensor 1215, and a proximity sensor 1216.

[0255] Accelerometer 1211 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established with terminal 1200. For example, accelerometer 1211 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 1201 can control display screen 1205 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1211. Accelerometer 1211 can also be used for games or for acquiring user motion data.

[0256] The gyroscope sensor 1212 can detect the orientation and rotation angle of the terminal 1200. The gyroscope sensor 1212 can work in conjunction with the accelerometer sensor 1211 to collect the user's 3D movements on the terminal 1200. Based on the data collected by the gyroscope sensor 1212, the processor 1201 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0257] The pressure sensor 1213 can be disposed on the side bezel of the terminal 1200 and / or on the lower layer of the display screen 1205. When the pressure sensor 1213 is disposed on the side bezel of the terminal 1200, it can detect the user's grip signal on the terminal 1200, and the processor 1201 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1213. When the pressure sensor 1213 is disposed on the lower layer of the display screen 1205, the processor 1201 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1205. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0258] The optical sensor 1215 is used to collect ambient light intensity. In one embodiment, the processor 1201 can control the display brightness of the display screen 1205 based on the ambient light intensity collected by the optical sensor 1215. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1205 is increased; when the ambient light intensity is low, the display brightness of the display screen 1205 is decreased. In another embodiment, the processor 1201 can also dynamically adjust the shooting parameters of the camera assembly 1206 based on the ambient light intensity collected by the optical sensor 1215.

[0259] The proximity sensor 1216, also known as a distance sensor, is typically mounted on the front panel of the terminal 1200. The proximity sensor 1216 is used to detect the distance between the user and the front of the terminal 1200. In one embodiment, when the proximity sensor 1216 detects that the distance between the user and the front of the terminal 1200 is gradually decreasing, the processor 1201 controls the display screen 1205 to switch from a screen-on state to a screen-off state; when the proximity sensor 1216 detects that the distance between the user and the front of the terminal 1200 is gradually increasing, the processor 1201 controls the display screen 1205 to switch from a screen-off state to a screen-on state.

[0260] Those skilled in the art will understand that Figure 12 The structure shown does not constitute a limitation on terminal 1200 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0261] The electronic device in the above method embodiments can be implemented as a server. For example, Figure 13This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1300 can vary significantly due to different configurations or performance. It can include one or more Central Processing Units (CPUs) 1301 and one or more memories 1302. The memories 1302 store at least one computer program, which is loaded and executed by the processor 1301 to implement the parameter determination method for determining motion control instructions provided in the various method embodiments described above. Of course, the server can also have wired or wireless network interfaces and input / output interfaces for input / output. The server can also include other components for implementing device functions, which will not be elaborated here.

[0262] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one computer program, wherein the at least one computer program is a parameter determination method executable by a processor to perform the motion control instructions in the above embodiments. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0263] In an exemplary embodiment, a computer program product or computer program is also provided, comprising one or more lines of program code stored in a computer-readable storage medium. One or more processors of an electronic device are capable of reading the one or more lines of program code from the computer-readable storage medium, and the one or more processors execute the one or more lines of program code, enabling the electronic device to execute the method for determining the determination parameters of the aforementioned motion control instructions.

[0264] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0265] It should be understood that determining B based on A does not mean determining B solely based on A; it also means determining B based on A and / or other information.

[0266] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0267] The above description is only an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining parameters for a motion control command, characterized in that, The method includes: Acquire real-world environmental data, which includes the environmental state of the target and the action control commands executed by the target; Based on at least two sets of data from the real environment data, at least two virtual environments corresponding to the real environment are generated, wherein one virtual environment is obtained based on one set of data, and at least two virtual environments are generated through different real environment data; Obtain an action control command from the real environment data as the first action control command; From the at least two virtual environments, a virtual environment is randomly selected as the target virtual environment, and the first action control command is executed in the target virtual environment to obtain simulated environment data; Simulated environment data generated during the execution of the first action control instruction in the target virtual environment is written to a cache in batches. In response to the data volume in the cache reaching a threshold, the simulated environment data in the cache is filtered based on a target filtering rule to obtain filtered simulated environment data. The target filtering rule is a filtering rule based on a target threshold, where the target threshold is the average or quantile of each simulated environment data item in the cache, and the target filtering rule indicates that data in the simulated environment data that is higher than the target threshold should be removed. Based on the filtered simulated environment data, the determination parameters of the action control command are updated to determine the second action control command; and the cache is cleared, and the batch writing and filtering steps are continued until the third target condition is reached to stop, thereby obtaining the filtered simulated environment data. The third target condition includes any one of the following: the determination parameters converge, and the number of loops reaches the target number. Continuing based on the second action control command, a virtual environment is randomly selected again from the at least two virtual environments as the target virtual environment. The second action control command is executed and the determined parameters are updated in this newly selected target virtual environment until the first target condition is met, thus obtaining the target determined parameters. The process of updating the determined parameters based on each control command is an iterative process. The target determined parameters are obtained through multiple iterations. In each iteration, after obtaining an action control command, a virtual environment is randomly selected from the at least two virtual environments as the target virtual environment for action control. Based on the real environment data and the second action control command, the target environment state after the target executes the second action control command in the real environment is determined; based on the target environment state and the simulated environment data obtained by executing the second action control command in the selected target virtual environment, the selected target virtual environment is updated.

2. The method according to claim 1, characterized in that, The step of generating at least two virtual environments corresponding to the real environment based on at least two sets of data from the real environment data includes: The real-world environmental data is grouped to obtain at least two sets of data; Based on the at least two sets of data, at least two neural networks are trained to obtain at least two virtual environments corresponding to the real environment, wherein one virtual environment is obtained by training one neural network based on one set of data.

3. The method according to claim 2, characterized in that, The real-world environment data is grouped to obtain at least two sets of data, including: The real environment data is randomly sampled at least twice to obtain at least two sets of data.

4. The method according to claim 2, characterized in that, The step of training at least two neural networks based on the at least two sets of data to obtain at least two virtual environments corresponding to the real environment includes: The at least two sets of data are respectively input into the at least two neural networks; For one of the at least two neural networks, the neural network determines the candidate environmental state of the target based on any environmental state in which the target is located and the action control instructions executed by the target in the environmental state; Based on the candidate environment state and the target environment state in the real environment data, the error of the candidate environment state is obtained; Based on the error, the network parameters of the neural network are adjusted until the second target condition is met.

5. The method according to claim 1, characterized in that, The step of executing the first action control command in the target virtual environment to obtain simulated environment data includes: Based on the environmental state of the target in the target virtual environment and the first action control command, the next environmental state of the target is obtained, and the next environmental state of the target is used as candidate environmental data.

6. The method according to claim 5, characterized in that, The target virtual environment is implemented based on a neural network; The step of obtaining the next environmental state of the target based on the environmental state of the target in the target virtual environment and the first action control command includes: The first action control command is input into the target neural network. The target neural network determines the environmental state change information of the target based on the environmental state of the target and the first action control command. Based on the environmental state change information, it outputs the next environmental state of the target.

7. The method according to claim 1, characterized in that, The step of filtering the simulated environment data in the cache based on the target filtering rules to obtain the filtered simulated environment data also includes: The simulated environment data in the cache is analyzed, and behavioral data related to the target event is removed to obtain the filtered simulated environment data.

8. The method according to claim 1, characterized in that, The step of updating the determination parameters of the motion control command based on the filtered simulated environment data to determine the second motion control command includes: Based on the filtered simulated environment data and the first action control command, the expected value is obtained; Obtain the gradient of the expected value; Based on the gradient, the determination parameters of the motion control command are updated; Based on the updated determined parameters, the filtered simulation environment data, and the first action control command, a second action control command is determined.

9. The method according to claim 1, characterized in that, The acquisition of real-world environmental data includes: Based on sensors, the environmental state of the target in a real environment and the action control commands executed by the target in that environmental state are collected to obtain the real environment data.

10. The method according to claim 8, characterized in that, The first objective condition is that the expected value converges; or the first objective condition is that the number of iterations reaches the target number.

11. A device for determining parameters of a motion control command, characterized in that, The device includes: The acquisition module is used to acquire real-world environmental data, which includes the environmental state of the target and the action control commands executed by the target. The generation module is used to generate at least two virtual environments corresponding to the real environment based on at least two sets of data in the real environment data, wherein one virtual environment is obtained based on one set of data, and at least two virtual environments are generated through different real environment data; The acquisition module is further configured to acquire an action control instruction from the real environment data as a first action control instruction; randomly select a virtual environment from the at least two virtual environments as a target virtual environment, execute the first action control instruction in the target virtual environment, and obtain simulated environment data; An update module is used to write simulated environment data generated during the execution of the first action control instruction in the target virtual environment into a cache in batches. In response to the data volume in the cache reaching a threshold, the simulated environment data in the cache is filtered based on a target filtering rule to obtain filtered simulated environment data. The target filtering rule is a filtering rule based on a target threshold, where the target threshold is the average or quantile of each simulated environment data item in the cache, and the target filtering rule instructs the removal of data in the simulated environment data that exceeds the target threshold. Based on the filtered simulated environment data, the determination parameters of the action control instruction are updated to determine a second action control instruction. The cache is then cleared, and the batch writing and filtering steps are continued until a third target condition is met, at which point the filtered simulated environment data is obtained. The third target condition includes any one of the following: parameter convergence or the number of iterations reaches a target number. The acquisition module and the update module are further configured to continue, based on the second action control instruction, randomly select a virtual environment from the at least two virtual environments as the target virtual environment, execute the second action control instruction and update the determined parameters in the newly selected target virtual environment, until the first target condition is met, and obtain the target determined parameters. The process of updating the determined parameters based on each control instruction is an iterative process, and the target determined parameters are obtained through multiple iterations. In each iteration, after acquiring an action control instruction, a virtual environment is randomly selected from the at least two virtual environments as the target virtual environment for action control; and... The determination module is used to determine the target environment state after the target executes the second action control command in the real environment, based on the real environment data and the second action control command; The update module is further configured to update the selected target virtual environment based on the target environment state and the simulated environment data obtained by executing the second action control command in the selected target virtual environment.

12. The apparatus according to claim 11, characterized in that, The generation module is used for: The real-world environmental data is grouped to obtain at least two sets of data; Based on the at least two sets of data, at least two neural networks are trained to obtain at least two virtual environments corresponding to the real environment, wherein one virtual environment is obtained by training one neural network based on one set of data.

13. The apparatus according to claim 12, characterized in that, The generation module is used for: The real environment data is randomly sampled at least twice to obtain at least two sets of data.

14. The apparatus according to claim 12, characterized in that, The generation module is used for: The at least two sets of data are respectively input into the at least two neural networks; For one of the at least two neural networks, the neural network determines the candidate environmental state of the target based on any environmental state in which the target is located and the action control instructions executed by the target in the environmental state; Based on the candidate environment state and the target environment state in the real environment data, the error of the candidate environment state is obtained; Based on the error, the network parameters of the neural network are adjusted until the second target condition is met.

15. The apparatus according to claim 11, characterized in that, The acquisition module is used for: Based on the environmental state of the target in the target virtual environment and the first action control command, the next environmental state of the target is obtained, and the next environmental state of the target is used as candidate environmental data.

16. The apparatus according to claim 15, characterized in that, The target virtual environment is implemented based on a neural network; the acquisition module is used to input the first action control command into the target neural network, and the target neural network determines the environmental state change information of the target based on the environmental state of the target and the first action control command, and outputs the next environmental state of the target based on the environmental state change information.

17. The apparatus according to claim 11, characterized in that, The update module is also used to perform: The simulated environment data in the cache is analyzed, and behavioral data related to the target event is removed to obtain the filtered simulated environment data.

18. The apparatus according to claim 11, characterized in that, The update module is used for: Based on the filtered simulated environment data and the first action control command, the expected value is obtained; Obtain the gradient of the expected value; Based on the gradient, the determination parameters of the motion control command are updated; Based on the updated determined parameters, the filtered simulation environment data, and the first action control command, a second action control command is determined.

19. The apparatus according to claim 11, characterized in that, The acquisition module is used to collect the environmental state of the target in the real environment based on sensors, as well as the action control commands executed by the target in the environmental state, to obtain the real environment data.

20. The apparatus according to claim 18, characterized in that, The first objective condition is that the expected value converges; or the first objective condition is that the number of iterations reaches the target number.

21. An electronic device, characterized in that, The electronic device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement the method for determining the parameters of the motion control command as described in any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that, The storage medium stores at least one computer program, which is loaded and executed by a processor to implement the method for determining the parameters of the motion control command as described in any one of claims 1 to 10.

23. A computer program product, characterized in that, The computer program product includes one or more lines of program code stored in a computer-readable storage medium. One or more processors of the electronic device are capable of reading the one or more lines of program code from the computer-readable storage medium. The one or more processors execute the one or more lines of program code, enabling the electronic device to execute a method for determining parameters to implement the motion control instructions as described in any one of claims 1 to 10.

Images