Automatic driving method, device, equipment and storage medium

Abstract

The present disclosure provides an autonomous driving method, apparatus, device, and storage medium, relating to the field of artificial intelligence, and more particularly, to the field of reinforcement learning technology. A specific implementation scheme comprises: obtaining interaction data generated by a mobile agent's autonomous driving according to an initial strategy and its interaction with the driving environment; determining an advantage function based on the interaction data; in response to a preset stopping condition being met, updating the parameters of the initial strategy according to the advantage function to obtain an optimal strategy; and determining an autonomous driving strategy for the intelligent mobile agent based on the optimal strategy for autonomous driving. This implementation method can utilize reinforcement learning to determine the optimal autonomous driving strategy for the mobile agent, thereby improving the safety of the mobile agent.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of reinforcement learning, and specifically to autonomous driving methods, devices, equipment, and storage media. Background Art

[0002] Reinforcement learning is a hot area in artificial intelligence, which is concerned with how agents perceive their environment. Reinforcement learning focuses on how an agent should take actions in an environment to maximize its cumulative reward. In many environments, safety is a critical issue, and certain errors are unacceptable: for example, a robotic system interacting with humans should not cause harm to humans while exploring. While it is currently typical to train reinforcement learning agents primarily or entirely in simulation, with few safety concerns, we expect that the challenges of simulating real-world complexities (such as AI interactions) will lead to a shift towards training reinforcement learning agents directly in the real world, where safety issues are important. Summary of the Invention

[0003] The present disclosure provides an autonomous driving method, apparatus, device, and storage medium.

[0004] According to a first aspect, an autonomous driving method is provided, comprising: obtaining interaction data generated by the interaction between a mobile intelligent body and a driving environment according to an initial autonomous driving strategy; determining an advantage function based on the interaction data; in response to a preset stop condition being met, updating the parameters of the initial strategy according to the advantage function to obtain an optimal strategy; and determining, based on the optimal strategy, an autonomous driving strategy for the intelligent mobile body for autonomous driving.

[0005] According to a second aspect, an automatic driving device is provided, comprising: an interaction data acquisition unit, configured to acquire interaction data generated by the interaction between the mobile intelligent body and the driving environment according to the initial automatic driving strategy; an advantage function determination unit, configured to determine the advantage function based on the interaction data; an optimal strategy determination unit, configured to update the parameters of the initial strategy according to the advantage function in response to the preset stop condition not being met, until the preset stop condition is met to obtain the optimal strategy; and an automatic driving unit, configured to determine the automatic driving strategy of the intelligent mobile body for automatic driving according to the optimal strategy.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor executes the method described in the first aspect.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method described in the first aspect.

[0008] According to a fifth aspect, a computer program product comprises a computer program, wherein the computer program implements the method described in the first aspect when executed by a processor.

[0009] The technology disclosed in the present invention can use reinforcement learning to determine the optimal autonomous driving strategy for a mobile agent, thereby improving the safety of the mobile agent.

[0010] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0012] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0013] Figure 2 is a flowchart of an embodiment of an autonomous driving method according to the present disclosure;

[0014] Figure 3 is a schematic diagram of an application scenario of the autonomous driving method according to the present disclosure;

[0015] Figure 4 is a flowchart of another embodiment of the autonomous driving method according to the present disclosure;

[0016] Figure 5 is a schematic structural diagram of an embodiment of an automatic driving device according to the present disclosure;

[0017] Figure 6 This is a block diagram of an electronic device used to implement the autonomous driving method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0019] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0020] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the autonomous driving method or autonomous driving device of the present disclosure may be applied.

[0021] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0022] The aforementioned terminal devices 101, 102, and 103 can be installed in a mobile intelligent entity, which can be an autonomously moving object such as an autonomous vehicle or an intelligent robot. The mobile intelligent entity can also be equipped with environmental sensing devices, such as image acquisition devices, temperature sensing devices, lidar, millimeter-wave radar, and the like. The mobile intelligent entity can interact with the server 105 via the network 104 to receive or send messages. For example, the mobile intelligent entity can send data sensed by each environmental sensing device to the server 105. Various communication client applications, such as voice recognition applications and data processing applications, can be installed on the terminal devices 101, 102, and 103.

[0023] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablet computers, e-book readers, car computers, laptop computers, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitations are given here.

[0024] Server 105 can be a server that provides various services, such as a backend server that provides autonomous driving strategies to terminal devices 101, 102, and 103. The backend server can use reinforcement learning algorithms to determine the optimal driving strategy for the intelligent mobile object at each location. It then feeds this optimal driving strategy back to terminal devices 101, 102, and 103. Terminal devices 101, 102, and 103 can then control the mobile object to perform autonomous driving based on this optimal driving strategy.

[0025] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitations are given here.

[0026] It should be noted that the autonomous driving method provided in the embodiments of the present disclosure can be executed by the terminal devices 101, 102, 103, or by the server 105. Accordingly, the autonomous driving device can be provided in the terminal devices 101, 102, 103, or in the server 105.

[0027] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0028] Continue to refer Figure 2 , shows a process 200 of an embodiment of an autonomous driving method according to the present disclosure. The autonomous driving method of this embodiment includes the following steps:

[0029] Step 201: Obtain interaction data generated by the mobile agent's automatic driving according to the initial strategy and the interaction with the driving environment.

[0030] In this embodiment, the executing entity of the autonomous driving method may set an initial strategy for the mobile agent. Specifically, the initial strategy may be obtained through initialization. The executing entity may calculate interaction data generated by the mobile agent's interaction with the driving environment during autonomous driving according to the initial strategy. Generally, the interaction process may include 30 rounds. The interaction data may include parameters such as state and speed.

[0031] Step 202: Determine an advantage function based on the interaction data.

[0032] After determining the interaction data, the execution entity can use it to determine an advantage function. This advantage function can include an advantage function for rewards and an advantage function for each constraint. The execution entity can obtain the advantage function formula and then substitute the parameter values from the interaction data into the formula to obtain the advantage function.

[0033] Step 203 : In response to the preset stopping condition not being met, the parameters of the initial strategy are updated according to the advantage function until the preset stopping condition is met, thereby obtaining the optimal strategy.

[0034] The execution entity can monitor in real time whether a preset stopping condition is met. This can be an iterative stopping condition, such as when the number of parameter updates reaches a preset value or when the advantage function stops converging. If the execution entity determines that the preset stopping condition is not met, the execution entity can update the parameters of the initial policy based on the advantage function. For example, the execution entity can update the parameters of the initial policy in the gradient direction until the preset stopping condition is met, resulting in the optimal policy.

[0035] Step 204: Determine an autonomous driving strategy for the intelligent mobile object based on the optimal strategy for autonomous driving.

[0036] In this embodiment, after determining the optimal strategy, the execution subject can feed back the optimal strategy to the mobile intelligent agent. The mobile intelligent agent can control itself according to the above optimal strategy, obtain the autonomous driving strategy, and finally perform autonomous driving according to the autonomous driving strategy.

[0037] Continue to see Figure 3 , which shows a schematic diagram of an application scenario of the autonomous driving method according to the present disclosure. Figure 3 In this application scenario, intelligent robot 301 interacts with server 302, and intelligent robot 301 can send its surroundings to server 302 in real time. Server 302 can calculate the interaction data obtained by intelligent robot 301's autonomous driving and interaction with the driving environment according to the initial strategy. Server 302 uses this interaction data and a reinforcement learning algorithm to determine an autonomous driving strategy for intelligent robot 301 and feeds the autonomous driving strategy back to intelligent robot 301. Intelligent robot 301 can then drive autonomously according to the autonomous driving strategy.

[0038] The autonomous driving method provided by the above-mentioned embodiments of the present disclosure can use reinforcement learning to determine the optimal autonomous driving strategy for the mobile intelligent body, thereby improving the safety of the mobile intelligent body.

[0039] Continue to see Figure 4 , which shows a process 400 of another embodiment of the automatic driving method according to the present disclosure. Figure 4 As shown, the method of this embodiment may include the following steps:

[0040] Step 401: Obtain interaction data generated by the mobile agent's automatic driving according to the initial strategy and the interaction with the driving environment.

[0041] Step 402: Determine a value function and an action-value function based on the interaction data; and determine an advantage function based on the value function and the action-value function.

[0042] In this embodiment, the execution entity can determine a value function and an action-value function based on the interaction data. Here, the value function can be understood as the expectation of obtaining a cumulative discounted reward using the current strategy in the current state. The action-value function refers to the expectation of obtaining a cumulative discounted reward after performing a certain action in the current state and continuing to use the current strategy. The execution entity can use the pre-designed formulas for the value function and action-value function by substituting the parameter values in the interaction data into these two formulas to obtain the value function and action-value function, respectively.

[0043] After obtaining the value function and the action value function, the execution subject can subtract the two and use the obtained function as the advantage function.

[0044] Step 403: In response to the preset stopping condition not being met, the constrained problem is converted into an unconstrained problem; the parameters of the initial strategy are updated according to the advantage function until the preset stopping condition is met, and the optimal solution of the unconstrained problem is determined; and the optimal solution is used as the optimal strategy.

[0045] In this embodiment, if the preset stopping condition is not met, the parameters of the initial strategy can be updated. Because the process of solving a constrained problem is complex, to simplify the calculation process, the execution entity can first convert the constrained problem into an unconstrained problem. It is understood that the converted unconstrained problem is equivalent to the constrained problem. That is, the optimal solution to the unconstrained problem can be used as the solution to the constrained problem. After obtaining the unconstrained problem, the parameters of the initial strategy can be updated based on the advantage function. Once the preset stopping condition is met, the optimal solution to the unconstrained problem can be obtained. This optimal solution is then used as the final optimal strategy.

[0046] In some optional implementations of this embodiment, the execution subject can Figure 4 The following steps, not shown in FIG, convert the constrained problem into an unconstrained problem: convert the constrained problem into an equivalent problem according to the importance sampling ratio; and convert the equivalent problem into an unconstrained problem using an exact penalty function.

[0047] In this implementation, a neural network is generally used as a strategy. For convenience, the importance sampling ratio can be used to express some formulas in the constraint problem using the importance sampling ratio. In this way, the constraint problem is converted into an equivalent problem. Then, the exact penalty function is further used to convert the equivalent problem into an unconstrained problem. Here, the exact penalty function is a very important concept in nonlinear programming. The L1 penalty function is used to convert the constrained optimization problem into an equivalent unconstrained optimization problem. The penalty factor only needs to be greater than a sufficiently large positive number to ensure the equivalence of the solution, and does not need to tend to positive infinity, thereby causing numerical problems. Here, when determining the exact penalty function, the penalty factor can be set to a preset value. The above preset value can be an empirical value obtained after multiple calculations.

[0048] In some optional implementations of this embodiment, the execution subject can Figure 4 The following steps, not shown in FIG, determine the optimal solution of the unconstrained problem: using the approximation error, convert the unconstrained problem into an approximate problem; according to the advantage function, conservatively update the parameters of the initial strategy within the trust region of the approximate problem to obtain the optimal solution of the approximate problem.

[0049] In this implementation, to further simplify computation, the execution entity can further utilize approximation error to transform the unconstrained problem into an approximate problem. Specifically, the execution entity can use existing sampled data to approximate the optimization variables of the policy. It is understood that during this approximate estimation, the approximate error can be calculated and controlled within an appropriate range, which can be referred to as the trust region. In specific applications, this approximate error can be related to the difference between the variables in the policy to be optimized and the existing policy, as well as the number of constraints. To improve the accuracy of the resulting approximate problem, the execution entity also needs to constrain the trust region. That is, if the trust region constraints are not met during the parameter update process, the execution entity can exit the update prematurely. During parameter iteration, proximal policy optimization (PPO) can also be used for conservative updates. The PPO algorithm is a new Policy Gradient algorithm. Policy Gradient algorithms are very sensitive to step size, but it is difficult to select an appropriate step size. If the difference between the new and old policies during training is too large, learning is not conducive. PPO proposes that the trimming objective function can achieve small-batch updates in multiple training steps, solving the problem of difficult step size determination in the Policy Gradient algorithm and improving sample efficiency and performance.

[0050] Step 404: Determine the autonomous driving strategy of the mobile agent based on the location of the mobile agent, the driving environment, and the optimal strategy.

[0051] After determining the optimal strategy within the strategy space, the execution entity can further determine the autonomous driving strategy for the mobile agent based on the mobile agent's location, driving environment, and optimal strategy. Specifically, the execution entity can calculate the interaction data between the mobile agent and the driving environment according to the optimal strategy starting from the aforementioned location, and determine the autonomous driving strategy based on this interaction data.

[0052] The autonomous driving method provided by the above-mentioned embodiments of the present disclosure, by introducing an exact penalty function, transforms a complex constrained optimization problem into an approximately equivalent unconstrained optimization problem, enabling direct optimization using first-order methods. This not only reduces implementation difficulty and improves sample efficiency, but also achieves superior performance. Furthermore, this method is easily scalable to multi-constraint scenarios and does not impose any feasibility requirements on the initial strategy.

[0053] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an automatic driving device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0054] like Figure 5 As shown, the automatic driving device 500 of this embodiment includes: an interaction data acquisition unit 501, an advantage function determination unit 502, an optimal strategy determination unit 503 and an automatic driving unit 504.

[0055] The interaction data acquisition unit 501 is configured to acquire interaction data generated by the mobile agent's automatic driving according to the initial strategy and the interaction with the driving environment.

[0056] The advantage function determining unit 502 is configured to determine an advantage function according to the interaction data.

[0057] The optimal strategy determining unit 503 is configured to update the parameters of the initial strategy according to the advantage function in response to the preset stopping condition not being met, until the preset stopping condition is met to obtain the optimal strategy.

[0058] The automatic driving unit 504 is configured to determine an automatic driving strategy for the intelligent mobile body for automatic driving according to the optimal strategy.

[0059] In some optional implementations of this embodiment, the advantage function determination unit 502 is further configured to: determine a value function and an action-value function according to the interaction data; and determine an advantage function according to the value function and the action-value function.

[0060] In some optional implementations of this embodiment, the optimal strategy determination unit 503 is further configured to: transform the constrained problem into an unconstrained problem; update the parameters of the initial strategy according to the advantage function to determine the optimal solution of the unconstrained problem; and use the optimal solution as the optimal strategy.

[0061] In some optional implementations of this embodiment, the optimal strategy determination unit 503 is further configured to: transform the constrained problem into an equivalent problem according to the importance sampling ratio; and transform the equivalent problem into an unconstrained problem using an exact penalty function.

[0062] In some optional implementations of this embodiment, the optimal strategy determination unit 503 is further configured to: utilize the approximate error to convert the unconstrained problem into an approximate problem; based on the advantage function, conservatively update the parameters of the initial strategy within the trust domain of the approximate problem to obtain the optimal solution to the approximate problem.

[0063] In some optional implementations of this embodiment, the autonomous driving unit 504 is further configured to determine the autonomous driving strategy of the mobile agent based on the location, driving environment, and optimal strategy of the mobile agent.

[0064] It should be understood that each unit recorded in the automatic driving device 500 is respectively Figure 2 Therefore, the operations and features described above for the autonomous driving method are also applicable to the device 500 and the units contained therein, and will not be repeated here.

[0065] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0066] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0067] Figure 6 A block diagram of an electronic device 600 for executing an autonomous driving method according to an embodiment of the present disclosure is shown. The term "electronic device" is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The term "electronic device" may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided for example purposes only and are not intended to limit the implementation of the present disclosure as described and / or claimed herein.

[0068] like Figure 6 As shown, the electronic device 600 includes a processor 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a memory 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The processor 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An I / O interface (input / output interface) 605 is also connected to the bus 604.

[0069] Multiple components in the electronic device 600 are connected to the I / O interface 605, including an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a memory 608, such as a magnetic disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0070] Processor 601 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 601 executes the various methods and processes described above, such as the autonomous driving method. For example, in some embodiments, the autonomous driving method may be implemented as a computer software program tangibly embodied in a machine-readable storage medium, such as memory 608. In some embodiments, part or all of the computer program may be loaded and / or installed onto electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by processor 601, one or more steps of the autonomous driving method described above may be performed. Alternatively, in other embodiments, processor 601 may be configured to execute the autonomous driving method by any other suitable means (e.g., via firmware).

[0071] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0072] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. The above program code can be packaged into a computer program product. These program codes or computer program products can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that the program code, when executed by the processor 601, causes the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0073] In the context of the present disclosure, a machine-readable storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable storage medium can be a machine-readable signal storage medium or a machine-readable storage medium. A machine-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0074] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0075] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0076] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is established by computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within a cloud computing service system that addresses the management difficulties and poor business scalability of traditional physical hosts and VPS services ("Virtual Private Servers," or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0077] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of this disclosure can be achieved, and this document is not limited here.

[0078] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. An autonomous driving method, comprising: Obtain the interaction data generated by the mobile agent's automatic driving based on the initial strategy and the interaction with the driving environment; determining an advantage function based on the interaction data; In response to a preset stopping condition being met, updating the parameters of the initial strategy according to the advantage function to obtain an optimal strategy, including: converting the constrained problem into an unconstrained problem, including: converting the constrained problem into an equivalent problem according to an importance sampling ratio; converting the equivalent problem into an unconstrained problem using an exact penalty function, wherein the penalty factor corresponding to the exact penalty function is a preset positive number; updating the parameters of the initial strategy according to the advantage function until the preset stopping condition is met, determining an optimal solution to the unconstrained problem; and using the optimal solution as the optimal strategy; Based on the optimal strategy, an autonomous driving strategy for the intelligent mobile body is determined for autonomous driving.

2. The method according to claim 1, wherein Determining the advantage function according to the interaction data includes: Determining a value function and an action-value function based on the interaction data; The advantage function is determined based on the value function and the action-value function.

3. The method according to claim 1, wherein The updating of the parameters of the initial strategy according to the advantage function to determine the optimal solution of the unconstrained problem includes: Using the approximation error, converting the unconstrained problem into an approximate problem; According to the advantage function, the parameters of the initial strategy are conservatively updated within the trust region of the approximation problem to obtain an optimal solution to the approximation problem.

4. The method according to claim 1, wherein Determining an autonomous driving strategy for the intelligent mobile object for autonomous driving based on the optimal strategy includes: Determine the automatic driving strategy of the mobile agent according to the position, driving environment and the optimal strategy of the mobile agent.

5. An automatic driving device comprising: An interaction data acquisition unit is configured to acquire interaction data generated by the mobile agent interacting with the driving environment according to the initial strategy automatic driving; an advantage function determining unit, configured to determine an advantage function according to the interaction data; The optimal strategy determination unit is configured to, in response to a preset stopping condition not being met, update the parameters of the initial strategy according to the advantage function until the preset stopping condition is met, thereby obtaining the optimal strategy, including: converting the constrained problem into an unconstrained problem, including: converting the constrained problem into an equivalent problem according to an importance sampling ratio; converting the equivalent problem into an unconstrained problem using an exact penalty function, wherein the penalty factor corresponding to the exact penalty function is a preset positive number; updating the parameters of the initial strategy according to the advantage function until the preset stopping condition is met, thereby determining the optimal solution of the unconstrained problem; and using the optimal solution as the optimal strategy; The automatic driving unit is configured to determine an automatic driving strategy for the intelligent mobile body for automatic driving according to the optimal strategy.

6. The device according to claim 5, wherein The advantage function determination unit is further configured to: Determining a value function and an action-value function based on the interaction data; The advantage function is determined based on the value function and the action-value function.

7. The device according to claim 5, wherein The optimal strategy determination unit is further configured to: Using the approximation error, converting the unconstrained problem into an approximate problem; According to the advantage function, the parameters of the initial strategy are conservatively updated within the trust region of the approximation problem to obtain an optimal solution to the approximation problem.

8. The device according to claim 5, wherein The autonomous driving unit is further configured to: Determine the automatic driving strategy of the mobile agent according to the position, driving environment and the optimal strategy of the mobile agent.

9. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 4.

11. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 4.

Images