Construction method and device of driving simulation scene, electronic equipment and program product

Abstract

The invention is suitable for the technical field of automatic driving, and provides a construction method and device of a driving simulation scene, electronic equipment and a program product. The method comprises the following steps: acquiring an initial driving simulation scene, wherein the initial driving simulation scene at least comprises a traffic simulation environment, an initial simulation pedestrian and an initial simulation animal; simulating the behavior of the initial simulation pedestrian based on a traffic simulation environment to obtain a target simulation pedestrian; simulating behaviors of the initial simulation animal based on a traffic simulation environment to obtain a target simulation animal; and constructing a target driving simulation scene according to the traffic simulation environment, the target simulation pedestrian and the target simulation animal. The simulated pedestrians and the simulated animals in the driving simulation scene constructed by the method are obtained through simulation based on the traffic simulation environment, so that the constructed driving simulation scene is more similar to a real driving scene, and the authenticity of the constructed driving simulation scene is improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method, device, electronic equipment, and program product for constructing a driving simulation scene. Background Art

[0002] In the field of autonomous driving technology, it is necessary to build driving simulation scenarios, which are then used to verify and train autonomous driving technology. Building driving simulation scenarios typically requires building simulated traffic environments, simulated pedestrians, and simulated animals.

[0003] Currently, when constructing simulated pedestrians and animals in driving simulation scenarios, they are typically set as obstacles; or the behavioral rules for these simulated pedestrians and animals are overly simple (for example, they are set to move along a preset route). It can be seen that the behavior of the simulated pedestrians and animals constructed currently does not change with changes in the traffic simulation environment. In other words, the behavior of the simulated pedestrians and animals constructed currently lacks interactivity with the traffic simulation environment. In real driving scenarios, the behavior of pedestrians and animals will change with changes in the traffic environment. Therefore, compared to real driving scenarios, the realism of the driving simulation scenarios constructed currently is lower. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide a method, device, electronic device and program product for constructing a driving simulation scene to solve the technical problem of low authenticity of the currently constructed driving simulation scenes.

[0005] In a first aspect, an embodiment of the present application provides a method for constructing a driving simulation scenario, comprising: Acquiring a pre-built initial driving simulation scene, wherein the initial driving simulation scene at least includes a traffic simulation environment, initial simulated pedestrians, and initial simulated animals; The behavior of the initial simulated pedestrian is simulated based on the traffic simulation environment using a pedestrian behavior simulation model to obtain a target simulated pedestrian; the pedestrian behavior simulation model is trained using a reinforcement learning algorithm and a shadow agent method; The behavior of the initial simulated animal is simulated based on the traffic simulation environment by using an animal behavior simulation model to obtain a target simulated animal; the animal behavior simulation model is obtained only by training with a reinforcement learning algorithm; A target driving simulation scene is constructed according to the traffic simulation environment, the target simulated pedestrians and the target simulated animals.

[0006] Optionally, the pedestrian behavior simulation model includes a target pedestrian strategy model and a shadow proxy model, wherein the shadow proxy model is obtained by training a preset pedestrian walking strategy; the pedestrian behavior simulation model is specifically trained in the following manner: Obtaining a pre-trained initial pedestrian policy model and the shadow agent model; Using the initial pedestrian strategy model and the shadow proxy model, the same training pedestrian behavior is simulated according to the same training environment to obtain the training pedestrian behavior corresponding to the initial pedestrian strategy model and the training pedestrian behavior corresponding to the shadow proxy model; Determining penalty reward information by comparing the training pedestrian behavior corresponding to the initial pedestrian strategy model with the training pedestrian behavior corresponding to the shadow proxy model; The penalty reward information is fed back to the initial pedestrian policy model to instruct the initial pedestrian policy model to be optimized according to the penalty reward information to obtain the target pedestrian policy model.

[0007] Optionally, the pedestrian behavior simulation model further includes a trained evaluation model; simulating the behavior of the initial simulated pedestrian based on the traffic simulation environment using the pedestrian behavior simulation model to obtain a target simulated pedestrian includes: The target pedestrian strategy model and the shadow proxy model are used to simulate the behavior of the initial simulated pedestrian based on the traffic simulation environment, thereby obtaining a first simulated pedestrian output by the target pedestrian strategy model and a second simulated pedestrian output by the shadow proxy model. The target simulated pedestrian is determined from the first simulated pedestrian and the second simulated pedestrian through the evaluation model.

[0008] Optionally, after simulating the behavior of the initial simulated pedestrian based on the traffic simulation environment using the pedestrian behavior simulation model to obtain a target simulated pedestrian, the method further includes: If the first simulated pedestrian is determined to be the target simulated pedestrian, outputting target reward information to the target pedestrian strategy model to instruct the target pedestrian strategy model to perform optimization according to the target reward information; If the second simulated pedestrian is determined as the target simulated pedestrian, target penalty information is output to the target pedestrian strategy model to instruct the target pedestrian strategy model to perform optimization according to the target penalty information.

[0009] Optionally, the animal behavior simulation model is trained specifically by the following methods: Obtain a pre-trained initial animal strategy model; Using the initial animal strategy model, simulating the behavior of the training animal according to the training environment to obtain the training animal behavior; Determining penalty-reward information for the training animal behavior based on a preset penalty-reward function and the training animal behavior; the penalty-reward function is used to reward the training animal for behaviors such as approaching a food source and moving away from a vehicle, and to penalize behaviors such as the training animal for behaviors that result in collisions and / or excessive energy consumption; The penalty reward information is fed back to the initial animal strategy model to instruct the initial animal strategy model to be optimized according to the penalty reward information to obtain the animal behavior simulation model.

[0010] Optionally, the initial driving simulation scenario is constructed in the following manner: Acquiring initial driving simulation scene data; the initial driving simulation scene data includes initial simulated pedestrian data and initial simulated animal data, and includes one or more of the following traffic simulation environment data: map data, road topology data, building data, obstacle data, traffic sign data, dynamic traffic signal data, dynamic weather data, dynamic vehicle data, and virtual dynamic sensor data; The dynamic initial driving simulation scene is constructed according to the initial simulated pedestrian data, the initial simulated animal data and the traffic simulation environment data.

[0011] Optionally, also include: Obtain training environment data corresponding to each training environment; Through the trained policy generalization model, the pedestrian behavior simulation model and the animal behavior simulation model are subjected to policy generalization processing based on the training environment data, and the pedestrian behavior simulation model after policy generalization processing and the animal behavior simulation model after policy generalization processing are output; the action difference between each target simulated pedestrian obtained by the pedestrian behavior simulation model after policy generalization processing according to the changing traffic simulation environment is lower than a first preset threshold, and the action difference between each target simulated animal obtained by the animal behavior simulation model after policy generalization processing according to the changing traffic simulation environment is lower than a second preset threshold.

[0012] In a second aspect, an embodiment of the present application provides a device for constructing a driving simulation scene, comprising: An initial scene acquisition unit is used to acquire a pre-built initial driving simulation scene, wherein the initial driving simulation scene at least includes a traffic simulation environment, initial simulated pedestrians, and initial simulated animals; a simulated pedestrian simulation unit, configured to simulate the behavior of the initial simulated pedestrian based on the traffic simulation environment using a pedestrian behavior simulation model to obtain a target simulated pedestrian; the pedestrian behavior simulation model is trained using a reinforcement learning algorithm and a shadow agent method; A simulated animal simulation unit, configured to simulate the behavior of the initial simulated animal based on the traffic simulation environment using an animal behavior simulation model to obtain a target simulated animal; the animal behavior simulation model is obtained only through training with a reinforcement learning algorithm; The target scene construction unit is used to construct a target driving simulation scene according to the traffic simulation environment, the target simulated pedestrians and the target simulated animals.

[0013] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor implements the steps of the method for constructing a driving simulation scenario as described in the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for constructing a driving simulation scene as described in the first aspect above.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed on an electronic device, enables the electronic device to execute the steps of the method for constructing a driving simulation scenario as described in the first aspect above.

[0016] The driving simulation scenario construction method, device, electronic device, and program product provided by the embodiments of the present application have the following beneficial effects: In the method for constructing a driving simulation scene provided in an embodiment of the present application, an initial driving simulation scene is first obtained, wherein the initial driving simulation scene includes at least a traffic simulation environment, an initial simulated pedestrian, and an initial simulated animal; then, a pedestrian behavior simulation model is used to simulate the behavior of the initial simulated pedestrian based on the traffic simulation environment to obtain a target simulated pedestrian; the pedestrian behavior simulation model is trained by a reinforcement learning algorithm and a shadow agent method; then, an animal behavior simulation model is used to simulate the behavior of the initial simulated animal based on the traffic simulation environment to obtain a target simulated animal; the animal behavior simulation model is trained only by a reinforcement learning algorithm; finally, a target driving simulation scene is constructed based on the traffic simulation environment, the target simulated pedestrian, and the target simulated animal. The target simulated pedestrians and target simulated animals in the driving simulation scene constructed by this method are both simulated based on the traffic simulation environment, so the constructed driving simulation scene is more similar to the real driving scene; in addition, in this method, the pedestrian behavior simulation model used to simulate the simulated pedestrians is trained by the reinforcement learning algorithm and the shadow agent method, and the animal behavior simulation model used to simulate the simulated animals is trained only by the reinforcement learning algorithm, which makes the behavior of the target simulated pedestrians more standardized and the behavior of the target simulated animals more random, which corresponds to the behavior of pedestrians and animals in the real driving scene, and thus can further improve the authenticity of the constructed driving simulation scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A flowchart of a method for constructing a driving simulation scenario provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a device for constructing a driving simulation scene provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] It should be noted that the terms used in the embodiments of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application. In the description of the embodiments of the present application, unless otherwise specified, "multiple" refers to two or more than two, and "at least one" and "one or more" refer to one, two or more. The terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the definition of "first" and "second" features can explicitly or implicitly include one or more of the features.

[0020] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0021] The execution subject of the driving simulation scene construction method provided in the embodiment of the present application can be an electronic device, which can execute each step of the driving simulation scene construction method provided in the embodiment of the present application. The electronic device can include but is not limited to a mobile phone, tablet computer, laptop computer, desktop computer, etc.

[0022] The method for constructing a driving simulation scenario provided in the embodiments of this application can be applied to various scenarios requiring the construction of a driving simulation scenario. For example, when a user needs to construct a driving simulation scenario for optimizing or verifying autonomous driving technology, the various steps of the method for constructing a driving simulation scenario provided in the embodiments of this application can be executed by an electronic device, thereby constructing a driving simulation scenario with high realism.

[0023] See also Figure 1 , Figure 1 This is a flowchart of a method for constructing a driving simulation scene provided in an embodiment of the present application. The method for constructing a driving simulation scene may include S101 to S104, which are described in detail as follows: In S101 , a pre-built initial driving simulation scene is obtained, where the initial driving simulation scene at least includes a traffic simulation environment, initial simulated pedestrians, and initial simulated animals.

[0024] In an embodiment of the present application, when a target driving simulation scene needs to be constructed, the electronic device may first obtain an initial driving simulation scene that at least includes a traffic simulation environment, initial simulated pedestrians, and initial simulated animals.

[0025] The initial simulated pedestrian may be a simulated pedestrian for which behavioral simulation has not yet been performed, and the initial simulated animal may be a simulated animal for which behavioral simulation has not yet been performed.

[0026] The traffic simulation environment may include at least one or more of the following: roads, vehicles, buildings, obstacles, plants, weather, and sensors.

[0027] Optionally, the electronic device can obtain initial driving simulation scene data, which includes initial simulated pedestrian data and initial simulated animal data, and includes one or more of the following traffic simulation environment data: map data, road topology data, building data, obstacle data, traffic sign data, dynamic traffic signal data, dynamic weather data, dynamic vehicle data, and virtual dynamic sensor data.

[0028] After obtaining the initial simulated pedestrian data, initial simulated animal data and traffic simulation environment data, a dynamic initial driving simulation scene is constructed.

[0029] In actual applications, electronic devices can use three-dimensional map construction technology, rendering technology, physical simulation engine, sensor simulation technology, real-time path planning algorithm and collision detection algorithm to build a dynamic initial driving simulation scene based on the initial driving simulation scene data.

[0030] In practical applications, the initial driving simulation scene data can be input by the user to generate the initial driving simulation scene required by the user; in addition, the initial driving simulation scene data can also be randomly generated to generate a random initial driving simulation scene, so that a random target driving simulation scene can be constructed to meet various practical needs.

[0031] In S102, the behavior of the initial simulated pedestrian is simulated based on the traffic simulation environment through the pedestrian behavior simulation model to obtain a target simulated pedestrian; the pedestrian behavior simulation model is trained by a reinforcement learning algorithm and a shadow agent method.

[0032] In an embodiment of the present application, after obtaining the initial driving simulation scene, the electronic device may first obtain a pedestrian behavior simulation model trained by a reinforcement learning algorithm and a shadow agent method.

[0033] The following is an explanation of the "shadow proxy method" in this application: In the field of autonomous driving, the "shadow proxy method" refers to a technology in which the autonomous driving system still operates but does not control the vehicle when a person is driving. It only optimizes its own algorithm by simulating decisions and comparing them with the driver's actual operations. The "shadow proxy method" in this application is similar to the "shadow proxy method" in the field of autonomous driving. The "shadow proxy method" in this application refers to: first, a shadow proxy model is obtained by training with a preset pedestrian walking strategy, and then the training pedestrian behavior corresponding to the shadow proxy model is compared with the training pedestrian behavior corresponding to the initial pedestrian strategy model pre-trained by the reinforcement learning algorithm, and the initial pedestrian strategy model is optimized by the reinforcement learning algorithm to finally obtain the target pedestrian strategy model.

[0034] Based on this, in one possible implementation, the pedestrian behavior simulation model may include a target pedestrian strategy model and a shadow agent model. The electronic device may train the pedestrian behavior simulation model using the reinforcement learning algorithm and shadow agent method described below. Detailed description is as follows: First, the electronic device can obtain a pre-trained initial pedestrian policy model and a shadow agent model.

[0035] In this implementation, the pre-trained initial pedestrian strategy model can be obtained by training using a preset reinforcement learning algorithm.

[0036] The shadow proxy model can be trained using a preset pedestrian walking strategy. The preset pedestrian walking strategy can be determined based on the actual behavior of real pedestrians in various traffic environments. Based on this, users can obtain the actual behavior of real pedestrians in various traffic environments and derive the pedestrian walking strategy based on the actual behavior of real pedestrians in various traffic environments. This allows the shadow proxy model to be constructed based on the pedestrian walking strategy.

[0037] After obtaining the initial pedestrian strategy model and the shadow agent model, the electronic device can optimize the initial pedestrian strategy model through the reinforcement learning algorithm and the shadow agent model to obtain the target pedestrian strategy model.

[0038] Specifically, the electronic device can simulate the behavior of the same training pedestrian according to the same training environment through the initial pedestrian strategy model and the shadow proxy model, and obtain the training pedestrian behavior corresponding to the initial pedestrian strategy model and the training pedestrian behavior corresponding to the shadow proxy model.

[0039] The electronic device may then determine the penalty-reward information by comparing the trained pedestrian behavior corresponding to the initial pedestrian policy model with the trained pedestrian behavior corresponding to the shadow proxy model. Specifically, when the trained pedestrian behavior corresponding to the initial pedestrian policy model and the trained pedestrian behavior corresponding to the shadow proxy model are the same, the electronic device may determine the penalty-reward information as the reward information; and when the trained pedestrian behavior corresponding to the initial pedestrian policy model and the trained pedestrian behavior corresponding to the shadow proxy model are different, the electronic device may determine the penalty-reward information as the penalty information.

[0040] Finally, the electronic device can feed back the penalty reward information to the initial pedestrian strategy model to instruct the initial pedestrian strategy model to optimize according to the penalty reward information to obtain the target pedestrian strategy model, thereby obtaining a pedestrian behavior simulation model.

[0041] In this implementation, the shadow proxy model is trained using a preset pedestrian walking strategy, which is determined based on the actual behavior of real pedestrians in various traffic environments. Therefore, the target simulated pedestrians obtained through the shadow proxy model are highly standardized, meaning that the target simulated pedestrians obtained through the shadow proxy model are less different from real pedestrians.

[0042] Since the target pedestrian strategy model is obtained by training and optimizing the shadow proxy model, it can be considered that the target simulated pedestrian obtained by the target pedestrian strategy model also has a strong standardization. In addition, since the initial pedestrian strategy model used to train the target pedestrian strategy model is obtained by training the preset reinforcement learning algorithm, it can be considered that the target simulated pedestrian obtained by the target pedestrian strategy model also has a certain degree of randomness, that is, the target simulated pedestrian obtained by the target pedestrian strategy model has a high degree of difference from the real pedestrian under a certain probability.

[0043] In summary, the target pedestrian policy model and the shadow proxy model share the same characteristics: both models simulate the behavior of simulated pedestrians based on the input traffic simulation environment to produce simulated pedestrians. However, the target pedestrian policy model and the shadow proxy model differ in that the target simulated pedestrians output by the shadow proxy model are more standardized than those output by the target pedestrian policy model, while the target simulated pedestrians output by the target pedestrian policy model exhibit a certain degree of randomness.

[0044] The advantage of jointly obtaining a target simulated pedestrian using a target pedestrian strategy model and a shadow proxy model with different characteristics is that if only the shadow proxy model is used to obtain the target simulated pedestrian, the target simulated pedestrian will completely follow the preset pedestrian walking strategy. Since the preset pedestrian walking strategy is not absolutely optimal, the target simulated pedestrian obtained using only the shadow proxy model may not be the optimal simulated pedestrian in some cases. However, the target simulated pedestrian obtained by jointly using the target pedestrian strategy model and the shadow proxy model retains both the standardization of the target simulated pedestrian output by the shadow proxy model and the randomness of the target simulated pedestrian output by the target pedestrian strategy model. The beneficial effect of retaining the randomness of the output target simulated pedestrian is that when the simulated pedestrian corresponding to the preset pedestrian walking strategy in a certain scenario is not the optimal simulated pedestrian, retaining the randomness of the output target simulated pedestrian enables the pedestrian behavior simulation model to still output the optimal simulated pedestrian in that scenario. Therefore, it can be considered that compared with the target simulated pedestrian obtained using only the shadow proxy model, the target simulated pedestrian obtained by jointly using the target pedestrian strategy model and the shadow proxy model is the optimal simulated pedestrian in all situations.

[0045] In one possible implementation, the pedestrian behavior simulation model further includes a trained evaluation model. The evaluation model can be used to evaluate a first simulated pedestrian output by the target pedestrian policy model and a second simulated pedestrian output by the shadow proxy model, thereby determining a target simulated pedestrian from the first simulated pedestrian output by the target pedestrian policy model and the second simulated pedestrian output by the shadow proxy model.

[0046] Based on this, the behavior of the initial simulated pedestrian is simulated based on the traffic simulation environment through the pedestrian behavior simulation model to obtain the target simulated pedestrian, which may include: first, the behavior of the initial simulated pedestrian is simulated based on the traffic simulation environment through the target pedestrian strategy model and the shadow proxy model, respectively, to obtain the first simulated pedestrian output by the target pedestrian strategy model and the second simulated pedestrian output by the shadow proxy model; then, the target simulated pedestrian is determined from the first simulated pedestrian and the second simulated pedestrian through the evaluation model.

[0047] The user can input evaluation rules into the initial evaluation model to obtain the evaluation model. In practical applications, the evaluation rules can be set according to actual needs. The user can determine the evaluation rules based on the characteristics of the target simulated pedestrian that the pedestrian behavior simulation model wants to output. For example, if the user wants the pedestrian behavior simulation model to output a target simulated pedestrian with high risk avoidance behavior, the evaluation rule can be: the higher the probability of collision with a vehicle, the lower the score of the target simulated pedestrian's behavior; the lower the probability of collision with a vehicle, the higher the score of the target simulated pedestrian's behavior.

[0048] In this implementation, after obtaining the target simulated pedestrian, if the first simulated pedestrian is determined as the target simulated pedestrian, the target reward information is output to the target pedestrian strategy model to instruct the target pedestrian strategy model to optimize according to the target reward information; if the second simulated pedestrian is determined as the target simulated pedestrian, the target penalty information is output to the target pedestrian strategy model to instruct the target pedestrian strategy model to optimize according to the target penalty information.

[0049] By continuously optimizing the target pedestrian strategy model through reinforcement learning algorithms, the target simulated pedestrian obtained through the pedestrian behavior simulation model can be made to better meet the evaluation rules input by the user, and ultimately a pedestrian behavior simulation model that satisfies the user can be obtained, thereby improving the authenticity of the constructed driving simulation scene.

[0050] In an embodiment of the present application, after obtaining the trained pedestrian behavior simulation model, the electronic device can input the traffic simulation environment and the initial simulated pedestrian into the trained pedestrian behavior simulation model to instruct the trained pedestrian behavior simulation model to output the target simulated pedestrian based on the traffic simulation environment and the initial simulated pedestrian.

[0051] It is understandable that different traffic simulation environments and initial simulated pedestrians will cause the pedestrian behavior simulation model to output different target simulated pedestrians. Compared to existing technologies where the simulated pedestrian's behavior does not change with changes in the traffic simulation environment, the target simulated pedestrian output by this method is more realistic.

[0052] In S103, the behavior of the initial simulated animal is simulated based on the traffic simulation environment through the animal behavior simulation model to obtain the target simulated animal; the animal behavior simulation model is obtained only through the reinforcement learning algorithm training.

[0053] In an embodiment of the present application, after obtaining the initial driving simulation scene, the electronic device may first obtain an animal behavior simulation model obtained only through training of a reinforcement learning algorithm.

[0054] The following is an explanation of "only through reinforcement learning algorithm" in this application: In contrast to the “through reinforcement learning algorithm and shadow agent method” in this application, the “only through reinforcement learning algorithm” in this application refers to a method of obtaining an animal behavior simulation model by training using only reinforcement learning algorithm without using shadow agent method.

[0055] In one possible implementation, the electronic device can use the following reinforcement learning algorithm to train an animal behavior simulation model. The details are as follows: First, the electronic device can obtain a pre-trained initial animal strategy model.

[0056] In this implementation, the pre-trained initial animal strategy model can be obtained through a preset reinforcement learning algorithm and trained based on a preset animal action strategy.

[0057] After obtaining the initial animal strategy model, the electronic device can optimize the initial animal strategy model through a reinforcement learning algorithm to obtain an animal behavior simulation model.

[0058] Specifically, the electronic device can simulate the behavior of the training animal according to the training environment through the initial animal strategy model to obtain the training animal behavior.

[0059] Afterwards, the electronic device can determine the penalty reward information of the trained animal behavior based on the preset penalty reward function and the trained animal behavior; the penalty reward function is used to reward the trained animal behavior for approaching the food source and moving away from the vehicle, and is used to punish the trained animal behavior for causing collision and / or excessive energy consumption.

[0060] For example, when a certain trained animal behavior is a behavior of approaching a food source, the penalty reward information of the trained animal behavior can be determined as reward information; when a certain trained animal behavior is a behavior of moving away from a vehicle, the penalty reward information of the trained animal behavior can be determined as reward information; when a certain trained animal behavior is a behavior that causes a collision and / or excessive energy consumption, the penalty reward information of the trained animal behavior can be determined as penalty information.

[0061] After obtaining the penalty reward information, the electronic device can feed the penalty reward information back to the initial animal strategy model to instruct the initial animal strategy model to optimize according to the penalty reward information to obtain an animal behavior simulation model.

[0062] In this implementation, since the initial animal strategy model used to train the animal behavior simulation model is obtained through training with a preset reinforcement learning algorithm, it can be considered that the target simulated animal obtained through the animal behavior simulation model has a high degree of randomness.

[0063] The following is a comparison between the pedestrian behavior simulation model and the animal behavior simulation model: Since the pedestrian behavior simulation model includes the target pedestrian strategy model and the shadow agent model, and since the target simulated pedestrian obtained by the shadow agent model has strong standardization, the target simulated pedestrian obtained by the target pedestrian strategy model has certain standardization and also certain randomness, while the animal behavior simulation model does not include the shadow agent model, it can be considered that the target simulated animal output by the animal behavior simulation model is compared with the target simulated pedestrian output by the target pedestrian strategy model. The target simulated animal output by the animal behavior simulation model has higher randomness and lower standardization, while the target simulated pedestrian output by the target pedestrian strategy model has lower randomness and higher standardization.

[0064] In actual driving scenarios, animal behavior is more random and less standardized, while pedestrian behavior is less random and more standardized. Therefore, the target simulated animals and target simulated pedestrians output by this method correspond to the behavior of pedestrians and animals in actual driving scenarios. It can be seen that the driving simulation scene constructed by this application is highly realistic.

[0065] In S104, a target driving simulation scene is constructed according to the traffic simulation environment, the target simulated pedestrians, and the target simulated animals.

[0066] In an embodiment of the present application, after obtaining the target simulated pedestrian and the target simulated animal, the electronic device can combine the traffic simulation environment, the target simulated pedestrian and the target simulated animal into a target driving simulation scene.

[0067] From the above, it can be seen that in the method for constructing a driving simulation scene provided in the embodiment of the present application, an initial driving simulation scene is first obtained, wherein the initial driving simulation scene includes at least a traffic simulation environment, an initial simulated pedestrian, and an initial simulated animal; then, the behavior of the initial simulated pedestrian is simulated based on the traffic simulation environment through the pedestrian behavior simulation model to obtain a target simulated pedestrian; the pedestrian behavior simulation model is trained by a reinforcement learning algorithm and a shadow agent method; then, the behavior of the initial simulated animal is simulated based on the traffic simulation environment through the animal behavior simulation model to obtain a target simulated animal; the animal behavior simulation model is trained only by a reinforcement learning algorithm; finally, a target driving simulation scene is constructed based on the traffic simulation environment, the target simulated pedestrian, and the target simulated animal. The target simulated pedestrians and target simulated animals in the driving simulation scene constructed by this method are both simulated based on the traffic simulation environment, so the constructed driving simulation scene is more similar to the real driving scene; in addition, in this method, the pedestrian behavior simulation model used to simulate the simulated pedestrians is trained by the reinforcement learning algorithm and the shadow agent method, and the animal behavior simulation model used to simulate the simulated animals is trained only by the reinforcement learning algorithm, which makes the behavior of the target simulated pedestrians more standardized and the behavior of the target simulated animals more random, which corresponds to the behavior of pedestrians and animals in the real driving scene, and thus can further improve the authenticity of the constructed driving simulation scene.

[0068] In one possible implementation, in order to make the pedestrian behavior simulation model and the animal behavior simulation model applicable to various traffic simulation environments and to improve the robustness of the pedestrian behavior simulation model and the animal behavior simulation model, the electronic device may first obtain the training environment data corresponding to each training environment, wherein the training environment may be used when training the pedestrian behavior simulation model and the animal behavior simulation model. Afterwards, the electronic device may perform policy generalization processing on the pedestrian behavior simulation model and the animal behavior simulation model based on the training environment data through the trained policy generalization model, and output the pedestrian behavior simulation model after policy generalization processing and the animal behavior simulation model after policy generalization processing, wherein the action difference between each target simulated pedestrian obtained by the pedestrian behavior simulation model after policy generalization processing according to the changing traffic simulation environment is lower than the first preset threshold, and the action difference between each target simulated animal obtained by the animal behavior simulation model after policy generalization processing according to the changing traffic simulation environment is lower than the second preset threshold. In actual applications, the first preset threshold and the second preset threshold can be set according to actual needs.

[0069] Through the above method, we can obtain the pedestrian behavior simulation model after strategy generalization processing and the animal behavior simulation model after strategy generalization processing. Since the action differences between the target simulated pedestrians and the action differences between the target simulated animals output by the pedestrian behavior simulation model after strategy generalization processing and the animal behavior simulation model after strategy generalization processing according to different traffic simulation environments are relatively low, it can be considered that the pedestrian behavior simulation model after strategy generalization processing and the animal behavior simulation model after strategy generalization processing can be applied to various different traffic simulation environments.

[0070] Based on the method for constructing a driving simulation scene provided in the above embodiment, the present application further provides a device for constructing a driving simulation scene to implement the above method embodiment. Figure 2 , Figure 2 This is a schematic diagram of a device for constructing a driving simulation scene provided in an embodiment of the present application. Figure 2 As shown, the driving simulation scene construction device 20 may include: an initial scene acquisition unit 21, a simulated pedestrian simulation unit 22, a simulated animal simulation unit 23 and a target scene construction unit 24. The initial scene acquisition unit 21 is used to acquire a pre-built initial driving simulation scene, which at least includes a traffic simulation environment, initial simulated pedestrians, and initial simulated animals.

[0071] The simulated pedestrian simulation unit 22 is used to simulate the behavior of the initial simulated pedestrian based on the traffic simulation environment through the pedestrian behavior simulation model to obtain the target simulated pedestrian; the pedestrian behavior simulation model is trained by the reinforcement learning algorithm and the shadow agent method.

[0072] The simulated animal simulation unit 23 is used to simulate the behavior of the initial simulated animal based on the traffic simulation environment through the animal behavior simulation model to obtain the target simulated animal; the animal behavior simulation model is obtained only through reinforcement learning algorithm training.

[0073] The target scene construction unit 24 is used to construct a target driving simulation scene according to the traffic simulation environment, target simulated pedestrians and target simulated animals.

[0074] Optionally, the pedestrian behavior simulation model includes a target pedestrian strategy model and a shadow proxy model, and the shadow proxy model is obtained by training a preset pedestrian walking strategy. The driving simulation scene construction device 20 may also include a first training unit, wherein: The first training unit is specifically used for: Get the pre-trained initial pedestrian policy model and shadow agent model; Through the initial pedestrian strategy model and the shadow proxy model, the same training pedestrian behavior is simulated according to the same training environment, and the training pedestrian behavior corresponding to the initial pedestrian strategy model and the training pedestrian behavior corresponding to the shadow proxy model are obtained; By comparing the training pedestrian behaviors corresponding to the initial pedestrian strategy model with those corresponding to the shadow agent model, the penalty reward information is determined. The penalty reward information is fed back to the initial pedestrian policy model to instruct the initial pedestrian policy model to optimize according to the penalty reward information to obtain the target pedestrian policy model.

[0075] Optionally, the pedestrian simulation unit 22 is specifically configured to: The target pedestrian strategy model and the shadow proxy model are used to simulate the behavior of the initial simulated pedestrian based on the traffic simulation environment, thereby obtaining the first simulated pedestrian output by the target pedestrian strategy model and the second simulated pedestrian output by the shadow proxy model. A target simulated pedestrian is determined from the first simulated pedestrian and the second simulated pedestrian by evaluating the model.

[0076] Optionally, the driving simulation scene construction device 20 may further include an optimization unit. The optimization unit is specifically used to: If the first simulated pedestrian is determined as the target simulated pedestrian, outputting target reward information to the target pedestrian strategy model to instruct the target pedestrian strategy model to optimize according to the target reward information; If the second simulated pedestrian is determined as the target simulated pedestrian, target penalty information is output to the target pedestrian strategy model to instruct the target pedestrian strategy model to perform optimization according to the target penalty information.

[0077] Optionally, the driving simulation scenario construction device 20 may further include a second training unit. The second training unit is specifically used for: Obtain a pre-trained initial animal strategy model; Through the initial animal strategy model, the behavior of the training animal is simulated according to the training environment to obtain the training animal behavior; Determining penalty reward information for the training animal behavior based on a preset penalty reward function and the training animal behavior; the penalty reward function is used to reward the training animal for behaviors such as approaching a food source and moving away from a vehicle, and is used to punish behaviors such as the training animal for behaviors that result in collisions and / or excessive energy consumption; The penalty reward information is fed back to the initial animal strategy model to instruct the initial animal strategy model to optimize according to the penalty reward information to obtain an animal behavior simulation model.

[0078] Optionally, the initial scene acquisition unit 21 is specifically configured to: Acquiring initial driving simulation scene data; the initial driving simulation scene data includes initial simulated pedestrian data and initial simulated animal data, and includes one or more of the following traffic simulation environment data: map data, road topology data, building data, obstacle data, traffic sign data, dynamic traffic signal data, dynamic weather data, dynamic vehicle data, and virtual dynamic sensor data; A dynamic initial driving simulation scenario is constructed based on initial simulated pedestrian data, initial simulated animal data, and traffic simulation environment data.

[0079] Optionally, the driving simulation scenario construction device 20 may further include a strategy generalization unit, wherein: The policy generalization unit is specifically used to: Obtain training environment data corresponding to each training environment; Through the trained policy generalization model, the pedestrian behavior simulation model and the animal behavior simulation model are subjected to policy generalization processing based on the training environment data, and the pedestrian behavior simulation model after policy generalization processing and the animal behavior simulation model after policy generalization processing are output; the action difference between each target simulated pedestrian obtained by the pedestrian behavior simulation model after policy generalization processing according to the changing traffic simulation environment is lower than the first preset threshold, and the action difference between each target simulated animal obtained by the animal behavior simulation model after policy generalization processing according to the changing traffic simulation environment is lower than the second preset threshold.

[0080] It should be noted that the information interaction, execution process and other contents between the above-mentioned units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0081] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 3 As shown, the electronic device 3 provided in this embodiment may include: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30. For example, a program corresponding to the method for constructing a driving simulation scene. When the processor 30 executes the computer program 32, the steps in the embodiment of the method for constructing a driving simulation scene are implemented, such as Figure 1 Alternatively, when the processor 30 executes the computer program 32, the functions of the modules / units in the embodiment corresponding to the device for constructing the driving simulation scene are realized, such as Figure 2 The functions of units 21 to 24 are shown.

[0082] Exemplarily, the computer program 32 may be divided into one or more modules / units, one or more modules / units being stored in the memory 31 and executed by the processor 30 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 32 in the electronic device 3. For example, the computer program 32 may be divided into an initial scene acquisition unit 21, a simulated pedestrian simulation unit 22, a simulated animal simulation unit 23, and a target scene construction unit 24. The specific functions of each unit can be found in Figure 2 The relevant descriptions in the corresponding embodiments are not repeated here.

[0083] Those skilled in the art will understand that Figure 3 This is merely an example of the electronic device 3 and does not constitute a limitation on the electronic device 3 . The electronic device 3 may include more or fewer components than shown in the figure, or may combine certain components, or may include different components.

[0084] The processor 30 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0085] The memory 31 can be an internal storage unit of the electronic device 3, such as the hard drive or memory of the electronic device 3. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, or flash card equipped on the electronic device 3. Furthermore, the memory 31 can include both the internal storage unit of the electronic device 3 and an external storage device. The memory 31 is used to store computer programs and other programs and data required by the electronic device. The memory 31 can also be used to temporarily store data that has been output or is about to be output.

[0086] Those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional units is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units as needed, that is, the internal structure of the device for constructing the driving simulation scene can be divided into different functional units to complete all or part of the functions described above. The functional units in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0087] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0088] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device implements the steps in the above-mentioned various method embodiments.

[0089] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0090] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0091] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for constructing a driving simulation scene, characterized in that: include: Acquire a pre-built initial driving simulation scene, wherein the initial driving simulation scene includes at least a traffic simulation environment, initial simulated pedestrians, and initial simulated animals; The behavior of the initial simulated pedestrian is simulated based on the traffic simulation environment using a pedestrian behavior simulation model to obtain a target simulated pedestrian; the pedestrian behavior simulation model is trained using a reinforcement learning algorithm and a shadow agent method; The behavior of the initial simulated animal is simulated based on the traffic simulation environment by using an animal behavior simulation model to obtain a target simulated animal; the animal behavior simulation model is obtained only by training with a reinforcement learning algorithm; A target driving simulation scene is constructed according to the traffic simulation environment, the target simulated pedestrians and the target simulated animals.

2. The method according to claim 1, characterized in that The pedestrian behavior simulation model includes a target pedestrian strategy model and a shadow proxy model. The shadow proxy model is trained by a preset pedestrian walking strategy. The pedestrian behavior simulation model is trained specifically by the following methods: Obtaining a pre-trained initial pedestrian policy model and the shadow agent model; Using the initial pedestrian strategy model and the shadow proxy model, the same training pedestrian behavior is simulated according to the same training environment to obtain the training pedestrian behavior corresponding to the initial pedestrian strategy model and the training pedestrian behavior corresponding to the shadow proxy model; Determining penalty reward information by comparing the training pedestrian behavior corresponding to the initial pedestrian strategy model with the training pedestrian behavior corresponding to the shadow proxy model; The penalty reward information is fed back to the initial pedestrian policy model to instruct the initial pedestrian policy model to be optimized according to the penalty reward information to obtain the target pedestrian policy model.

3. The method according to claim 2, characterized in that The pedestrian behavior simulation model also includes a trained evaluation model; the pedestrian behavior simulation model simulates the behavior of the initial simulated pedestrian based on the traffic simulation environment to obtain a target simulated pedestrian, including: The target pedestrian strategy model and the shadow proxy model are used to simulate the behavior of the initial simulated pedestrian based on the traffic simulation environment, thereby obtaining a first simulated pedestrian output by the target pedestrian strategy model and a second simulated pedestrian output by the shadow proxy model. The target simulated pedestrian is determined from the first simulated pedestrian and the second simulated pedestrian through the evaluation model.

4. The method according to claim 3, characterized in that After simulating the behavior of the initial simulated pedestrian based on the traffic simulation environment using the pedestrian behavior simulation model to obtain a target simulated pedestrian, the method further includes: If the first simulated pedestrian is determined to be the target simulated pedestrian, outputting target reward information to the target pedestrian strategy model to instruct the target pedestrian strategy model to perform optimization according to the target reward information; If the second simulated pedestrian is determined as the target simulated pedestrian, target penalty information is output to the target pedestrian strategy model to instruct the target pedestrian strategy model to perform optimization according to the target penalty information.

5. The method according to claim 1, wherein The animal behavior simulation model is specifically trained in the following manner: Obtain a pre-trained initial animal strategy model; Using the initial animal strategy model, simulating the behavior of the training animal according to the training environment to obtain the training animal behavior; Determining penalty-reward information for the training animal behavior based on a preset penalty-reward function and the training animal behavior; the penalty-reward function is used to reward the training animal for behaviors such as approaching a food source and moving away from a vehicle, and to penalize behaviors such as the training animal for behaviors that result in collisions and / or excessive energy consumption; The penalty reward information is fed back to the initial animal strategy model to instruct the initial animal strategy model to be optimized according to the penalty reward information to obtain the animal behavior simulation model.

6. The method according to claim 1, characterized in that The initial driving simulation scenario is constructed in the following way: Acquiring initial driving simulation scene data; the initial driving simulation scene data includes initial simulated pedestrian data and initial simulated animal data, and includes one or more of the following traffic simulation environment data: map data, road topology data, building data, obstacle data, traffic sign data, dynamic traffic signal data, dynamic weather data, dynamic vehicle data, and virtual dynamic sensor data; The dynamic initial driving simulation scene is constructed according to the initial simulated pedestrian data, the initial simulated animal data and the traffic simulation environment data.

7. The method according to any one of claims 1 to 6, characterized in that Also includes: Obtain training environment data corresponding to each training environment; Through the trained policy generalization model, the pedestrian behavior simulation model and the animal behavior simulation model are subjected to policy generalization processing based on the training environment data, and the pedestrian behavior simulation model after policy generalization processing and the animal behavior simulation model after policy generalization processing are output; the action difference between each target simulated pedestrian obtained by the pedestrian behavior simulation model after policy generalization processing according to the changing traffic simulation environment is lower than a first preset threshold, and the action difference between each target simulated animal obtained by the animal behavior simulation model after policy generalization processing according to the changing traffic simulation environment is lower than a second preset threshold.

8. A device for constructing a driving simulation scene, characterized in that: include: An initial scene acquisition unit is used to acquire a pre-built initial driving simulation scene, wherein the initial driving simulation scene at least includes a traffic simulation environment, initial simulated pedestrians, and initial simulated animals; a simulated pedestrian simulation unit, configured to simulate the behavior of the initial simulated pedestrian based on the traffic simulation environment using a pedestrian behavior simulation model to obtain a target simulated pedestrian; the pedestrian behavior simulation model is trained using a reinforcement learning algorithm and a shadow agent method; A simulated animal simulation unit, configured to simulate the behavior of the initial simulated animal based on the traffic simulation environment using an animal behavior simulation model to obtain a target simulated animal; the animal behavior simulation model is obtained only through training with a reinforcement learning algorithm; The target scene construction unit is used to construct a target driving simulation scene according to the traffic simulation environment, the target simulated pedestrians and the target simulated animals.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, each step in the method for constructing a driving simulation scenario as claimed in any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that When the computer program product is executed by a processor, each step in the method for constructing a driving simulation scenario as claimed in any one of claims 1 to 7 is implemented.

Images