Driving scene data generation method, device and electronic equipment

Abstract

This application provides a method, device, electronic device, computer-readable storage medium, and computer program product for generating data for a driving scene. The method includes: obtaining a first semantic description of the driving scene; processing the first semantic description using a generation model in a scene generation diffusion model to obtain a first semantic vector; wherein the scene generation diffusion model is an inverse operation model of a scene understanding model, and the scene understanding model is used to process scene data to obtain a semantic description; processing the first semantic vector using a diffusion model in the scene generation diffusion model to obtain predicted scene data corresponding to the first semantic description; wherein the predicted scene data corresponding to the first semantic description is used to train an autonomous driving model. Through this application, scene data can be automatically generated based on the semantic description, improving the efficiency of scene data generation and facilitating better training of the autonomous driving model.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to a method, device, electronic device, computer-readable storage medium, and computer program product for generating data of a driving scene. Background Art

[0002] Autonomous driving refers to the vehicle's ability to perform environmental perception, navigation planning, decision-making and control functions without direct human operation, thereby completing autonomous driving tasks safely and reliably. The development of autonomous driving technology is of great significance in improving road traffic safety, alleviating traffic congestion, reducing energy consumption and reducing environmental pollution.

[0003] With the development of computer technology, autonomous driving is currently usually achieved through specially trained autonomous driving models. The training of autonomous driving models requires a large amount of scene data. However, in the solutions provided by related technologies, scene data is mainly collected from actual driving scenarios or relies on manual configuration of scene data, resulting in low efficiency in scene data generation. Summary of the Invention

[0004] The present application provides a driving scene data generation method, device, electronic device, computer-readable storage medium and computer program product, which can automatically generate scene data based on semantic descriptions, improve the efficiency of scene data generation, and help better train autonomous driving models.

[0005] The technical solution of this application is achieved as follows:

[0006] This application provides a method for generating driving scene data, comprising:

[0007] Obtaining a first semantic description of the driving scene;

[0008] Processing the first semantic description using a generative model in a scene generation diffusion model to obtain a first semantic vector; wherein the scene generation diffusion model is an inverse operation model of a scene understanding model, and the scene understanding model is used to process scene data to obtain a semantic description;

[0009] Processing the first semantic vector using a diffusion model in the scene generation diffusion model to obtain predicted scene data corresponding to the first semantic description;

[0010] Among them, the predicted scene data corresponding to the first semantic description is used to train the autonomous driving model.

[0011] The present application provides a driving scene data generation device, comprising:

[0012] An acquisition module, configured to acquire a first semantic description of the driving scene;

[0013] a first reasoning module, configured to process the first semantic description using a generative model in a scene generation diffusion model to obtain a first semantic vector; wherein the scene generation diffusion model is an inverse operation model of a scene understanding model, which is configured to process scene data to obtain a semantic description;

[0014] The first reasoning module is further configured to process the first semantic vector using a diffusion model in the scene generation diffusion model to obtain predicted scene data corresponding to the first semantic description;

[0015] Among them, the predicted scene data corresponding to the first semantic description is used to train the autonomous driving model.

[0016] The present application provides an electronic device, including:

[0017] a memory for storing executable instructions;

[0018] The processor is configured to implement the driving scene data generation method provided in the present application when executing the executable instructions stored in the memory.

[0019] The present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the driving scenario data generation method provided in the present application.

[0020] The present application provides a computer program product, which includes executable instructions for causing a processor to execute the instructions to implement the driving scenario data generation method provided in the present application.

[0021] This application has the following beneficial effects:

[0022] This application obtains a first semantic description of the driving scene, processes the first semantic description through the generative model in the scene generation diffusion model, and obtains a first semantic vector, wherein the scene generation diffusion model is the inverse operation model of the scene understanding model, and the scene understanding model is used to process the scene data to obtain a semantic description. In this way, the first semantic description is restored in the direction of the driving scene through the generative model to obtain a first semantic vector that is more closely related to the driving scene and contains more comprehensive information. Then, the first semantic vector is processed by the diffusion model in the scene generation diffusion model to obtain predicted scene data corresponding to the first semantic description. In this way, on the basis of realizing the automatic generation of scene data, the generated scene data can be closer to the actual driving scene, which helps to improve the training effect of the autonomous driving model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 This is a schematic diagram of the architecture of the driving scenario data generation system provided by an embodiment of the present application;

[0025] Figure 2 This is a schematic diagram of the structure of the server provided in the embodiment of the present application;

[0026] Figure 3A This is a first flow chart of the method for generating driving scene data provided by an embodiment of the present application;

[0027] Figure 3B This is a second flow chart of the method for generating driving scene data provided by an embodiment of the present application;

[0028] Figure 3C This is a third flow chart of the method for generating driving scene data provided by an embodiment of the present application;

[0029] Figure 4 This is a schematic diagram of the architecture of the scene understanding model provided in the embodiment of the present application;

[0030] Figure 5 This is a schematic diagram of the architecture of the scene generation diffusion model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0032] In the following description, references to "some embodiments" describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. In the following description, the term "plurality" refers to at least two.

[0033] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0035] Before further explaining the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0036] 1) Driving scenarios: These refer to the specific environments and situations a vehicle encounters during driving, encompassing a variety of external and internal factors. Scenario data is a direct representation of the driving scenario and can take the form of images, videos, or structured data. Structured data uses parameters (such as key-value pairs) to describe the driving scenario. For example, structured data includes the coordinates of various objects in the driving scenario (e.g., the vehicle, surrounding vehicles, and the road).

[0037] 2) Scene understanding model: A language model based on the Encoder-Decoder architecture, which is used to process scene data to obtain semantic descriptions.

[0038] 3) Scene generation diffusion model: a language model implemented based on the decoder-only architecture, which is used to process the semantic description to obtain scene data. The scene generation diffusion model is the inverse operation model of the scene understanding model. Specifically, the scene generation diffusion model is opposite to the scene understanding model in the data flow direction, and is at least partially opposite in the model architecture. In an embodiment of the present application, the scene generation diffusion model includes a generation model and a diffusion model, wherein the generation model is used to process the semantic description to obtain a semantic vector; the diffusion model is used to perform a reverse process (or denoising processing) on ​​the basis of the semantic vector to obtain scene data.

[0039] 4) Simulation platform: Use simulation technology to construct virtual driving scenarios to simulate actual traffic conditions, road environments, vehicle behaviors, etc., in order to train autonomous driving models.

[0040] The present invention provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for generating driving scene data, which can automatically generate scene data based on semantic descriptions, thereby improving the efficiency of scene data generation. The following describes exemplary applications of the electronic device provided by the present invention. The electronic device provided by the present invention can be implemented as various types of terminal devices or as a server.

[0041] See also Figure 1 , Figure 1 This is an architectural diagram of a data generation system 100 for a driving scenario provided in an embodiment of the present application. A terminal device 400 is connected to a server 200 via a network 300, wherein the network 300 may be a wide area network or a local area network, or a combination of the two.

[0042] In some embodiments, assuming the electronic device is a terminal device, the driving scenario data generation method provided in the embodiments of the present application can be implemented by the terminal device. For example, the terminal device 400 obtains a first semantic description input by a user; processes the first semantic description using a generation model in a scene generation diffusion model to obtain a first semantic vector; processes the first semantic vector using a diffusion model in the scene generation diffusion model to obtain predicted scenario data corresponding to the first semantic description; and trains the autonomous driving model based on the predicted scenario data corresponding to the first semantic description.

[0043] In some embodiments, taking the electronic device as a server as an example, the driving scene data generation method provided in the embodiments of the present application can be implemented by the server. For example, server 200 obtains a first semantic description automatically generated according to specific rules; processes the first semantic description using a generation model in a scene generation diffusion model to obtain a first semantic vector; processes the first semantic vector using a diffusion model in the scene generation diffusion model to obtain predicted scene data corresponding to the first semantic description; and trains an autonomous driving model based on the predicted scene data corresponding to the first semantic description. In the above method, server 200 typically has powerful computing power and storage resources, capable of efficiently processing large amounts of data generation tasks.

[0044] In some embodiments, the driving scenario data generation method provided in the embodiments of the present application can be implemented collaboratively by a terminal device and a server. For example, the terminal device 400 obtains a first semantic description input by the user and sends the first semantic description to the server 200, so that the server 200 generates corresponding predicted scenario data based on the first semantic description. In this way, the user can freely define the first semantic description and have the server 200 quickly generate the corresponding predicted scenario data, ensuring that the predicted scenario data meets the user's needs.

[0045] In some embodiments, after completing the training of an autonomous driving model, an electronic device (such as a terminal device 400 or a server 200) may use the autonomous driving model (referring to the trained autonomous driving model) for an actual autonomous driving task. For example, the electronic device may obtain scene data of a driving scenario in which a vehicle is located, process the scene data using the autonomous driving model to obtain an autonomous driving strategy, and transmit the autonomous driving strategy to the vehicle for execution, thereby completing the autonomous driving task. Alternatively, the electronic device may directly deploy the autonomous driving model to the vehicle, and the vehicle may independently perform the autonomous driving task based on the autonomous driving model.

[0046] Take the electronic device provided in the embodiment of the present application as an example, which is a server. Figure 2 , Figure 2 is a structural diagram of the server 200 provided in an embodiment of the present application, Figure 2 The server 200 shown includes: at least one processor 210, a memory 250 and at least one network interface 220. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2 Various buses are labeled as bus system 240 .

[0047] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0048] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0049] The memory 250 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0050] In some embodiments, the memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0051] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0052] A network communication module 252 for reaching other computing devices via one or more (wired or wireless) network interfaces 220 , exemplary network interfaces 220 including Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB);

[0053] In some embodiments, the driving scene data generation device provided in the embodiments of the present application can be implemented in software. Figure 2 The driving scene data generation device 255 stored in the memory 250 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: an acquisition module 2551 and a first reasoning module 2552. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module are described below.

[0054] The data generation method for the driving scenario provided by the embodiment of the present application will be explained in combination with the exemplary application and implementation of the electronic device provided by the embodiment of the present application.

[0055] See also Figure 3A , Figure 3A This is a flow chart of a method for generating data of a driving scene according to an embodiment of the present application. The method for generating data of a driving scene according to an embodiment of the present application can be implemented by an electronic device, such as a terminal device and / or a server, and will be combined with Figure 3A The steps shown are explained.

[0056] In step 101, a first semantic description of a driving scene is obtained.

[0057] Here, the first semantic description describes the driving scene in text form. The embodiment of the present application does not limit the source of the semantic description involved. For example, it can be input by the user, generated by a scene understanding model, or automatically generated by specific rules (for example, setting the range of the semantic description and randomly generating the semantic description within the range).

[0058] In some embodiments, the first semantic description includes at least one of the following: the driving behavior of the vehicle; the static part of the driving scene; the dynamic part of the driving scene; the relationship between the vehicle and the static part; and the relationship between the vehicle and the dynamic part.

[0059] Here, the driving scene is observed from the perspective of the vehicle. Accordingly, the semantic description involved in the embodiment of the present application (such as the first semantic description) includes at least one of the following information:

[0060] 1) Driving behavior of the vehicle, such as lane changing, detouring, and U-turns.

[0061] 2) The static part of the driving scene, such as the status information of lane lines, traffic lights, and static obstacles around the vehicle.

[0062] 3) The dynamic part of the driving scene, such as the driving behavior of the surrounding vehicle (or other vehicles).

[0063] 4) The relationship between the ego vehicle and the static parts (or called the spatiotemporal relationship). For example, the lane the ego vehicle is in, the positional relationship between the ego vehicle and the surrounding traffic lights, and the positional relationship between the ego vehicle and the surrounding static obstacles.

[0064] 5) The relationship between the ego vehicle and the dynamic components (or the spatiotemporal relationship). For example, the positional relationship between the ego vehicle and surrounding vehicles; and / or the impact of surrounding vehicles on the ego vehicle, such as a surrounding vehicle cutting in and causing the ego vehicle to brake, slow down, or deviate from its original direction of travel.

[0065] The format of the semantic description can be set according to the needs of the actual application scenario, that is, the semantic description includes at least one of the above information 1) to 5). For example, the semantic description can be set to include the above information 1) to 5), so as to be able to comprehensively and carefully describe the driving scene, which helps to improve the quality of the subsequently generated scene data.

[0066] In step 102, the first semantic description is processed by a generative model in a scene generation diffusion model to obtain a first semantic vector; wherein the scene generation diffusion model is an inverse operation model of the scene understanding model, and the scene understanding model is used to process scene data to obtain a semantic description.

[0067] Here, the first semantic description is input into the generative model within the scene generation and diffusion model, resulting in a first semantic vector as output. The scene generation and diffusion model is the inverse of the scene understanding model, which processes scene data to generate a semantic description. Therefore, the generative model within the scene generation and diffusion model can transform the first semantic description into a driving scenario, generating a first semantic vector that is more closely aligned with the driving scenario and contains more comprehensive information.

[0068] It is worth noting that the scene generation diffusion model is the inverse operation model of the scene understanding model. This means that the scene generation diffusion model and the scene understanding model are opposite in data flow direction and at least partially opposite in model architecture. For example, the output layer of the scene understanding model is used as the input layer of the scene generation diffusion model, the activation function in the scene understanding model and the activation function in the scene generation diffusion model are inverse functions of each other, and the linear weighting function in the scene understanding model (referring to the linear weighting function between neurons) and the linear weighting function in the scene generation diffusion model are inverse functions of each other. As for which part of the model architecture the scene generation diffusion model and the scene understanding model are opposite, it can be determined based on experimental results and is not limited to this.

[0069] As an example, the present application provides the following embodiments: Figure 4 The following is a schematic diagram of the scene understanding model architecture. The model architecture of the scene understanding model includes, in sequence, a tokenizer, an embedding layer, an attention layer, a normalization layer (layerNorm), a fully connected layer (MLP), and a normalization layer (layerNorm). The input of the scene understanding model is scene data, and the output of the scene understanding model is a semantic description. It is worth noting that the tokenizer and embedding layer are used to extract text features from text-based scene data. If the scene data is image or video, the tokenizer and embedding layer can also be replaced with an image feature extraction model, such as a CNN model or YOLO model, to extract image features from the scene data.

[0070] After performing a reverse design on a part of the model architecture of the scene understanding model, a scene generation diffusion model can be obtained. As an example, the embodiment of the present application provides the following Figure 5 The figure shows an architectural diagram of the scene generation diffusion model. The scene generation diffusion model includes a generation model and a diffusion model. In the model architecture of the generation model, it includes a tokenizer, an embedding layer, a normalization layer (layerNorm), a fully connected layer (MLP), an attention layer, and a normalization layer (layerNorm). The input of the generation model is a semantic description, and the output of the generation model is a semantic vector; the input of the diffusion model is a semantic vector, and the output is scene data.

[0071] In step 103, the first semantic vector is processed by the diffusion model in the scene generation diffusion model to obtain predicted scene data corresponding to the first semantic description; wherein the predicted scene data corresponding to the first semantic description is used to train the autonomous driving model.

[0072] Here, the first semantic vector output by the generative model is input into the diffusion model, and the diffusion model reconstructs scene data based on the first semantic vector. For ease of distinction, the reconstructed scene data is named the predicted scene data corresponding to the first semantic description. Compared to directly inputting the semantic description into the diffusion model, the embodiment of the present application first converts the semantic description into a semantic vector through the generative model, making the semantic vector more closely connected to the driving scene and containing more comprehensive information. The semantic vector is then input into the diffusion model, thereby effectively improving the reconstruction effect of the diffusion model and making the resulting predicted scene data closer to the scene data of the actual driving scene.

[0073] It's worth noting that the diffusion model defines a process for gradually transforming the data distribution into a Gaussian noise distribution (the forward process). It then learns how to perform the inverse of this process (the reverse process), starting with pure noise and gradually "de-noising" it through a series of reverse steps, ultimately generating samples that approximate the original data distribution. Step 103 involves the reverse process of the diffusion model, namely, reconstructing the scene data through denoising.

[0074] After obtaining the predicted scenario data corresponding to the first semantic description, the autonomous driving model can be trained based on the predicted scenario data corresponding to the first semantic description. This embodiment of the application does not limit the type of autonomous driving model and the training method. For example, the autonomous driving model can be a planning and control model (planning and control model), and the training method can be reinforcement learning.

[0075] In some embodiments, after processing the first semantic vector through the diffusion model in the scene generation diffusion model to obtain the predicted scene data corresponding to the first semantic description, the driving scene data generation method also includes: creating a virtual driving scene in the simulation platform according to the predicted scene data corresponding to the first semantic description; executing the autonomous driving task in the virtual driving scene through the autonomous driving model; and training the autonomous driving model according to the execution results of the autonomous driving task.

[0076] Here, the predicted scene data corresponding to the first semantic description can be imported into the simulation platform, that is, a virtual driving scene is created in the simulation platform based on the predicted scene data corresponding to the first semantic description. Then, the autonomous driving model performs the autonomous driving task (i.e., controls the movement of the vehicle) in the virtual driving scene, and trains the autonomous driving model based on the execution results of the autonomous driving task. The execution results of the autonomous driving task, such as whether the vehicle collides or deviates from the road, can intuitively reflect the performance of the autonomous driving model. Therefore, the autonomous driving model can be trained in a targeted manner to improve its ability to cope with complex driving scenarios. In the above method, the training of the autonomous driving model is realized based on the simulation platform, without the need for actual vehicles and actual driving environments, which greatly reduces the training cost. At the same time, the simulation platform can quickly create a variety of virtual driving scenes, so that the autonomous driving model can be trained in large quantities in a short period of time, thereby accelerating the iteration and optimization of the model.

[0077] like Figure 3A As shown, the embodiment of the present application obtains a first semantic description of the driving scene, processes the first semantic description through the generative model in the scene generation diffusion model, and obtains a first semantic vector, wherein the scene generation diffusion model is the inverse operation model of the scene understanding model, and the scene understanding model is used to process the scene data to obtain a semantic description. In this way, the first semantic description is restored in the direction of the driving scene through the generative model to obtain a first semantic vector that is more closely related to the driving scene and contains more comprehensive information. Then, the first semantic vector is processed by the diffusion model in the scene generation diffusion model to obtain predicted scene data corresponding to the first semantic description. In this way, on the basis of realizing the automatic generation of scene data, the generated scene data can be closer to the actual driving scene, which helps to improve the training effect of the autonomous driving model.

[0078] In some embodiments, see Figure 3B , Figure 3B This is a flow chart of a method for generating data for a driving scenario according to an embodiment of the present application. Figure 3A , before step 102 ( Figure 3B (taking the process before step 101 as an example), steps 201 to 205 may also be performed.

[0079] In step 201 , a second semantic description of the driving scene is obtained.

[0080] Steps 102 to 103 describe the model inference phase of the scene generation diffusion model. Prior to this, the scene generation diffusion model can be trained in the model training phase to improve its processing effect.

[0081] During the model training phase of the scene generation diffusion model, a second semantic description of the driving scene is first obtained. It is worth noting that the semantic descriptions with different names (such as the first semantic description and the second semantic description) involved in the embodiments of the present application can describe the same driving scene or different driving scenes, and the same applies to the scene data.

[0082] In step 202, the second semantic description is processed by the generative model to obtain a second semantic vector.

[0083] Here, the second semantic description is input into the generation model to obtain a second semantic vector.

[0084] In step 203, the second semantic vector is processed by a diffusion model to obtain predicted scene data corresponding to the second semantic description.

[0085] Here, the second semantic vector output by the generation model is input into the diffusion model to obtain the predicted scene data corresponding to the second semantic description.

[0086] In step 204, a first loss value is calculated based on the second semantic description and the predicted scene data corresponding to the second semantic description.

[0087] Here, the first loss value is calculated based on the second semantic description and the predicted scene data corresponding to the second semantic description, wherein the higher the degree of conformity between the predicted scene data corresponding to the second semantic description and the second semantic description, the smaller the first loss value; and the lower the degree of conformity between the predicted scene data corresponding to the second semantic description and the second semantic description, the larger the first loss value.

[0088] In some embodiments, the above-mentioned calculation of the first loss value based on the second semantic description and the predicted scene data corresponding to the second semantic description can be achieved in the following manner: performing any one of the following processing: processing the predicted scene data corresponding to the second semantic description through a scene understanding model to obtain a predicted semantic description, and calculating the first loss value based on the second semantic description and the predicted semantic description; extracting the first valid information in the second semantic description, extracting the second valid information in the predicted scene data corresponding to the second semantic description, and calculating the first loss value based on the first valid information and the second valid information.

[0089] Here, two ways of calculating the first loss value are provided:

[0090] 1) The predicted scene data corresponding to the second semantic description is input into the scene understanding model to obtain a predicted semantic description, and the difference between the second semantic description and the predicted semantic description is calculated as the first loss value. It can be understood that the smaller the difference between the second semantic description and the predicted semantic description, the higher the degree of conformity between the predicted scene data corresponding to the second semantic description and the second semantic description. Method 1) Utilizes the scene understanding model to convert the predicted scene data corresponding to the second semantic description into a predicted semantic description, thereby measuring the loss value based on the semantic description.

[0091] 2) Extract the first valid information from the second semantic description, extract the second valid information from the predicted scene data corresponding to the second semantic description, and calculate the difference between the first valid information and the second valid information as the first loss value. The first valid information and the second valid information correspond to the same information type, for example, both include the number of vehicles per week. The first valid information can be obtained by performing text recognition on the second semantic description. Similarly, the second valid information can be obtained by performing recognition on the predicted scene data corresponding to the second semantic description, wherein, if the predicted scene data corresponding to the second semantic description is structured data, text recognition is used; if the predicted scene data corresponding to the second semantic description is an image or video, image recognition is used. In order to facilitate the calculation of the difference between the first valid information and the second valid information, the first valid information and the second valid information can be converted into vectors and then the difference is calculated. Method 2) Measuring the loss value based on valid information can focus on the most critical and effective parts of the driving scene.

[0092] In step 205 , the generation model and the diffusion model are trained according to the first loss value.

[0093] For example, back propagation is performed in the scene generation diffusion model (including the generation model and the diffusion model) according to the first loss value, and the parameters of the scene generation diffusion model are updated along the gradient descent direction during the back propagation process to realize the training of the scene generation diffusion model.

[0094] In some embodiments, the diffusion model can be pre-trained before the scene generation diffusion model training phase. This allows only fine-tuning to be performed during the scene generation diffusion model training phase, improving training efficiency and reducing training costs.

[0095] like Figure 3BAs shown, in the model training phase of the scene generation diffusion model, the embodiment of the present application calculates a first loss value based on the input second semantic description and the predicted scene data corresponding to the output second semantic description, and trains the scene generation diffusion model based on the first loss value, thereby effectively improving the performance of the scene generation diffusion model. Furthermore, in the model inference phase of the scene generation diffusion model, the reliability of the predicted scene data corresponding to the output first semantic description can be improved.

[0096] In some embodiments, see Figure 3C , Figure 3C This is a flow chart of a method for generating data for a driving scenario provided by an embodiment of the present application. Figure 3B Step 201 shown can be implemented through steps 301 to 303 , which will be described in conjunction with each step.

[0097] In step 301, second scene data of a driving scene is acquired.

[0098] The second semantic description used in the model training phase of the scenario generation diffusion model can be manually set or automatically generated by the scenario understanding model. For the latter, second scene data of the driving scene can be obtained first. For example, scene data of an actual driving scene can be obtained as the second scene data.

[0099] In step 302, the second scene data is processed by a scene understanding model to obtain a predicted semantic description corresponding to the second scene data.

[0100] Here, the second scene data is input into the scene understanding model to obtain the predicted semantic description corresponding to the second scene data output by the scene understanding model.

[0101] In some embodiments, before processing the second scene data through the scene understanding model, the driving scene data generation method also includes: obtaining third scene data of the driving scene and the corresponding third semantic description; processing the third scene data through the scene understanding model to obtain a predicted semantic description corresponding to the third scene data; calculating a second loss value based on the third semantic description and the predicted semantic description corresponding to the third scene data; and training the scene understanding model based on the second loss value.

[0102] Before the model inference phase of the scene understanding model, the scene understanding model can be trained in the model training phase. In the model training phase of the scene understanding model, first, third scene data of the driving scene and the corresponding third semantic description are obtained. For example, the third semantic description can be obtained by manually annotating the third scene data to ensure its accuracy; the third scene data is input into the scene understanding model to obtain a predicted semantic description corresponding to the third scene data output by the scene understanding model; the difference between the third semantic description and the predicted semantic description corresponding to the third scene data is calculated as a second loss value; and the scene understanding model is trained based on the second loss value.

[0103] The trained scene understanding model can output accurate semantic descriptions for the scene data, so it can enter the model inference stage, that is, use the trained scene understanding model for step 302 to improve the accuracy of the predicted semantic description corresponding to the second scene data.

[0104] In step 303, the predicted semantic description corresponding to the second scene data is determined as the second semantic description.

[0105] Here, the predicted semantic description corresponding to the second scene data output by the scene understanding model is determined as the second semantic description for use in training the scene generation diffusion model.

[0106] like Figure 3C As shown, the embodiment of the present application obtains second scene data of the driving scene, processes the second scene data through the scene understanding model, obtains a predicted semantic description corresponding to the second scene data, and determines the predicted semantic description corresponding to the second scene data as the second semantic description. Compared with semantic descriptions, scene data is easier to obtain. The embodiment of the present application automatically generates an accurate semantic description for the scene data, can quickly prepare a training data set for the scene generation diffusion model, and saves the labor cost of manually annotating the scene data.

[0107] The following continues to describe the exemplary structure of the driving scene data generation device 255 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2As shown, the software modules in the data generation device 255 of the driving scene stored in the memory 250 may include: an acquisition module 2551, used to obtain a first semantic description of the driving scene; a first reasoning module 2552, used to process the first semantic description through the generation model in the scene generation diffusion model to obtain a first semantic vector; the scene generation diffusion model is an inverse operation model of the scene understanding model, and the scene understanding model is used to process the scene data to obtain a semantic description; the first reasoning module 2552 is also used to process the first semantic vector through the diffusion model in the scene generation diffusion model to obtain predicted scene data corresponding to the first semantic description; wherein the predicted scene data corresponding to the first semantic description is used to train the autonomous driving model.

[0108] In some embodiments, the data generation device 255 for the driving scene also includes a first training module, which is used to: obtain a second semantic description of the driving scene; process the second semantic description through a generation model to obtain a second semantic vector; process the second semantic vector through a diffusion model to obtain predicted scene data corresponding to the second semantic description; calculate a first loss value based on the second semantic description and the predicted scene data corresponding to the second semantic description; and train the generation model and the diffusion model based on the first loss value.

[0109] In some embodiments, the first training module is also used to perform any of the following processing: processing the predicted scene data corresponding to the second semantic description through a scene understanding model to obtain a predicted semantic description, and calculating a first loss value based on the second semantic description and the predicted semantic description; extracting the first valid information in the second semantic description, extracting the second valid information in the predicted scene data corresponding to the second semantic description, and calculating the first loss value based on the first valid information and the second valid information.

[0110] In some embodiments, the driving scene data generation device 255 also includes a second reasoning module, which is used to: obtain second scene data of the driving scene; process the second scene data through a scene understanding model to obtain a predicted semantic description corresponding to the second scene data; and determine the predicted semantic description corresponding to the second scene data as a second semantic description.

[0111] In some embodiments, the data generation device 255 for the driving scene also includes a second training module, which is used to: obtain third scene data of the driving scene and a corresponding third semantic description; process the third scene data through a scene understanding model to obtain a predicted semantic description corresponding to the third scene data; calculate a second loss value based on the third semantic description and the predicted semantic description corresponding to the third scene data; and train the scene understanding model based on the second loss value.

[0112] In some embodiments, the first semantic description includes at least one of the following: the driving behavior of the vehicle; the static part of the driving scene; the dynamic part of the driving scene; the relationship between the vehicle and the static part; and the relationship between the vehicle and the dynamic part.

[0113] In some embodiments, the driving scene data generation device 255 also includes a third training module, which is used to: create a virtual driving scene in a simulation platform based on the predicted scene data corresponding to the first semantic description; perform an autonomous driving task in the virtual driving scene through an autonomous driving model; and train the autonomous driving model based on the execution results of the autonomous driving task.

[0114] The present invention provides a computer program product or computer program, which includes executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, causing the electronic device to perform the driving scenario data generation method described in the present invention.

[0115] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the data generation method for the driving scene provided in the embodiment of the present application.

[0116] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0117] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0118] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0119] As an example, executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0120] The above are merely examples of the present application and are not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A method for generating driving scene data, characterized in that: include: Obtaining a first semantic description of the driving scene; Processing the first semantic description using a generative model in a scene generation diffusion model to obtain a first semantic vector; wherein the scene generation diffusion model is an inverse operation model of a scene understanding model, and the scene understanding model is used to process scene data to obtain a semantic description; Processing the first semantic vector using a diffusion model in the scene generation diffusion model to obtain predicted scene data corresponding to the first semantic description; Among them, the predicted scene data corresponding to the first semantic description is used to train the autonomous driving model.

2. The method according to claim 1, characterized in that Before processing the first semantic description using the generative model in the scene generation diffusion model, the method further includes: Obtaining a second semantic description of the driving scene; Processing the second semantic description using the generative model to obtain a second semantic vector; Processing the second semantic vector using the diffusion model to obtain predicted scene data corresponding to the second semantic description; Calculating a first loss value according to the second semantic description and the predicted scene data corresponding to the second semantic description; The generation model and the diffusion model are trained according to the first loss value.

3. The method according to claim 2, characterized in that The calculating the first loss value according to the second semantic description and the predicted scene data corresponding to the second semantic description includes: Perform any of the following: Processing the predicted scene data corresponding to the second semantic description through the scene understanding model to obtain a predicted semantic description, and calculating a first loss value based on the second semantic description and the predicted semantic description; Extract the first valid information from the second semantic description, extract the second valid information from the predicted scene data corresponding to the second semantic description, and calculate the first loss value based on the first valid information and the second valid information.

4. The method according to claim 2, characterized in that The obtaining of the second semantic description of the driving scene includes: Acquire second scene data of the driving scene; Processing the second scene data using the scene understanding model to obtain a predicted semantic description corresponding to the second scene data; The predicted semantic description corresponding to the second scene data is determined as the second semantic description.

5. The method according to claim 4, characterized in that Before processing the second scene data by the scene understanding model, the method further includes: Obtaining third scene data of the driving scene and a corresponding third semantic description; Processing the third scene data using the scene understanding model to obtain a predicted semantic description corresponding to the third scene data; Calculating a second loss value according to the third semantic description and the predicted semantic description corresponding to the third scene data; The scene understanding model is trained according to the second loss value.

6. The method according to any one of claims 1 to 5, characterized in that The first semantic description includes at least one of the following: the driving behavior of the vehicle; the static part of the driving scene; the dynamic part of the driving scene; the relationship between the vehicle and the static part; and the relationship between the vehicle and the dynamic part.

7. The method according to any one of claims 1 to 5, characterized in that After processing the first semantic vector using the diffusion model in the scene generation diffusion model to obtain predicted scene data corresponding to the first semantic description, the method further includes: Creating a virtual driving scenario in a simulation platform based on the predicted scenario data corresponding to the first semantic description; executing an autonomous driving task in the virtual driving scenario by using the autonomous driving model; The autonomous driving model is trained according to the execution result of the autonomous driving task.

8. A driving scene data generation device, characterized in that: include: An acquisition module, configured to acquire a first semantic description of the driving scene; a first reasoning module, configured to process the first semantic description using a generative model in a scene generation diffusion model to obtain a first semantic vector; wherein the scene generation diffusion model is an inverse operation model of a scene understanding model, which is configured to process scene data to obtain a semantic description; The first reasoning module is further configured to process the first semantic vector using a diffusion model in the scene generation diffusion model to obtain predicted scene data corresponding to the first semantic description; Among them, the predicted scene data corresponding to the first semantic description is used to train the autonomous driving model.

9. An electronic device, characterized in that: include: a memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 7 when executing the executable instructions stored in the memory.

10. A computer-readable storage medium, characterized in that Executable instructions are stored, and when executed by a processor, they are used to implement the method described in any one of claims 1 to 7.

11. A computer program product, characterized in that The method comprises executable instructions for implementing the method according to any one of claims 1 to 7 when executed by a processor.

Images