Control method, training method, electronic device, chip, vehicle and medium

Abstract

The present disclosure relates to a control method, a training method, an electronic device, a chip, a vehicle and a medium, the control method comprising: obtaining first scene data collected by a movable device at a first time; obtaining first condition information based on the first scene data; performing a plurality of generation tasks guided by the first condition information through a generation model to obtain a plurality of first generation objects corresponding to the plurality of generation tasks; wherein the plurality of first generation objects are associated in a processing link for controlling the movable device to move based on the first scene data; and controlling the movable device to move based on at least part of the plurality of first generation objects.

Description

Technical Field

[0001] This disclosure relates to the field of automatic control technology, and more specifically, to a control method for a mobile device, a training method for an automatic control system, electronic devices, chips, vehicles, and computer-readable storage media. Background Technology

[0002] Vehicles and other mobile devices can move automatically without human intervention under the control of an automatic control system. Specifically, the automatic control system determines control parameters for the mobile device based on scene data collected by the device and controls its movement according to these parameters. Regarding the processing chain of the automatic control system controlling the movement of the mobile device based on scene data, if the processing chain is too long, information loss and error accumulation will occur during information transmission. Conversely, shortening the processing chain by compressing intermediate links may result in insufficient information mining, leading to the loss of useful information. Both of these factors will affect the performance of the automatic control system. Summary of the Invention

[0003] In view of this, the present disclosure proposes a new technical solution that is beneficial to improving the performance of automatic control systems.

[0004] According to a first aspect of this disclosure, a vehicle control method is provided, the method comprising:

[0005] Acquire first-scene data collected by mobile devices in real time;

[0006] First condition information is obtained based on the first scenario data;

[0007] Guided by the first condition information, a generative model executes each of the multiple generative tasks to obtain multiple first generated objects that correspond one-to-one with the multiple generative tasks; wherein, the multiple first generated objects are associated on the processing link that controls the movement of the mobile device based on the first scene data.

[0008] The mobile device is controlled to move based on at least a portion of the plurality of first generated objects.

[0009] According to a second aspect of this disclosure, a training method for an automatic control system is also provided, the automatic control system including a generative model for performing multiple generative tasks, the multiple generative tasks being associated with controlling the movement of a mobile device based on scene data, the training method comprising:

[0010] Obtain a training sample set; wherein, the training samples in the training sample set include scene data and a truth set of generated objects for the multiple generation tasks obtained based on the scene data;

[0011] Noise is added to the truth set of the generated objects of the training samples to obtain a dataset with added noise;

[0012] Conditional information is generated based on the scene data of the training samples;

[0013] The conditional information corresponding to the training samples and the dataset with added noise are input into the generative model, so that the generative model predicts the noise added to the dataset based on the conditional information, and obtains the noise prediction value.

[0014] A loss function is constructed using the first loss between the predicted noise value and the true noise value, and the model parameters of the generated model are updated.

[0015] According to a third aspect of this disclosure, an electronic device is provided according to some embodiments, the electronic device comprising:

[0016] processor;

[0017] Memory used to store processor-executable instructions;

[0018] The processor is configured to implement the method according to the first and / or second aspects of this disclosure when executing instructions stored in the memory.

[0019] According to a fourth aspect of this disclosure, a chip is provided according to some embodiments, the chip including:

[0020] Storage unit for storing computer programs; and,

[0021] A processing unit configured to implement the method according to the first and / or second aspects of this disclosure when executing a computer program stored in the storage unit.

[0022] According to a fourth aspect of this disclosure, a mobile device is provided according to some embodiments, which may include an electronic device according to a third aspect of this disclosure; or, include a chip according to a fourth aspect of this disclosure; or, may include:

[0023] processor;

[0024] Memory used to store processor-executable instructions;

[0025] The processor is configured to implement the method according to the first and / or second aspects of this disclosure when executing instructions stored in the memory.

[0026] According to a fifth aspect of this disclosure, a non-volatile computer-readable storage medium is also provided, having stored thereon computer program instructions that, when executed by a processor, implement the method according to a first aspect of this disclosure.

[0027] According to embodiments of this disclosure, a vehicle control method is provided. This method executes multiple generation tasks related to the movement trajectory of a mobile device based on scene data control, using the same generative model. This enables the training of the generative model to jointly optimize multiple generation tasks. On the one hand, by compressing the length of the processing link, it reduces the accumulation of errors and information loss caused by information transmission. On the other hand, by learning the correspondence between multiple tasks, it avoids information loss caused by insufficient information mining, thereby improving the performance of the automatic control system.

[0028] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0030] Figure 1 This is a schematic diagram of an intelligent connected system to which the methods provided in the embodiments of this disclosure can be applied;

[0031] Figure 2 It is based on Figure 1 The illustrated embodiment provides a schematic diagram of a mobile device;

[0032] Figure 3 This is a flowchart illustrating a control method according to some embodiments;

[0033] Figure 4 This is a flowchart illustrating a control method according to other embodiments;

[0034] Figure 5 This is a schematic diagram of the training and inference processes of an automatic control system according to some embodiments;

[0035] Figure 6 This is a flowchart illustrating a training method according to some embodiments;

[0036] Figure 7 This is a flowchart illustrating a training method according to some other embodiments;

[0037] Figure 8 This is a schematic diagram of the hardware structure of a chip according to some embodiments;

[0038] Figure 9This is a schematic diagram of the hardware structure of an electronic device according to some embodiments;

[0039] Figure 10 This is a schematic diagram of the hardware structure of a mobile device according to some embodiments. Detailed Implementation

[0040] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0041] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0042] Techniques, methods, and apparatus known to those skilled in the art in the relevant field may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

[0043] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0044] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0045] It should be noted that all actions involving the collection, storage, use, processing, transmission, provision, disclosure, and deletion of data in this disclosure are carried out in accordance with the relevant data protection laws and regulations of the country or region where the data is located, and with the full authorization of the relevant data owner.

[0046] This disclosure relates to a technical solution for an automatic control system that controls the automatic movement of a mobile device within a given scene based on scene data collected by the mobile device. The automatic control system can be deployed on the mobile device, on a server communicating with the mobile device, or on both the mobile device and the server; no limitation is made herein. Taking a vehicle as an example, the automatic control system used in a vehicle is an autonomous driving system.

[0047] In one related technology, the overall task of controlling the movement of a mobile device based on scene data can be decomposed into multiple tasks to modularize the automatic control system. These tasks include, for example, a perception task based on scene data, a trajectory prediction task based on scene perception results to predict the movement trajectories of other mobile devices, pedestrians, and other dynamic scene elements, and a trajectory planning task based on the trajectory prediction results of dynamic scene elements to plan the movement trajectory of the mobile device. The control parameters of the execution components acting on the mobile device are determined based on the planned movement trajectory, and then the execution components drive the mobile device to move according to the planned movement trajectory. The control parameters include, for example, throttle coefficient, braking coefficient, and steering wheel angle, etc., which will not be elaborated here. In this related technology, the processing link of the automatic control system controlling the movement of the mobile device based on scene data includes multiple tasks completed sequentially. The automatic control system can be configured to complete different tasks on the processing link through different modules, where the output of one module is the input of another. For example, the automatic control system performs a perception task through a target detection module and outputs the scene perception result (or target detection result) to a trajectory prediction module used to perform the trajectory prediction task. The automatic control system completes the movement control of the mobile device in a task-serialized manner.

[0048] This task-chaining approach leads to error accumulation because trajectory planning results are generated by multiple tasks. Furthermore, automatic control systems typically use query vectors for inter-module information transfer, but this transfer only includes target information with defined labels, potentially causing information loss and impacting system performance. Additionally, each task has its own optimization objective, and the learned labels for different tasks differ significantly in geometric shape and semantic meaning. This makes joint optimization and coordination of these modules difficult; conversely, even if these modules are jointly optimized, stable performance of the automatic control system in application cannot be guaranteed.

[0049] In another related technology, the automatic control system can also be configured to extract feature vectors from a scene image through a feature extraction network, and then directly obtain control parameters by using these feature vectors as input to a control parameter prediction network. For this type of automatic control system, the ground truth labels of the corresponding training samples will be the control parameters. While this can shorten the processing chain, it also leads to a deterioration in the constraints on the inherent processing logic of the automatic control system. This may cause the automatic control system's information mining to be incompatible with the actual scene, resulting in the automatic control system failing to exhibit stable performance due to a lack of effective information, or even exhibiting relatively basic errors.

[0050] To address the aforementioned problems in related technologies, this disclosure proposes a method that executes multiple tasks on a processing link using a single model, thereby reducing error accumulation, information loss, or information missing, and thus improving the performance of the automatic control system in at least one aspect such as stability, real-time performance, and security.

[0051] The method described in this disclosure can be applied to intelligent connected systems. Figure 1 An intelligent connected system 100 that can apply the methods provided in the embodiments of this disclosure is illustrated. Figure 1 As shown, the intelligent connected system 100 may include: a mobile device 101, a server 102, and a user terminal 103.

[0052] In some examples, the mobile device 101 can be a vehicle with autonomous driving capabilities, a robot capable of autonomous movement, etc. Autonomous driving, also known as driverless or intelligent driving, refers to a vehicle with autonomous driving capabilities that can perform driving tasks such as environmental perception, decision-making, planning, and control execution. The levels of autonomous driving can refer to the vehicle intelligence classification standards established by the Society of Automotive Engineers (SAE), for example, L0 is manual driving, L1 is driver assistance, L2 is partial autonomous driving, L3 is conditional autonomous driving, L4 is highly automated driving, and L5 is fully automated driving. The above classification of autonomous driving levels is merely an example, and this disclosure does not limit the classification standards and levels of autonomous driving.

[0053] In some examples, server 102 can be a single server or a distributed server cluster consisting of multiple servers, and its deployment can include local servers and / or cloud servers. Server 102 can communicate with mobile device 101 and / or user terminal 103 via a communication network, providing various services to mobile device 101 and / or user terminal 103. For example, the server can receive sensing data sent by mobile device 101, and provide services such as high-precision maps, data analysis, and decision planning to mobile device 101. Alternatively, the server can receive query commands or control commands sent by user terminal 102, providing corresponding services to the user.

[0054] In some examples, the user terminal 103 can be any form of electronic device that provides services to the user, such as a personal computer, laptop, smart tablet, smartphone, smart wearable device, etc. The user can interact with the mobile device or server through the human-computer interaction terminal configured on the mobile device 101, or through the user terminal 103. For example, the user terminal can query the status and / or parameters of the mobile device, or control the mobile device to perform set tasks and / or modify configuration parameters, etc. The user terminal 103 runs an application based on the intelligent network system to achieve interaction with the mobile device or server. This application can be a local application, a web application, or a mini-program, etc., without limitation.

[0055] In some examples, the aforementioned application running on user terminal 103 can provide authentication or authorization services to users. Users who are successfully authenticated and granted the corresponding permissions can query and / or control the mobile device within the scope of the granted permissions.

[0056] The mobile device 101, server 102, and user terminal 103 can communicate via a communication link provided by communication network 104. This communication network 104 can include one or more networks of any type, such as the Internet, Local Area Network (LAN), Wide Area Network (WAN), Virtual Private Network (VPN), Public Switched Telephone Network (PSTN), satellite communication network, Wi-Fi, 2G, 3G, 4G, 5G, 6G, NB-IoT, eMTC, infrared, Bluetooth, NFC, or a combination of these networks. The communication networks between mobile device 101 and server 102, between user terminal 103 and server 102, and between user terminal 103 and mobile device 101 can be the same or different.

[0057] It should be noted that, Figure 1 The structure of the intelligent connected system 100 shown is merely illustrative. The intelligent connected system in this embodiment is not limited to the above structure and may include more or fewer devices as needed, and the devices may be combined or separated. For example, the intelligent connected system may not include... Figure 1 The user terminal in the middle.

[0058] The method provided in this disclosure can be implemented by the mobile device 101, the server 102, or both. Furthermore, those skilled in the art should understand that the detection automatic control system method and optimization method provided in this disclosure can also be implemented by other devices with data processing capabilities, independent of the intelligent network system; simply importing the scene data collected by the mobile device into that device is sufficient, and no limitation is made herein.

[0059] Figure 2 It is based on Figure 1 The illustrated embodiment provides a schematic diagram of a mobile device 101. As shown... Figure 2 As shown, the mobile device 101 may include a sensing component 1011, a computing platform 1012, an execution component 1013, etc. The sensing component 1011, the computing platform 1012, and the execution component 1013 may be connected via a bus or other means.

[0060] In some examples, the sensing component 1011 can be used to collect information about the mobile device itself or its external environment. The sensing component 1011 may include a visual sensing unit and a motion sensing unit. The visual sensing unit may include one or more cameras. The motion sensing unit may include a wheel speedometer and / or an inertial measurement unit (IMU). In other examples, the sensing component 1011 may also include radar, a positioning and navigation unit, etc., without limitation. The radar may include at least one of lidar, millimeter-wave radar, ultrasonic radar, or other radar types. The wheel speedometer can be of any type, such as a magnetoelectric wheel speedometer, an photoelectric wheel speedometer, a mechanical wheel speedometer, a Hall effect wheel speedometer, or a visual wheel speedometer. The positioning and navigation unit may include at least one of a GPS system, a BeiDou system, or other global positioning systems.

[0061] In some examples, the computing platform 1012 may include a computing-capable device for processing the sensing information collected by the sensing component 1011 to obtain control information, and sending corresponding control commands to the execution component 1013 to cause the execution component 1013 to perform corresponding actions, thereby realizing the control of the mobile device 101. For example, the computing platform 1012 can perform Simultaneous Localization and Mapping (SLAM), path planning, and behavior decision-making on the mobile device, thereby realizing autonomous control of the mobile device. The computing platform 1012 may include at least one processor and at least one memory, and each processor may execute instructions stored in the memory individually or jointly to implement the methods provided in the embodiments of this disclosure. The processor in the embodiments of this disclosure may include at least one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), a System on Chip (SOC), an Application Specific Integrated Circuit (ASIC), a Microcontroller Unit (MCU), or other processors. Memory can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. In addition to storing instructions, memory can also store data, such as high-definition maps, path information, and data on the location, direction, and speed of mobile devices. The data stored in memory can be accessed and used by the processor.

[0062] In some examples, the computing platform of a mobile device can perform computing tasks independently or communicate with a server to complete computing tasks. For example, the computing platform of a mobile device can cooperate with a server to complete the corresponding computing tasks.

[0063] The computing platform 1012 can be located in the mobile device 101. Some or all of the computing platform 1012 can also be located in the server corresponding to the mobile device. For example, some functions of the computing platform 1012 with high real-time requirements can be located in the mobile device, while other functions with low real-time requirements can be located in the server corresponding to the mobile device.

[0064] In some examples, the execution component 1013 is used to perform corresponding actions based on the control of the computing platform 1012, enabling the mobile device 101 to complete the movement task. The execution component 1013 may include, for example, a power component, a braking component, a transmission component, a steering component, etc.

[0065] It should be noted that, Figure 2 The structure of the mobile device 101 shown is merely illustrative. The mobile device in this embodiment is not limited to the above structure and may include more or fewer components as needed. The device may also be combined or disassembled. For example, the mobile device may not include the aforementioned computing platform. Furthermore, the mobile device may also include communication components, interface components, multimedia components, input components, output components, etc.

[0066] The following is combined with Figure 1 and Figure 2 Various embodiments of this disclosure are described below.

[0067] <First Embodiment>

[0068] Figure 3 A method for controlling a mobile device according to some embodiments is illustrated. This method can, for example, be controlled by... Figure 1 The medium-sized mobile device 101 can also be implemented by Figure 1 The implementation can be carried out by server 102, or it can be implemented jointly by mobile device 101 and server 102, specifically by an automatic control system deployed on at least one of mobile device 101 and server 102. For example... Figure 3 As shown, the control method of this embodiment may include the following steps S310 to S340.

[0069] Step S310: Obtain the first scene data collected by the mobile device in the first moment.

[0070] In this embodiment, the mobile device can collect scene data through its onboard sensing components. The sensing components include at least one of a camera and radar. Accordingly, the scene data can include at least one of image data collected by the camera and point cloud data collected by the radar.

[0071] Scene data reflects the scene information of the location of a mobile device and can be used to determine scene element information, which includes static element information and dynamic element information. Static element information includes, for example, roads, lanes, traffic lights, signs, road markings, etc.; dynamic element information includes, for example, other mobile devices, pedestrians, and other traffic participants.

[0072] In this embodiment, scene data collected by the mobile device at the same time is referred to as a frame of data. The sensing component 1011 of the mobile device 101 can collect scene data at a set sampling frequency. Data in the same frame corresponds to the same timestamp, and data in different frames corresponds to different timestamps. The timestamp is used to mark the data collection time. It is understood that data in the same frame corresponding to the same timestamp does not mean that the collection time of these data is strictly aligned. In practical applications, a time window for alignment can be set. That is to say, data collected within a time window can be considered to correspond to the same timestamp.

[0073] The first time in step S310 can be any acquisition time determined based on the sampling frequency. Steps S310 to S340 are one processing of a frame of data acquired by the automatic control system for the mobile device. In practical applications, the automatic control system can repeatedly execute steps S310 to S340 for each frame of data to achieve continuous movement control of the mobile device.

[0074] In some examples, scene data can be used to generate map fragments reflecting the scene's conditions. These map fragments can be bird's-eye-view (BEV) maps, and BEV feature maps reflecting scene characteristics can be further obtained from the BEV images. Depending on the detection requirements, this map can be a two-dimensional view or a three-dimensional view with height information. Bird's-eye-view presentations scene information in a simple way, facilitating scene feature extraction.

[0075] In some examples, scene data may include scene images captured by one or more cameras. Taking a vehicle as an example, a vehicle may be equipped with six cameras, including a front-view camera, a rear-view camera, and four additional cameras located on both sides. The first scene data may include multiple scene images captured by multiple cameras in the first moment. By stitching these scene images together, a map fragment reflecting the scene can be obtained.

[0076] In another example, the map fragment can also be obtained based on point cloud data, which is not limited here.

[0077] Step S320: Obtain first condition information based on the first scenario data.

[0078] In this embodiment, since scene data can reflect the scene information of the scene where the mobile device is located, when generating condition information based at least on scene data, the condition information can provide scene information to the generation model. This allows the generation model to construct a generation object that conforms to the scene information of the scene where the mobile device is located when performing related tasks of mobile control, thereby realizing mobile control of the mobile device.

[0079] In this embodiment, the conditional information generated based on the first scene data collected by the mobile device at a first time is referred to as the first conditional information. Similarly, the conditional information generated based on the second scene data collected by the mobile device at a second time is referred to as the second conditional information. The second time is prior to the first time; for example, the second time and the first time can be two adjacent collection times.

[0080] In this embodiment, the condition information, as part of the input to the generative model, can be input to the generative model in the form of a vector so that the generative model can use the condition information.

[0081] To reduce computational resource consumption and improve the training and inference speed of the generative model, in some examples, the first scene data can be converted into first scene features, so that the first scene features are at least used as first conditional information input into the generative model. These first scene features are, for example, represented in the form of a BEV feature map.

[0082] In this example, scene features can be extracted from the first scene data using a pre-trained feature extraction model, resulting in the first scene features. This feature extraction model, for example, can use a BEV generator (BEVFormer) to convert a panoramic image composed of multiple scene images acquired in the first instance into a BEV feature map representing the first scene features. BEVFormer can efficiently extract spatial features through a combination of convolutional layers and transformer layers to obtain scene features related to the scene data.

[0083] In this example, to improve training efficiency and the stability of the trained model, the generative model and the feature extraction model that provides scene features to the generative model can be jointly trained. That is, the model parameters of the feature extraction model and the generative model can be determined through joint training using the same training samples. Here, since the scene features output by the feature extraction model are input into the generative model as at least partial conditional information, the loss generated by the feature extraction model during joint training can be included in the first loss generated by the generative model. In other words, the loss generated by the feature extraction model during joint training is reflected in the first loss, allowing for joint training of the generative model and the feature extraction model using the first loss.

[0084] Furthermore, in converting the first scene data into first scene features, historical scene features can be fused to capture the temporal information of the dynamic scene, thereby enhancing the understanding of the dynamic scene and improving the accuracy and spatiotemporal representation of feature extraction. Here, historical scene features are scene features obtained based on historical scene data, the collection time of which is before the first time. Combining this with an example of feature extraction using a feature extraction model, the feature extraction model can, for example, fuse historical scene features based on a Convolutional Long Short-Term Memory (ConvLSTM) network to extract features from the first scene data, thus obtaining the first scene features.

[0085] Step S330: Guided by the first condition information, a generation model is used to execute each of the multiple generation tasks set, resulting in multiple first generation objects that correspond one-to-one with the multiple generation tasks.

[0086] This embodiment uses the same generation model and the same condition information to execute each of the multiple generation tasks. In other words, in one operation for a frame of data, the generation model synchronously executes multiple generation tasks guided by the same condition information. In this embodiment, "multiple" means "at least two". For example, multiple generation tasks means at least two generation tasks.

[0087] In this embodiment, the generated object generated by the generative model guided by the first conditional information is referred to as the first generated object. Similarly, the generated object generated by the generative model guided by the second conditional information will be referred to as the second generated object.

[0088] In this embodiment, the generation task is related to controlling the movement of a mobile device based on scene data, and the generation objects of multiple generation tasks are associated on the processing link of controlling the movement of a mobile device based on scene data. Accordingly, the multiple first generation objects generated by the generation model for multiple generation tasks guided by the first condition information are associated on the processing link of controlling the movement of a mobile device based on the first scene data.

[0089] Since the processing chain reflects the processing sequence, correlation on the processing chain can be manifested as correlation based on processing sequence. Correlation on the processing chain can include: the i-th first generated object is used in the processing chain to obtain the (i+1)-th first generated object, where i is an integer greater than or equal to 1 and less than or equal to (N-1), and N is the number of first generated objects obtained. That is, for multiple first generated objects generated by the generative model, every two adjacent first generated objects form an object group, and the two objects in the object group are the input and output of a processing node on the processing chain, respectively. From another perspective, correlation on the processing chain can also mean that multiple first generated objects can be connected in series to form at least a part of the processing chain for controlling the movement of the mobile device based on the first scene data.

[0090] In this embodiment, when the generative model executes a generation task, in addition to conditional information, a task carrier can also be provided to the generative model. The task carrier carries initial data for the generative model to execute the generation task, and this initial data can be, for example, randomly generated. Therefore, step S330, which involves executing multiple set generation tasks guided by the first conditional information through a generative model, can include: for each of the multiple generation tasks, the generation task is executed on the corresponding task carrier through the generative model guided by the first conditional information, resulting in a first generation object for that generation task.

[0091] In some examples, in a single job based on a single frame of data, the multiple generation tasks performed by the generative model can correspond one-to-one with multiple task carriers. Different task carriers can carry different initial data on carriers of the same specifications, allowing the generative model to complete different generation tasks. Using different task carriers to complete multiple generation tasks helps the generative model distinguish tasks and learn the relationships between different generation tasks, thereby improving generation efficiency and accuracy. Using different task carriers of the same specifications facilitates information alignment between multiple tasks. Taking an image-based carrier as an example, the same specifications can include the same size, the same number of channels, etc. Those skilled in the art will understand that in different jobs targeting different frames of data, the initial data carried by the task carrier corresponding to the same generation task can be different; that is, in different jobs, the same generation task can correspond to different task carriers.

[0092] In another example, in a single job based on one frame of data, the multiple generation tasks performed by the generative model can also correspond to the same task carrier. That is, the generative model can execute multiple generation tasks on the same task carrier and generate multiple first generated objects on that task carrier, without any limitation. Those skilled in the art will understand that in each job targeting each frame of data, the initial data carried by the task carrier used by the generative model to execute the generation task each time can be different.

[0093] In some examples, the processing chain for controlling the movement of a mobile device based on first scene information includes the perception task, trajectory prediction task, and trajectory planning task mentioned in related technologies. Accordingly, multiple generation tasks include a first generation task that generates scene perception results, a second generation task that generates the movement trajectories of dynamic scene elements, and a third generation task that generates the movement trajectory of the mobile device.

[0094] In this example, the object generated by the first generation task is the scene perception result. The scene perception result can include the category and location of each scene element obtained from object detection. The scene perception result can identify the detected scene elements using bounding boxes. The object generated by the first generation task is the output of the perception task in this processing chain and also the input of the trajectory prediction task. The object generated by the second generation task is the movement trajectory of dynamic scene elements. The object generated by the second generation task is the output of the trajectory prediction task in this processing chain and also the input of the trajectory planning task. The object generated by the first generation task is used to obtain the object generated by the second generation task in this processing chain. The object generated by the third generation task is the movement trajectory of a mobile device. The object generated by the third generation task is the output of the trajectory planning task in this processing chain. The object generated by the second generation task is used to obtain the object generated by the third generation task in this processing chain.

[0095] As can be seen, in this example, the association of multiple generated objects on this processing chain is at least manifested in the following way: on this processing chain, the generated object of the first generation task is used to obtain the generated object of the second generation task, and the generated object of the second generation task can be used to obtain the generated object of the third generation task. In other words, the multiple generated objects output by the generative model are associated on the processing chain that includes the perception task, trajectory prediction task, and trajectory planning task. From another perspective, the multiple generated objects form two object groups. The two generated objects in one object group are the generated objects of the first generation task and the generated objects of the second generation task, and the two generated objects in this object group are the input and output of the trajectory prediction task on the processing chain, respectively. The two generated objects in the other object group are the generated objects of the second generation task and the generated objects of the third generation task, and the two generated objects in this object group are the input and output of the trajectory planning task on the processing chain, respectively.

[0096] In this example, the time taken for the second generation task to predict the trajectory of dynamic scene elements and the time taken for the third generation task to plan the trajectory of the mobile device can be equal. That is, the second generation task generates the movement trajectory of the dynamic scene elements within a set time window, while the third generation task generates the movement trajectory of the mobile device within the same set time window. The start time of the set time window is the scene data acquisition time corresponding to the condition information guiding the generation model to execute the generation task, and the window length is a set length. This set length can be set according to system response speed and control accuracy requirements, for example, within the range of 5s-10s. If the hardware processing speed allows, a shorter window length can be set to improve control accuracy; this is not limited here. In the operation where the generation model is guided by the first condition information, the set time window is the first time window, and the start time of the first time window is the first time.

[0097] In this example, the first generated object of the first generation task is the scene perception result that meets the first condition information; the first generated object of the second generation task is the first movement trajectory that meets the first condition information, which is the movement trajectory of a dynamic scene element within a first time window; the first generated object of the third generation task is the second movement trajectory that meets the first condition information, which is the movement trajectory of a mobile device within a first time window.

[0098] In another example, multiple generation tasks may include a first generation task and a second generation task, and use other models to plan the movement trajectory of the mobile device based on the generated objects of the generation model. In yet another example, multiple generation tasks may include a first generation task and a third generation task, while omitting the second generation task that predicts the movement trajectory of dynamic scene elements. Accordingly, the corresponding processing chain determines the movement trajectory of the mobile device based on the scene perception results.

[0099] In another example, the overall task of controlling the movement of mobile devices based on scene data can be broken down into more granular or other forms to form corresponding generation tasks, which is not limited here.

[0100] Based on the processing chain of multiple generation tasks, and the correlation between multiple generated objects in the processing chain, these generation tasks should ideally have a sequential execution order. However, in this embodiment, the generation model can learn the correspondence between these generation tasks during training and execute them synchronously during inference to obtain multiple generated objects. Here, the generation model is configured to execute multiple related generation tasks synchronously, rather than just executing a single generation task such as generating the movement trajectory of a mobile device. On the one hand, this allows the generation model to constrain its internal processing logic based on the correspondence or dependency between multiple generation tasks in terms of geometric and semantic consistency, avoiding instability of the automatic control system due to missing effective information and improving generation efficiency and quality. On the other hand, the synchronous execution of these generation tasks by the generation model can avoid information loss and accumulated errors caused by information transfer between different modules, further improving the stability of the system.

[0101] Those skilled in the art should understand that "synchronization" here does not imply strict alignment in generation time. Rather, it should be understood as the generative model starting to execute the generation task corresponding to the subsequent processing node before completing the generation task of the preceding processing node. Continuing with the example of multiple generation tasks corresponding to perception, trajectory prediction, and trajectory planning tasks in the processing chain, the generative model can begin executing the second generation task, or even the third generation task, before completing the first generation task to obtain the corresponding perception task's generation object; or, the generative model can begin executing the third generation task before completing the second generation task to obtain the corresponding trajectory prediction task's generation object. Furthermore, the generative model can also start executing multiple generation tasks simultaneously and output multiple generation objects simultaneously. Those skilled in the art can configure this as needed, and no limitations are imposed here.

[0102] The generative model in this embodiment can be a diffusion model, a model based on generative adversarial networks (GANs), a model based on variational autoencoders (VAEs), etc. It can learn the data distribution and perform generative tasks on the task carrier to obtain any type of generative model that meets the condition information.

[0103] In some examples, the generative model is a diffusion model, where the generative model executes multiple generative tasks simultaneously. This can be achieved by the generative model executing multiple tasks based on a set number of time steps. Each generative task is completed when the total number of time steps T is reached, resulting in the generated object for that task. In other words, the generative model executes each of the multiple generative tasks at each time step. For each generative task, the task carrier containing the initial data serves as the input for the first time step, the output of the current time step becomes the input for the next time step, and the output of the last time step, or the T-th time step, is the generated object for that task.

[0104] In this example, when the generative model performs a generation task at time step j, in addition to using the output of the generative model for the generation task at time step (j-1) as the input of time step j, it can also refer to the output of the generative model for other generation tasks at time step (j-1) to perform the generation task. j is an integer greater than or equal to 2 and less than or equal to the total number of steps T, which is not limited here.

[0105] In some examples, the task carrier can be an image-based carrier. The generative model represents the generated object for each generation task using an image to improve the understandability of the generated object. Here, the task carrier serves as the initial image used to generate the target image representing the generated object. For example, the initial data carried by the task carrier can be pure noise data that conforms to a predetermined distribution law such as Gaussian distribution. In other words, the initial form of the task carrier can be a pure noise image that conforms to a predetermined distribution law.

[0106] In this example, the generation model, guided by the first condition information, executes the generation task on the task carrier corresponding to the generation task to obtain the first generated object of the generation task. This can be further included: using the generation model, guided by the first condition information, to restore or recover the first generated object of the generation task from the task carrier corresponding to the generation task, and obtain the first target image representing the first generated object.

[0107] In an example with multiple generation tasks, including a first generation task, a second generation task, and a third generation task, each generation task can correspond to different pure noise images in the task carrier set. That is, the task carrier set can include three pure noise images, which can be randomly generated based on a set distribution rule, so that the three pure noise images carry different initial data. Accordingly, the generation model will output three first target images that correspond one-to-one with the three generation tasks, and each first target image is used to represent the first generation object of the corresponding generation task.

[0108] In this embodiment, the first condition information is used as part of the input to the generative model to guide the generative model to perform a generation task and obtain a generated object that conforms to the first condition information. In order to improve the temporal consistency of the generated objects, in some examples, in addition to the first scene data, the first condition information can also be obtained based on multiple second generated objects and a first movement state. The multiple second generated objects are generated by the generative model on the task carrier of the corresponding generation task with the guidance of the above-mentioned second condition information; the first movement state is the movement state of the mobile device at the first time, such as forward, left turn, right turn or braking, which is matched with the control parameters at the corresponding time.

[0109] In application, the first scene features, second generated objects, and first movement states obtained based on the first scene data can be input into the generation model as components of the first condition information. That is, in step S320, the first condition information can be obtained based on the first scene data, multiple second generated objects, and the first movement state. The components of the first condition information can be directly input into the generation model, or they can be converted into feature vectors and input into the generation model; this is not limited here.

[0110] In the example where the generative model is a diffusion model, the first conditional information may also include the set total number of time steps t. emb To guide the diffusion model according to the set total number of time steps t emb Execute the generation task. Correspondingly, the first condition information (cond) for the first scene data at the first time t. t It can be represented as:

[0111]

[0112] In formula (1), This represents the first scene feature of the first scene data; This indicates the first movement state of the mobile device at the first time t; This refers to multiple second generated objects generated based on the scene data collected at the second time (t-1).

[0113] In the example of introducing second-generation objects to generate first-condition information, to prevent the generative model from over-relying on historical scene features and ignoring current scene features, multiple second-generation objects from the previous time step can be used. Random dropout is performed to ensure that the generative model generates the first generated object with temporal consistency, based primarily on the current first scene features. The random dropout ratio can be set as needed, such as 0.5 or 0.3, and is not limited here.

[0114] Step S340: Control the movement of the mobile device based on at least a portion of the generated objects among a plurality of first generated objects.

[0115] In step S340, control parameters for the execution components of the mobile device can be obtained based on at least a portion of the first generated object output by the generative model, and then the movement of the mobile device can be controlled based on the control parameters.

[0116] When the multiple first generated objects of the generative model include the movement trajectory of the mobile device, the control parameters of the execution component of the mobile device can be determined based on the first generated object corresponding to the movement trajectory of the mobile device, or the control parameters can be determined based on multiple first generated objects, without limitation here.

[0117] In cases where multiple first generated objects do not include the movement trajectory of the mobile device, the movement trajectory of the mobile device can be obtained first based on at least some of the first generated objects, and then the control parameters of the execution components for the mobile device can be determined based on the movement trajectory of the mobile device.

[0118] In addition, if the movement trajectory of the mobile device is not included in the multiple first generated objects, a pre-trained model can be used to learn the mapping relationship between at least some of the first generated objects and the control parameters, so that at least some of the first generated objects can be input into the model to obtain the control parameters for the actuator, without limitation.

[0119] The generated objects output by the generative model are usually latent space data. In order to facilitate the conversion of the output of the generative model into control parameters, in some examples, the step S340 of controlling the movement of the mobile device based on at least a portion of the generated objects among a plurality of first generated objects may further include: determining the vectorized trajectory of the mobile device in a first time window based on at least a portion of the first generated objects, and determining control parameters for the mobile device based on the vectorized trajectory, so as to control the movement of the mobile device based on the control parameters.

[0120] In this example, the start time of the first time window is the first time when the first scene data is collected, and the window length of the first time window is a set length.

[0121] In this example, if multiple first generated objects include the movement trajectory of the mobile device, the vectorized trajectory of the mobile device in the first time window can be extracted from the corresponding first generated objects. This vectorized trajectory includes the position coordinates of the mobile device at each time point within the first time window. Based on this vectorized trajectory, information such as the velocity, acceleration, and heading angle of the mobile device at each time point can be obtained to determine the corresponding control parameters.

[0122] In this example, if multiple first generated objects do not include the movement trajectory of the mobile device, the latent space data of the movement trajectory can be obtained first based on at least some of the first generated objects, and then the vectorized trajectory can be extracted from the latent space data of the movement trajectory; alternatively, the vectorized trajectory can be extracted directly based on at least some of the first generated objects, etc., without limitation here.

[0123] In some examples, at least a portion of the first generated object can be input into a pre-trained trajectory extraction model to obtain the vectorized trajectory of the mobile device in a first time window. In this example, during training, the model parameters of the trajectory extraction model and the generation model can be determined synchronously based on the same training samples. This synchronous training is based on a loss function that includes a first loss generated by the generation model and a second loss generated by the trajectory extraction model, which can be expressed as:

[0124] L=L1+L2 (2)

[0125] In formula (2), L represents the total loss, L1 represents the first loss generated by the generative model, and L2 represents the second loss generated by the trajectory extraction model.

[0126] The trajectory extraction model can employ a query vector-based neural network, which includes, for example, an embedding layer, a transformer layer, and a multi-layer perceptron (MLP) layer. Trajectory information is extracted through the interaction between the query vector and the input data. For instance, the generative model outputs a first target image representing multiple first generated objects, including the movement trajectory of a mobile device. The first target image output by the generative model is a latent space image from a top-down (BEV) perspective. The first target image is input into the trajectory extraction model, which divides the first target image into a series of processing units (tokens) through the embedding layer. The learnable query vector is cross-correlated with the token sequence through a series of transformer layers, and then the learned query is decoded into a vectorized trajectory of the mobile device through the MLP layer.

[0127] According to steps S310 to S340 above, the method of this embodiment executes multiple generation tasks synchronously through a generative model, resulting in multiple generated objects that were originally generated separately by multiple serially connected modules, thereby realizing the motion control of mobile devices. In this embodiment, on the one hand, because the processing link is compressed, the transmission of information between different modules can be effectively reduced or even avoided, which can effectively reduce error accumulation and information loss; on the other hand, because the generative model can learn the correspondence between multiple generation tasks or multiple generated objects during training, and use this correspondence as a constraint to mine effective information when executing multiple generation tasks, it can avoid the loss of effective information, thereby improving the performance of the automatic control system in at least one aspect such as stability, real-time performance, and security.

[0128] <Second Embodiment>

[0129] This embodiment, in addition to controlling the movement of the mobile device, can also visualize the scene where the mobile device is located based on the generated objects output by the generative model. Therefore, based on the first embodiment, the control method of this embodiment can further include: decoding multiple first generated objects into pixel space to obtain a pixel space image, and displaying the scene where the mobile device is located based on the obtained pixel space image.

[0130] Since the first generated object generated by the generative model is usually a latent space data representation with poor visibility, in order to enable users to intuitively see the scene understood by the automatic control system based on scene data, in this embodiment, multiple first generated objects can be decoded into pixel space by a model with decoding capabilities such as a Variational Autoencoder Encoder (VAE). For example, multiple first target images representing multiple first generated objects output by the generative model can be input into the VAE to obtain the corresponding pixel space images.

[0131] After obtaining the pixel space image, it can be displayed on the terminal of the mobile device. The scene display of the scene where the mobile device is located can be realized by updating the pixel space image.

[0132] In some examples, the generative model can render the generated objects with colors set during training while performing the generation task, thereby increasing the information dimension of the pixel-space image for scene representation.

[0133] The method in this embodiment can not only control the movement of mobile devices based on scene images, but also intuitively display the scene where the mobile device is located based on multiple generated objects of the generative model, such as scene perception results, trajectory prediction results, and trajectory planning results, making it convenient for users to view.

[0134] <Third Embodiment>

[0135] Figure 4 and Figure 5 A flowchart of a method according to a third embodiment is shown, corresponding to the automatic control system implementing the control method. Figure 5 The reasoning process in the model. In this embodiment, the generative model adopts a diffusion model. The multiple generative tasks performed by the diffusion model include the first generative task, the second generative task, and the third generative task mentioned above. Based on the correspondence of the task execution results, the first generative task, the second generative task, and the third generative task respectively correspond to the perception task, trajectory prediction task, and trajectory planning task on the processing link. Figure 4 and Figure 5 As shown, the control method of this embodiment includes the following steps S410 to S450:

[0136] Step S410: Obtain the first scene data collected by the mobile device in the first moment.

[0137] In this embodiment, the first scene data includes multiple scene images captured by multiple cameras in a single moment. These multiple scene images are also known as... Figure 5 The panoramic image in the image.

[0138] Step S420: Extract scene features from the first scene data using a feature extraction model to obtain the first scene features from an overhead view, and then use these first scene features as the basis for the extraction. First movement state A second target image set representing multiple second generated objects and time step t emb The first condition information, cond, is obtained. t .

[0139] Step S430: The pure Gaussian noise image set X, which serves as the task carrier... t and the first condition information cond t The input is fed into the generative model, which, guided by the first conditional information, reconstructs the first generated object of each generation task from the pure Gaussian noise image corresponding to each generation task, thus obtaining a first target image set comprising multiple first generated objects.

[0140] Step S440, set the first target image set The data is input into the trajectory extraction model, which then extracts images from the first target image set. The vectorized trajectory of the mobile device is extracted, and the movement of the mobile device is controlled based on the vectorized trajectory.

[0141] Step S450, the first target image set Decode to pixel space to obtain pixel space image, and display the scene where the mobile device is located based on the obtained pixel space image.

[0142] In this embodiment, the model parameters of the feature extraction model and the model parameters of the generation model can be determined through joint training based on the same training samples, and the loss generated by the feature extraction model during joint training is included in the first loss generated by the generation model.

[0143] In this embodiment, the model parameters of the trajectory extraction model and the model parameters of the generation model can be determined by synchronous training based on the same training samples. The loss function on which the synchronous training is based includes the first loss generated by the generation model and the second loss generated by the trajectory extraction model.

[0144] The above examples of control methods based on automatic control systems are given in a non-limiting manner. The following describes various examples of training methods for such automatic control systems.

[0145] <Fourth Embodiment>

[0146] This embodiment relates to a training method for an automatic control system, which includes a generative model for performing multiple generative tasks associated with controlling the movement of a mobile device based on scene data. The training method can be... Figure 1 The medium-sized mobile device 101 can also be implemented by Figure 1 The training method can be implemented using server 102, or it can be implemented jointly by mobile device 101 and server 102; furthermore, this training method can be implemented offline or online, without limitation. Figure 6 As shown, the training method of this embodiment may include the following steps S610 to S640.

[0147] Step S610: Obtain the training sample set.

[0148] In this embodiment, the training samples in the training sample set include scene data and truth sets of generated objects for multiple generation tasks obtained based on the scene data.

[0149] The scene data is, for example, a panoramic image.

[0150] The truth set of the generated object can be obtained through manual annotation or through automatic annotation using a truth annotation tool; no limitation is made here.

[0151] The ground truth values ​​of generated objects for different generation tasks can come from different ground truth images. For example, VAEs can be used to reduce the dimension of the task-annotated ground truth image to the latent space, thereby obtaining the corresponding ground truth values ​​of generated objects.

[0152] For example, multiple generation tasks include a first generation task, a second generation task, and a third generation task. A ground truth image set is constructed for each of these three tasks. During construction, scene elements from the stitched panoramic image at time T (i.e., a map fragment) and the bounding boxes of targets for scene perception can be rasterized onto the first canvas from a top-down perspective, resulting in the first ground truth image at time T used for the first generation task. The movement trajectories of the dynamic scene elements are then plotted on a second canvas to obtain the second ground truth image at time T for the second generation task. Furthermore, the movement trajectory of the mobile device is plotted on a third canvas to obtain a third ground truth image at time T for the third generation task. The three ground truth images in the ground truth image set at time T have the same specifications and are aligned with information. Time T is any time when the training sample data is collected.

[0153] First truth image In this model, different scene elements can be rendered using different colors. By rendering different scene elements with different colors, the generative model can learn how to represent generated objects for the first generation task during training. This enables the generative model to generate generated objects that distinguish different scene elements by color when performing the first generation task during inference, thus improving its ability to express generated objects.

[0154] In the third ground truth image In this model, the color of the movement trajectory can gradually change over time, representing the temporal relationship between different points on the trajectory. Furthermore, RGB values ​​of overlapping trajectories can be interpolated to preserve all information. By rendering the ground truth of the movement trajectory with a temporal color gradient, the generative model learns how to represent generated objects for the third generation task during training. This allows the model to generate objects representing the movement trajectory of the mobile device using gradient colors during inference, improving its expressive ability for generated objects.

[0155] Second ground truth image Similarly, the movement trajectory of dynamic elements can be marked with time-series gradient colors, which will not be elaborated here.

[0156] Step S620: Add noise to the true value set of generated objects of the training samples to obtain the dataset with added noise.

[0157] Step S630: Obtain conditional information based on the scene data of the training samples.

[0158] In some examples, scene features can be extracted from the scene data of the training samples using a feature extraction model to obtain conditional information based on the scene features.

[0159] Step S640: Input the conditional information of the training samples and the dataset with added noise into the generative model, so that the generative model predicts the noise added to the dataset based on the conditional information and obtains the noise prediction value.

[0160] Step S650: Construct a loss function using the first loss of the noise prediction value relative to the true value of the added noise, and update the model parameters of the generated model.

[0161] In this embodiment, the model parameters of the generated model are updated based on the loss function until the training stops under certain conditions, such as the loss function converging or reaching a set number of training steps, which are not limited here.

[0162] In the example of extracting scene features from scene data through a feature extraction model, the feature extraction model and the generation model can be jointly trained. In this way, the step S650 of constructing a loss function with the first loss of the noise prediction value relative to the added noise ground truth value and updating the model parameters of the generation model can include: constructing a loss function with the first loss of the noise prediction value relative to the noise ground truth value and updating the model parameters of the generation model and the model parameters of the feature extraction model.

[0163] In this example, the loss generated by the feature extraction model can be reflected in the first loss. Therefore, the generative model and the feature extraction model can be jointly trained based on the first loss generated by the generative model.

[0164] In some examples, in addition to the generative model, the automatic control system may also include a trajectory extraction model, which is used to output a vectorized trajectory of the mobile device based on at least a portion of the generated object output by the generative model. In this example, the training method may further include: inputting the ground truth set of generated objects from the training samples into the trajectory extraction model to obtain extracted values ​​of the vectorized trajectory of the mobile device. In this example, the step S660 of constructing a loss function with a first loss of the noise prediction value relative to the added noise ground truth value and updating the model parameters of the generative model may include: constructing a loss function with the first loss of the noise prediction value relative to the noise ground truth value and a second loss of the extracted value of the vectorized trajectory relative to the ground truth value of the vectorized trajectory, and synchronously updating the model parameters of the generative model and the model parameters of the feature extraction model.

[0165] When multiple generation tasks include a third generation task that generates the movement trajectory of a mobile device within a set time window, the ground truth of the vectorized trajectory can be determined based on the movement trajectory represented by the third ground truth image.

[0166] The generative model in this embodiment is used to perform multiple generation tasks. Therefore, during training, the correspondence between multiple generation tasks can be learned by jointly training multiple generation tasks. In the inference process, multiple generation tasks can be executed simultaneously to generate multiple generated objects that meet the requirements, thereby improving the performance of the autonomous driving system.

[0167] In this embodiment, since generative models such as diffusion models, GANs, and VAEs can explicitly or implicitly learn data distributions to perform set generative tasks, this can effectively improve the performance of automatic control systems in scene perception, trajectory prediction, and trajectory planning compared to knowledge distillation models, thereby improving the performance of autonomous driving systems.

[0168] <Fifth Embodiment>

[0169] Figure 7 and Figure 5 A flowchart of the training method according to the fifth embodiment is shown. In this embodiment, the generative model adopts a diffusion model. The multiple generative tasks performed by the diffusion model include the first generative task, the second generative task, and the third generative task mentioned above. Based on the correspondence of the task execution results, the first generative task, the second generative task, and the third generative task respectively correspond to the perception task, trajectory prediction task, and trajectory planning task on the link. Figure 7 and Figure 5 As shown, the control method of this embodiment includes the following steps S710 to S750:

[0170] Step S710: Obtain the training sample set.

[0171] The training samples in the training sample set include the around-view image at time T as the scene image, and the ground truth sets of generated objects for multiple generation tasks obtained based on the around-view image. In this embodiment, the truth set of the generated object at time T Including three latent space BEV images, generate object truth sets. For example, the above-mentioned ground truth image set can be obtained by dimensionality reduction and compression using VAE.

[0172] Step S720, based on the set total number of steps t emb Gradually generate the true value set of the training samples. Adding noise yields a dataset with added noise.

[0173] Step S730: Generate conditional information based on the scene data of the training samples.

[0174] In this embodiment, scene features from a top-down view are extracted using a feature extraction model to generate the guiding condition (cond).T Among them, the feature extraction model can integrate historical scene features to extract features from scene data.

[0175] In this embodiment, the conditional information cond corresponding to the training samples at time T... T It can be represented as follows:

[0176]

[0177] In formula (3), This represents the scene features obtained from the panoramic image at time T; This indicates the movement state at time T; This represents the truth set of the generated objects at time (T-1).

[0178] Step S741, convert the conditional information cond corresponding to the training samples. T and datasets with added noise The input is given to the generative model, enabling the generative model to be based on the conditional information cond. T Prediction dataset Added noise ∈ T Obtain the noise prediction value

[0179] Step S742, generate the truth set of the object. Input the data into the trajectory extraction model to extract the vectorized trajectory of the mobile device, and obtain the extracted value of the vectorized trajectory of the mobile device.

[0180] Step S750, using noise prediction values The true value of noise relative to the added noise ∈ T The loss function L is constructed by first loss L1 and second loss L2, which is the extracted value of the vectorized trajectory relative to the true value of the vectorized trajectory. The model parameters of the generative model, trajectory extraction model and feature extraction model are updated.

[0181] <Sixth Embodiment>

[0182] This embodiment provides a chip capable of implementing the methods of the embodiments of this disclosure, such as... Figure 8 As shown, the chip 800 includes a storage unit 820 and a processing unit 810. The storage unit 820 is used to store a computer program. The processing unit 810 is configured to implement a control method and / or a training method according to any embodiment of this disclosure when executing the computer program stored in the storage unit.

[0183] Chip 800 can be a processor chip with data processing capabilities.

[0184] Chip 800 can be a processor chip used in Automated Driving Control Units (ADCUs).

[0185] Chip 800 can be a system-on-a-chip (SoC). Chip 900 integrates multiple processors, multiple memories, I / O interfaces, etc., to achieve miniaturized controller design.

[0186] <Seventh Embodiment>

[0187] This embodiment provides an electronic device, such as... Figure 9 As shown, the electronic device 900 includes a memory 902 and a processor 901. The memory 902 is used to store a computer program executed by the processor 901. The processor 901 is configured to implement the method according to any embodiment of the present disclosure when executing the computer program stored in the memory 902.

[0188] The electronic device 900 may be a server or controller for implementing the methods of the embodiments of this disclosure. The controller may be, for example, an intelligent driving domain controller or a vehicle's central controller.

[0189] <Eighth Embodiment>

[0190] This embodiment provides a vehicle that may include a chip 1000 according to a sixth embodiment or a controller according to a seventh embodiment.

[0191] In some embodiments, such as Figure 10 As shown, the vehicle 1000 may also include a memory 1002 and a processor 1001. The memory 1002 is used to store a computer program executed by the processor 1001. The processor 1001 is configured to implement a control method and / or training method according to any embodiment of the present disclosure when executing the computer program stored in the memory 1002.

[0192] In addition, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that is loaded and executed by a processor to implement the method according to any embodiment of this disclosure.

[0193] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0194] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0195] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0196] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0197] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0198] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0199] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0200] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0201] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A control method of a mobile device, characterized by, The method comprises: acquiring first scene data collected by a movable device at a first time; obtaining first condition information based on the first scene data, the first condition information being used to provide scene information to a generative model; synchronously executing each of a plurality of preset generation tasks by the generative model guided by the first condition information to obtain a plurality of first generation objects corresponding to the plurality of generation tasks; wherein the plurality of first generation objects can be concatenated to form at least part of a processing link for controlling the movable device to move based on the first scene data; controlling the movable device to move based on at least part of the plurality of first generation objects.

2. The method of claim 1, wherein, The plurality of generation tasks comprise a first generation task of generating a scene perception result, a second generation task of generating a moving track of a dynamic scene element within a preset time window, and a third generation task of generating a moving track of the movable device within the preset time window; wherein a start time of the preset time window is a scene data collection time corresponding to condition information for guiding the generative model to execute the generation tasks, and a window length of the preset time window is a preset length.

3. The method of claim 1, wherein, The obtaining first condition information based on the first scene data comprises: obtaining first condition information based on the first scene data, a first moving state, and a plurality of second generation objects of the generative model; wherein the second generation objects are generated by the generative model on a task carrier of a corresponding generation task guided by second condition information, the second condition information is condition information corresponding to second scene data, the second scene data is collected by the movable device at a second time, the second time is before the first time, and the first moving state is a moving state of the movable device at the first time.

4. The method of claim 1, wherein, After the obtaining a plurality of first generation objects corresponding to the plurality of generation tasks, the method further comprises: decoding the plurality of first generation objects to a pixel space to obtain a pixel space image; performing scene display on a scene where the movable device is located based on the pixel space image.

5. The method of claim 1, wherein, The executing each of a plurality of preset generation tasks by the generative model guided by the first condition information to obtain a plurality of first generation objects corresponding to the plurality of generation tasks comprises: for each of the plurality of generation tasks, executing the generation task on a task carrier corresponding to the generation task by the generative model guided by the first condition information to obtain a first generation object of the generation task; wherein the plurality of generation tasks correspond to a plurality of task carriers one by one, and different task carriers carry different initial data.

6. The method of claim 5, wherein, The task carrier is an image form carrier, and the initial data of the task carrier is pure noise data; the executing the generation task on the task carrier corresponding to the generation task by the generative model guided by the first condition information to obtain a first generation object of the generation task comprises: restore, from the task carrier corresponding to the generation task, the first generation object of the generation task based on the first conditional information, to obtain a first target image representing the first generation object.

7. The method of claim 1, wherein, The method further includes: determining a vectorized trajectory of the movable device in a first time window based on the at least part of the first generation objects, wherein a start time of the first time window is the first time, and a window length of the first time window is a set length; determining a control parameter for the movable device based on the vectorized trajectory; controlling the movable device to move based on the control parameter.

8. The method of claim 7, wherein, The method further includes: inputting the at least part of the first generation objects into a trajectory extraction model to obtain the vectorized trajectory of the movable device in the first time window; wherein model parameters of the trajectory extraction model and model parameters of the generation model are determined based on a same training sample through synchronous training, and a loss function based on which the synchronous training is performed includes a first loss generated by the generation model and a second loss generated by the trajectory extraction model.

9. The method according to any one of claims 1 to 8, characterized in that, The first conditional information includes a first scene feature extracted from the first scene data, and the first conditional information is obtained based on the first scene data, including: extracting the first scene feature from the first scene data through a feature extraction model; wherein model parameters of the feature extraction model and model parameters of the generation model are determined based on a same training sample through joint training, and a loss generated by the feature extraction model in the joint training is included in the first loss generated by the generation model.

10. The method of claim 9, wherein, The feature extraction model is configured to obtain the first scene feature based on the first scene data and historical scene features, wherein the historical scene features are scene features obtained by the feature extraction model based on historical scene data, and a collection time of the historical scene data is before the first time.

11. A training method of an automatic control system, characterized by, The automatic control system includes generation models for synchronously performing a plurality of generation tasks, the plurality of generation tasks can be concatenated to form at least part of a processing link for controlling a movable device to move based on scene data, and the training method includes: obtaining a training sample set, wherein a training sample in the training sample set includes scene data and a ground truth set of generation objects of the plurality of generation tasks based on the scene data; adding noise to the ground truth set of generation objects of the training sample to obtain a noise-added data set; generating conditional information based on the scene data of the training sample, the conditional information being used to provide scene information to the generation model; inputting the conditional information corresponding to the training sample and the noise-added data set into the generation model, so that the generation model predicts the noise added in the data set based on the conditional information to obtain a noise prediction value; and determining a vectorized trajectory of the movable device in a first time window based on the at least part of the first generation objects, wherein a start time of the first time window is the first time, and a window length of the first time window is a set length; determining a control parameter for the movable device based on the vectorized trajectory; controlling the movable device to move based on the control parameter. construct a loss function with a first loss of the noise prediction value relative to a noise true value, and update model parameters of the generative model.

12. The method of claim 11, wherein, The automatic control system further comprises a trajectory extraction model, and the method further comprises: inputting a true value data set of the training sample into the trajectory extraction model to obtain an extracted value of a vectorized trajectory of the movable device; constructing a loss function with a first loss of the noise prediction value relative to a noise true value, and updating model parameters of the generative model, comprises: constructing a loss function with a first loss of the noise prediction value relative to the noise true value, and a second loss of the extracted value of the vectorized trajectory relative to a true value of the vectorized trajectory, and synchronously updating model parameters of the generative model and model parameters of the trajectory extraction model.

13. The method according to claim 11 or 12, characterized in that, The conditional information comprises scene features extracted from scene images, and the conditional information is generated based on the scene images of the training sample, comprising: extracting scene features from the scene images of the training sample by a feature extraction model; constructing a loss function with a first loss of the noise prediction value relative to a noise true value, and updating model parameters of the generative model, comprises: constructing a loss function with a first loss of the noise prediction value relative to the noise true value, and updating model parameters of the generative model and model parameters of the feature extraction model.

14. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the method of any one of claims 1 to 13 when executing the instructions stored in the memory.

15. A chip, characterized by comprise: a storage unit for storing a computer program; and a processing unit configured to implement the method of any one of claims 1 to 13 when executing the computer program stored in the storage unit. comprise the electronic device according to claim 14; or the movable device comprises the chip according to claim 15; or the movable device comprises:

16. A mobile device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the method of any one of claims 1 to 13 when executing the instructions stored in the memory. The computer program instructions, when executed by the processor, implement the method of any one of claims 1 to 13.

17. A non-transitory computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by the processor, implement the method of any one of claims 1 to 13.

Images