Control method, data generation method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a control method, a data generation method, electronic equipment and a storage medium, and relates to the technical field of automatic control. The control method comprises the following steps: acquiring target environment data sensed by a plurality of target sensors of the mobile equipment; and inputting the target environment data into a pre-generated target control model, and performing motion control on the mobile equipment through the target control model. Wherein the data used for training the target control model can comprise virtual perception data generated by a data generation model, and the data generation model can be used for generating the virtual perception data at least partially according to a target environment image corresponding to a target sensor, the target environment image can be an image generated according to sensor parameters of the target sensor and object description information for describing a target environment object perceivable by the target sensor.

Description

Technical Field

[0001] This disclosure relates to the field of automatic control technology, and more specifically, to a control method, a data generation method, an electronic device, and a storage medium. Background Technology

[0002] With the development of artificial intelligence (AI) technology, AI-based control models can be deployed in the control systems of vehicles and other mobile devices to achieve motion control of the devices. However, to improve the reliability of these control models, a large amount of high-quality training data is required. Therefore, the quantity and quality of training data become key factors limiting the reliability of control models.

[0003] In related technologies, the efficiency of producing training data by manually-led data collection and labeling is relatively low. How to efficiently produce high-quality training data has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the present disclosure proposes a new technical solution regarding a control method.

[0005] According to a first aspect of the present disclosure, a control method is provided, the method comprising: Acquire target environment data perceived by multiple target sensors of a mobile device; The target environment data is input into a pre-generated target control model, and the motion control of the mobile device is performed through the target control model. The data used to train the target control model includes virtual perception data generated by a data generation model. The data generation model generates the virtual perception data at least in part based on the target environment image corresponding to the target sensor. The target environment image is an image generated based on the sensor parameters and object description information of the target sensor. The object description information is used to describe the target environment objects that the target sensor can perceive.

[0006] Optionally, the virtual sensing data is generated in the following manner: Obtain data description information used to generate the virtual perception data; wherein, the data description information includes the object description information; For each target sensor of the mobile device, a target environment image corresponding to the target sensor is generated based on the sensor parameters of the target sensor and the object description information; The target environment image is input into the data generation model to obtain virtual perception data output by the data generation model.

[0007] Optionally, the data generation model includes a first encoding module, a data generation module, and a data adjustment module; the step of inputting the target environment image into the data generation model to obtain the virtual perception data output by the data generation model includes: The first encoding feature is obtained by encoding the target environment images corresponding to the multiple target sensors based on the first encoding module. The data generation module generates first perceptual data based at least in part on the first encoded feature; The data adjustment module adjusts the spatial resolution of the first sensing data at least in part according to the first encoding feature to obtain the second sensing data. The virtual sensing data is generated based on the first sensing data and / or the second sensing data.

[0008] Optionally, the data generation module includes a first submodule and a second submodule, wherein the first submodule is used to determine the temporal information of a first data frame at multiple times in the first sensed data, and the second submodule is used to generate spatial information of each different target environment object in the first data frame.

[0009] Optionally, the data adjustment module includes a third sub-module, which is used to adjust the spatial resolution of each different target environment object in the first data frame to obtain a second data frame at multiple times in the second perception data; the third sub-module and the second sub-module are neural network models with the same structure.

[0010] Optionally, the data description information further includes device description information for describing the device information of the mobile device and / or environmental description information for describing the environment in which the mobile device is located; the data generation model further includes a second encoding module, which is used to encode the device description information and / or the environmental description information to obtain a second encoding feature; The generation of first perceptual data based on the data generation module, at least in part, the first encoded feature, includes: The first encoded feature and the second encoded feature are input into the data generation module to obtain the first perceived data output by the data generation module.

[0011] Optionally, the data generation module and the data adjustment module are modules that have been trained independently.

[0012] Optionally, the target environment object includes a static environment object and / or a dynamic environment object.

[0013] According to a second aspect of the present disclosure, a data generation method is provided, the method comprising: Acquire data description information for generating virtual sensing data; wherein, the virtual sensing data is used to train a target control model, the target control model is used to perform motion control on a mobile device, the mobile device is configured with multiple target sensors for sensing the environment, and the data description information includes object description information describing the target environmental objects that the target sensors can sense; For each target sensor of the mobile device, a target environment image corresponding to the target sensor is generated based on the sensor parameters of the target sensor and the object description information; The target environment image is input into the data generation model to obtain virtual perception data output by the data generation model.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, including a memory and a processor, the memory being configured to store computer instructions, and the processor being configured to invoke the computer instructions from the memory to perform the method as described in the first or second aspect.

[0015] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first or second aspect.

[0016] Based on the control method provided in this disclosure, a data generation model can automatically generate virtual sensing data for training a target control model. By using the target environment images corresponding to multiple target sensors of the mobile device, the target environment objects that each target sensor can perceive can be described more clearly and accurately. This enables fine-grained control of the data generation model, making the virtual sensing data generated by the data generation model clearer and more accurate, and closer to the real sensing data of each target sensor. As a result, high-quality virtual sensing data can be automatically and efficiently generated, further improving the reliability of the target control model trained by the virtual sensing data, and also improving the reliability of motion control of the mobile device based on the target control model.

[0017] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the present disclosure.

[0019] Figure 1 This is a schematic diagram of an intelligent connected system to which the methods provided in the embodiments of this disclosure can be applied.

[0020] Figure 2 It is based on Figure 1 The illustrated embodiment provides a schematic diagram of a mobile device.

[0021] Figure 3 This is a flowchart illustrating a control method provided in an embodiment of this disclosure.

[0022] Figure 4 This is a schematic flowchart of a data generation method provided in an embodiment of this disclosure.

[0023] Figure 5 This is a schematic diagram of a data generation model provided in an embodiment of this disclosure.

[0024] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0025] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0026] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0027] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0028] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0029] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0030] The elements involved in the embodiments of this disclosure may represent part or all of an element. For example, the elements involved in the embodiments of this disclosure may be at least a part of an element or all of an element.

[0031] The elements involved in the embodiments of this disclosure may be one or more, such as "a", "the", "the above", "the", "the foregoing", etc., which are used to indicate that the corresponding element is mentioned for the first time or is mentioned again, and do not have the meaning of limiting the number.

[0032] It should be noted that all actions involving the collection, storage, use, processing, transmission, provision, disclosure, and deletion of data in this disclosure are carried out in accordance with the relevant data protection laws and regulations of the country or region where the data is located, and with the full authorization of the relevant data owner.

[0033] First, the application scenarios of the embodiments of this disclosure will be described.

[0034] Figure 1 This is a schematic diagram of an intelligent connected system 100 to which the methods provided in the embodiments of this disclosure can be applied. Figure 1 As shown, the intelligent connected system 100 may include: a mobile device 101, a server 102, and a user terminal 103.

[0035] In some examples, the mobile device 101 can be a mobile device such as a vehicle, robot, ship, or aircraft, for example, a vehicle, ship, or aircraft with a driving automation feature, or an autonomously moving robot (such as a cargo robot, a probe robot, or a sweeping robot).

[0036] The driving automation function can include advanced driver assistance functions (ADAS) and automated driving functions (AWD). Automated driving, also known as intelligent driving or driverless driving, refers to vehicles equipped with driving automation functions that can perform some or all of the driving tasks, such as environmental perception, decision-making, planning, and control execution. The levels of driving automation functions can refer to the vehicle intelligence classification standards established by the Society of Automotive Engineers (SAE), for example, divided into six levels from L0 to L5. L0 is emergency assistance, L1 is partial driver assistance, L2 is combined driver assistance, L3 is conditional automated driving, L4 is highly automated driving, and L5 is fully automated driving. The above classification of driving automation function levels is merely an example, and this disclosure does not limit the classification standards and levels of driving automation functions.

[0037] In some examples, server 102 can be a single server or a distributed server cluster consisting of multiple servers, and its deployment method can include local servers or cloud servers. Server 102 can communicate with mobile device 101 and / or user terminal 103 via a communication network, providing various services to mobile device 101 and / or user terminal 103. For example, the server can receive sensing data sent by mobile device 101, provide services such as high-precision maps, data analysis, and decision planning for mobile device 101, or receive query commands or control commands sent by user terminal 102, providing corresponding services to the user.

[0038] In some examples, user terminal 103 can be any form of electronic device providing services to the user, such as a personal computer, laptop, smart tablet, smartphone, smart wearable device, etc. The user can interact with the mobile device or server through the human-computer interaction terminal configured on the mobile device 101, or through user terminal 103. For example, the user can query the status and / or parameters of the mobile device, or control the mobile device to perform set tasks and / or modify configuration parameters, etc. The user terminal runs an application based on the intelligent network system to achieve interaction with the mobile device or server. This application can be a local application, a web application, or a mini-program, etc., and is not limited thereto.

[0039] In some examples, the aforementioned application running on the user's terminal can provide authentication or authorization services to the user. The user who is successfully authenticated and granted the corresponding permissions can query and / or control the mobile device within the scope of the granted permissions.

[0040] The mobile device 101, server 102, and user terminal 103 can communicate via a communication link provided by communication network 104. This communication network 104 can include one or more networks of any type, such as the Internet, Local Area Network (LAN), Wide Area Network (WAN), Virtual Private Network (VPN), Public Switched Telephone Network (PSTN), satellite communication network, Wi-Fi, 2G, 3G, 4G, 5G, 6G, NB-IoT, eMTC, infrared, Bluetooth, NFC, or a combination of these networks. The communication networks between the mobile device 101 and server 102, between the user terminal 103 and server 102, and between the user terminal 103 and mobile device 101 can be the same or different.

[0041] It should be noted that, Figure 1The structure of the intelligent connected system 100 shown is merely illustrative. The intelligent connected system in this embodiment is not limited to the above structure and may include more or fewer devices as needed, and the devices may be combined or split. For example, the intelligent connected system may not include user terminals and / or servers; as another example, user terminals and servers may be deployed together.

[0042] Figure 2 It is based on Figure 1 The illustrated embodiment provides a schematic diagram of a mobile device 101. As shown... Figure 2 As shown, the mobile device 101 may include a sensing component 1011, a computing platform 1012, an execution component 1013, etc. The sensing component 1011, the computing platform 1012, and the execution component 1013 may be connected via a bus or other means.

[0043] In some examples, the sensing component 1011 can be used to collect information about the mobile device itself or externally. The sensing component 1011 may include at least one of a visual sensing unit, radar, positioning and navigation unit, inertial measurement unit (IMU) or other sensing unit. The visual sensor unit may include one or more cameras, the radar may include at least one of lidar, millimeter-wave radar, ultrasonic radar or other radar, and the positioning and navigation unit may include at least one of a GPS system, BeiDou system or other global positioning system.

[0044] In some examples, the computing platform 1012 may include a computing-capable device for processing the sensing information collected by the sensing component 1011 to obtain control information, and sending corresponding control commands to the execution component 1013 to cause the execution component 1013 to perform corresponding actions, thereby realizing the control of the mobile device 101. For example, the computing platform 1012 can perform one or more of the following actions on the mobile device: information collection and processing, positioning, decision-making, planning, and control, thereby realizing the autonomous control of the mobile device. The computing platform 1012 may include at least one processor and at least one memory, wherein each processor can individually or jointly execute instructions stored in the memory to implement the methods provided in the embodiments of this disclosure. The processor in this disclosure embodiment may include at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Neural-network Processing Unit (NPU), Tensor Processing Unit (TPU), Data Processing Unit (DPU), Digital Signal Processor (DSP), Field Programmable Gate Array (FPGA), Programmable Logic Array (PLA), System on Chip (SOC), Application Specific Integrated Circuit (ASIC), Micro Controller Unit (MCU), or other processors. The memory may be implemented using any type of volatile or non-volatile computer-readable storage medium or a combination thereof. In addition to storing instructions, the memory may also store data, such as map data, image data, sound data, text data, configuration parameters of the mobile device, location, orientation, speed, etc. The data stored in the memory can be accessed and used by the processor.

[0045] In some examples, the computing platform of a mobile device can perform computing tasks independently or communicate with a server to complete computing tasks. For example, the computing platform of a mobile device can cooperate with a server to complete corresponding computing tasks. These computing tasks can include any task performed to achieve autonomous control of the mobile device, such as information collection and processing, positioning, decision-making, planning, or control of the mobile device.

[0046] The computing platform 1012 can be located in the mobile device 101. Some or all of the computing platform 1012 can also be located in the server corresponding to the mobile device. For example, some functions of the computing platform 1012 with high real-time requirements can be located in the mobile device, while other functions with low real-time requirements can be located in the server corresponding to the mobile device.

[0047] In some examples, the execution component 1013 is used to perform corresponding actions based on the control of the computing platform 1012, enabling the mobile device 101 to complete the movement task. The execution component 1013 may include, for example, a power component, a braking component, a transmission component, a steering component, etc.

[0048] It should be noted that, Figure 2 The structure of the mobile device 101 shown is merely illustrative. The mobile device in this embodiment is not limited to the above structure and may include more or fewer components as needed. The device may also be combined or disassembled. For example, the mobile device may not include the aforementioned computing platform. Furthermore, the mobile device may also include communication components, interface components, multimedia components, input components, output components, display components, etc.

[0049] In some embodiments of this disclosure, the mobile device may be configured with a control system, which may include some or all of the aforementioned sensing components, computing platform, and execution components. The control system can, alone or in conjunction with the user, perform one or more functions such as environmental perception, decision-making and planning, and control execution of the mobile device. For example, the control system may deploy an artificial intelligence-based control model, which enables motion control of the mobile device. For example, the control model can determine the environmental information of the mobile device based on input environmental data, and make decisions or plans based on the environmental information and the device information of the mobile device to perform motion control. For example, if the mobile device is a vehicle, the control model can control the vehicle's speed and / or direction of travel. To improve the reliability of the control model, a large amount of high-quality training data is needed to train the model. Therefore, the quantity and quality of the training data become key factors restricting the reliability of the control model.

[0050] In related technologies, training data can be generated through manually led data collection and annotation. For example, real-world perception data can be collected by the sensors of a mobile device during its movement. This real-world perception data is then manually annotated to obtain corresponding label information, and training data is obtained based on the annotated real-world perception data and label information. However, this method of data collection and annotation is inefficient, leading to a bottleneck in data supply throughput. This results in the production rate of high-quality training data being far lower than the model iteration cycle, affecting the reliability of the control model.

[0051] To address the problems in related technologies, this disclosure provides embodiments such as Figure 3 The diagram illustrates a control method. This control method can be executed by any electronic device, for example, it can be implemented by... Figure 1 The server shown is executing. (As indicated) Figure 3 As shown, the control method of this embodiment may include the following steps S310 to S320.

[0052] Step S310: Acquire target environment data perceived by multiple target sensors of the mobile device.

[0053] In some examples, the target sensor may include a visual sensor and / or an ultrasonic sensor. The visual sensor may include multiple cameras mounted on the mobile device, such as multiple cameras with different perspectives in the front, back, left, and right. The ultrasonic sensor may include ultrasonic radars mounted at different locations on the mobile device. The target environment data may include visual perception data (e.g., video or images) captured by multiple cameras from different perspectives, and / or ultrasonic data (e.g., ultrasonic point cloud images) measured by ultrasonic radars at different locations.

[0054] Different target sensors can be used to perceive environmental data in different areas of a mobile device. Taking a vehicle as an example, with cameras mounted on the vehicle as the target sensors, four or more cameras can be installed at the front, rear, left, and right of the vehicle. Different cameras can capture visual perception data for different areas. For example, each camera can capture image data, or video data can be constructed based on image data from multiple times.

[0055] Step S320: Input the target environment data into the pre-generated target control model, and use the target control model to perform motion control on the mobile device.

[0056] In some examples, the target control model can parse target environment data to obtain corresponding environmental information, and at least based on this environmental information, perform motion control on the mobile device. For example, it can make decisions or plans based on the environmental information and the device information of the mobile device, and then perform motion control on the mobile device. For instance, if the mobile device is a vehicle, the target control model can be a model with driving automation functions, such as a vehicle's intelligent driving model. It should be noted that the specific model structure and training method of the target control model can be found in descriptions in related technologies, and this disclosure does not limit these aspects.

[0057] Furthermore, the environmental information obtained based on the target environmental data may include object information of one or more environmental objects. These environmental objects may include static and / or dynamic environmental objects. Static environmental objects may include, for example, lane lines, curbs, road markings, buildings, cones, traffic lights, and traffic signs. The object information of static environmental objects may include their position (e.g., center point coordinates) and geometry (e.g., length, width, and height). Dynamic environmental objects may include, for example, other mobile devices and pedestrians. The object information of dynamic environmental objects may include their position, geometry, speed, and orientation angle. It should be noted that the position of environmental objects can be described based on a global coordinate system or a device coordinate system of the mobile device. This device coordinate system can be constructed by using the center point of the mobile device as the origin, the direction of movement of the mobile device as the x-axis, and the direction perpendicular to the direction of movement of the mobile device as the y-axis.

[0058] Taking a target sensor as an example, the aforementioned multiple target sensors can include four or more target cameras installed on the front, back, left, and right sides of the mobile device. Different target cameras can capture environmental images or videos of different areas. The target control model can perform image processing (e.g., distortion correction and / or perspective transformation) on the environmental images captured by each target camera to obtain a top-down view characterizing the surrounding environment of the mobile device. This top-down view can also be called a bird's-eye view (BEV). The target control model can further analyze this top-down view to obtain corresponding environmental information and perform motion control on the mobile device based on this environmental information.

[0059] In some examples, the target control model described above can be pre-generated based on a preset artificial intelligence model. The data used to train the target control model may include virtual perception data generated by a data generation model. This data generation model can be used to generate virtual perception data at least in part based on the target environment image corresponding to the target sensor. The target environment image may be an image generated based on the sensor parameters of the target sensor and object description information describing the target environment objects that the target sensor can perceive.

[0060] For example, the data used to train the target control model can be called target training data, and the target control model can be generated after training based on the target training data. The data generation model can automatically generate the virtual sensing data as at least a part of the target training data.

[0061] The aforementioned virtual sensing data can be used to simulate environmental data obtained by the target sensors of a mobile device sensing the environment. For example, the virtual sensing data and the real environmental data sensed by the target sensor (such as the aforementioned target environmental data) can be in the same format, and both can represent the environmental objects sensed by the target sensor based on the same format. For example, if the sensors of a mobile device include multiple visual sensors and / or multiple ultrasonic sensors, the virtual sensing data can include virtual visual sensing data (such as virtual video or virtual images) used to simulate different visual sensors capturing images, and / or virtual ultrasonic data (such as virtual ultrasonic point cloud images) used to simulate different ultrasonic radar measurements.

[0062] In some examples, the data generation model can be any artificial intelligence model, such as Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), Diffusion Models (DMs), Diffusion Transformers (DiTs) that combine diffusion models and Transformer architectures, and other generative models with arbitrary structures. Based on this data generation model, virtual perceptual data such as images, videos, or point clouds can be generated from the input information. This data generation model can be a pre-trained model, and the pre-training method can be based on supervised or unsupervised learning. It should be noted that the specific structure and pre-training method of this data generation model can also be found in descriptions in related technologies, and will not be repeated in this embodiment.

[0063] Using the above method, target environment data perceived by multiple target sensors of a mobile device is acquired. This target environment data is then input into a pre-generated target control model, which controls the motion of the mobile device. The data used to train the target control model can include virtual perception data generated by a data generation model. This model can generate virtual perception data, at least partially, based on the target environment images corresponding to the target sensors. These target environment images can be generated based on the sensor parameters of the target sensors and object description information describing the target environment objects that the target sensors can perceive. In this way, the data generation model can automatically generate virtual perception data for training the target control model. The target environment images corresponding to multiple target sensors of the mobile device can more clearly and accurately describe the target environment objects that each target sensor can perceive, enabling fine-grained control of the data generation model. This makes the virtual perception data generated by the data generation model clearer and more accurate, and closer to the actual perception data of each target sensor. This allows for the automated and efficient generation of high-quality virtual perception data, further improving the reliability of the target control model trained on this virtual perception data, and also improving the reliability of motion control of the mobile device based on the target control model.

[0064] Figure 4 This is a schematic flowchart illustrating a data generation method provided in an embodiment of this disclosure. This data generation method can be executed by any electronic device, for example, by... Figure 1 The server shown is executing. (As shown) Figure 4 As shown, the data generation method of this embodiment may include the following steps S410 to S430.

[0065] Step S410: Obtain data description information used to generate virtual perception data.

[0066] In some examples, the virtual sensing data can be used to train a target control model, which can be used to control the motion of a mobile device that can be configured with multiple target sensors for sensing the environment.

[0067] In some examples, this data description information can be user input or information generated by automated tools, used to describe the virtual perceptual data that the user expects to simulate. This data description information can be used to control the data generation model in generating virtual perceptual data, and can also be called control information. The format of this data description information can include one or more of text, images, video, and sound.

[0068] For example, the data description information may include one or more of object description information, device description information, and environment description information.

[0069] The object description information can describe target environmental objects in the environment where the mobile device is located. For example, it can include information describing target environmental objects that can be perceived by target sensors. These target environmental objects can include static and / or dynamic environmental objects. Static environmental objects can include stationary environmental objects such as lane lines, curbs, road markings, buildings, cones, traffic lights, and traffic signs; dynamic environmental objects can include dynamically moving environmental objects such as other mobile devices and pedestrians. For example, the object description information can include target object description information for one or more target environmental objects. Each target object description can accurately describe the location, geometry, and other information of the target environmental object. For example, if the target environmental object is a lane line, the target object description information corresponding to the lane line can be an n*3 matrix, where n represents the number of location points of the lane line, and 3 represents the three-dimensional coordinates of each location point of the lane line. The target object description information can also include the color and / or type of the lane line, such as a white dashed line, a white solid line, or double yellow lines, etc. It should be noted that the target environmental object can also be called a target environmental feature, and the static environmental object can be a map feature or map element obtained through a map. This object description information can also be called feature control information or feature description information.

[0070] The device description information can be used to describe the device information of a mobile device. This device information may include one or more of the following: location information, geometric shape information, status information, trajectory information, and sensor information. The geometric shape information may include the length, width, and height of the mobile device; the status information may include one or more of the following: speed information, acceleration information, and orientation angle information; the sensor information may include sensor parameters (such as intrinsic and extrinsic parameters) of one or more cameras installed on the mobile device; and the trajectory information may include the position of the mobile device at various times, forming the motion trajectory of the mobile device through the positions at multiple times.

[0071] This environmental description information can be used to describe the environment in which the mobile device is located. For example, it can describe one or more aspects of the scene information, such as weather, time, lighting, and area type. The area type can represent the category of the area where the mobile device is located, such as urban commercial area, urban residential area, rural area, school, factory, toll station, highway, etc. Lighting information can include natural light intensity and / or streetlight intensity. This environmental description information provides a macro-level context, helping the data generation model determine the overall scene information, thereby generating more accurate virtual perception data.

[0072] In some examples, the data description information mentioned above may include data frame description information at multiple times. Each data frame description information at any time may include one or more of the device description information, object description information, and environment description information mentioned above, to describe the mobile device, external object, and environment at that time.

[0073] Step S420: For each target sensor of the mobile device, generate a target environment image corresponding to the target sensor based on the sensor parameters and object description information of the target sensor.

[0074] In some examples, the target environment image corresponding to the target sensor can be a target environment image from the perspective of the target sensor. This target environment image can be used to characterize the semantic information of various environmental objects. For example, the target environment image can be a line drawing image (also known as a semantic sketch), which can intuitively describe the environmental object information from the perspective of the target sensor.

[0075] The sensor parameters of the aforementioned target sensor can include camera intrinsic and extrinsic parameters. The camera intrinsic parameters determine the camera's imaging model and may include parameters such as focal length, principal point coordinates (e.g., the origin of the imaging plane), and distortion coefficients. The camera extrinsic parameters determine the position and orientation of the target sensor. For example, the camera extrinsic parameters may include, for instance, the rotation matrix and translation vector of the target sensor within a specific coordinate system. This specific coordinate system could be a device coordinate system with the center point of the mobile device as its origin or a world coordinate system determined based on latitude and longitude. Coordinate transformation between image coordinates and this specific coordinate system can be achieved through the camera intrinsic and extrinsic parameters.

[0076] In some examples, the object description information mentioned above may include the three-dimensional coordinate information of one or more target environment objects. The three-dimensional coordinate information of these objects can constitute a three-dimensional scene. Using perspective projection techniques in computer graphics, one or more target environment objects in the three-dimensional scene can be projected onto a two-dimensional image plane, thereby obtaining a target environment image. For example, if the three-dimensional coordinate information of the target environment objects is in the aforementioned specific coordinate system, the three-dimensional coordinate information of the target environment objects can be converted into coordinate information in the camera coordinate system using camera extrinsic parameters. Then, based on the converted coordinate information, perspective projection is performed using camera intrinsic parameters to transform the target environment objects into the image coordinate system.

[0077] For example, if the target environment object is represented in the form of a line (e.g., lane lines), each position point on the linear target environment object can be projected and transformed into the image coordinate system in the manner described above. Connecting the transformed position points will yield the representation of the linear target environment object in the target environment image. If the target environment object is represented in the form of a detection box (e.g., a vehicle or pedestrian), the three-dimensional coordinates of each corner point of the detection box can be transformed into the image coordinate system in the manner described above. Connecting the different corner points according to the original connection order of the detection box will yield the representation of the detection box in the target environment image.

[0078] Furthermore, the target environment image may also include color information of the target environment objects, for example, distinguishing white solid lines from yellow solid lines. The type of target environment objects can be distinguished by color and / or shape, with different types of target environment objects using different colors and / or shapes.

[0079] Step S430: Input the target environment image into the data generation model to obtain the virtual perception data output by the data generation model.

[0080] The data generation model can include generative models of any structure, such as generative adversarial networks, variational autoencoders, diffusion models, and diffusion transformation models (DiT) that combine diffusion models and Transformer architectures.

[0081] In some examples, the virtual sensing data output by the data generation model can be used to train the aforementioned target control model. For instance, the target control model can be trained solely based on the virtual sensing data output by the data generation model, or it can be trained using both virtual sensing data and labeled real sensing data to improve the reliability of the target control model, thereby enhancing the reliability of motion control for mobile devices.

[0082] The virtual sensing data (video, image, or point cloud) can have spatial resolution, which characterizes the actual physical size represented by each pixel or point in the virtual sensing data. Based on this spatial resolution, the level of detail and the ability to capture details of objects that can be displayed in the virtual sensing data can be characterized. Higher spatial resolution means that there are more and denser pixels or points within a fixed physical size (such as the sensing range of a sensor). High spatial resolution can more clearly present the edges, textures, and subtle features of environmental objects.

[0083] In some examples, the spatial resolution of the virtual sensing data can be a user-preset target resolution. This target resolution can be one or more; for example, the data generation model can output multiple sets of virtual sensing data at different resolutions. Training the target control model using these different resolutions as part or all of the training data can improve the reliability of the target control model at different resolutions. If there are multiple target resolutions, they can include a pre-set minimum resolution (e.g., 720P) and a maximum resolution (e.g., 4K), and can also include one or more resolutions between the minimum and maximum resolutions (e.g., 1080P and 2K).

[0084] Figure 5 This is a schematic diagram of a data generation model provided in an embodiment of this disclosure. This data generation model can be deployed in any electronic device to execute the data generation method provided in this embodiment.

[0085] like Figure 5 As shown, the data generation model 500 may include a data generation module 510 and a data adjustment module 520. The data generation module can focus on extracting and generating temporal information, and spatially generate spatial information at a preset resolution. The data adjustment module can focus on extracting and generating spatial information, adjusting the resolution of the spatial information at the preset resolution to generate spatial information at a target resolution, which may be greater than or equal to the preset resolution. By processing temporal and spatial information separately in two modules, the complexity of the data generation model can be reduced, and the efficiency of model training and operation can be improved.

[0086] In some examples, the data generation model 500 may further include a first encoding module 530, which can be used to encode the target environment image to obtain first encoded features. The first encoding module may include any neural network, such as a convolutional neural network, a deep neural network, or similar structures. Further, the data generation module can be used to generate first perceptual data at a preset resolution, at least partially, based on the first encoded features; the data adjustment module can be used to adjust the spatial resolution of the first perceptual data, at least partially, based on the first encoded features, to obtain second perceptual data. The first encoded features can be directly input into the data generation module, or they can be concatenated and fused with random noise in terms of feature dimensions before being input into the data generation module. The random noise can serve as a random seed for data generation, and the data generation module can start from the random noise and gradually generate data based on control information (such as the first encoded features or other control information) to obtain the first perceptual data. Optionally, the random noise can be Gaussian noise or other arbitrary random noise.

[0087] In some examples, the data generation model 500 may further include a second encoding module 540. The first encoding module can be used to encode environmental description information and / or device description information in the data description information to obtain second encoded features. The second encoding module may include any neural network, such as a convolutional neural network, a deep neural network, or similar structures. The second encoded features can be input into the data generation module, which can then generate first perceptual data based on the first and second encoded features. For example, the second encoded features can be injected into the data generation module using an attention mechanism to control the generation of the first perceptual data.

[0088] It should be noted that, in Figure 5 In the example shown, the data description information used to generate virtual sensing data may include one or more of object description information, device description information, and environment description information. The object description information can be converted into a target environment image and then input into the first encoding module. The target environment image can be an environment image from the perspective of the target sensor; different target sensors correspond to different perspectives of the target environment image. Optionally, the target environment image can also be an environment image from a global perspective, such as a BEV image.

[0089] based on Figure 5 The data generation model shown above, in step S430, the method of generating virtual perception data based on the data generation model may include the following steps S431 to S434.

[0090] Step S431: Encode the target environment images corresponding to multiple target sensors based on the first encoding module to obtain the first encoded features.

[0091] For example, the first encoding module may include any neural network structure with feature extraction function, such as a convolutional neural network. After processing the target environment image, such as compression encoding, the first encoding feature with physical meaning is extracted from the target environment image. The first encoding feature can be used to characterize one or more of the following features: the overall structure of the target environment image, the position, shape, type, edge, texture, etc. of each target environment object in the target environment image.

[0092] The first encoding module can compress and refine the target environment image to obtain the most critical information for subsequent data generation tasks, and filter out redundant details, thereby improving the processing efficiency and accuracy of the data generation module. Furthermore, since the target environment image used in this embodiment corresponds to the target sensor—that is, it is the image from the target sensor's perspective—the subsequent data generation module can perform fine-grained control over the scene information from the target sensor's perspective, improving the accuracy and clarity of the generated perception data.

[0093] Step S432: The data generation module generates first perceptual data based at least in part on the first coded features.

[0094] The data generation module can generate first perceptual data that conforms to the conditions indicated by the first encoding feature, based on the first encoding feature as at least part of the generation conditions. This makes the generated first perceptual data have a high similarity or correlation with the content of the target environment image. For example, the first perceptual data can simulate the position and / or shape of each target environment object in the target environment image.

[0095] In some examples, the data generation module may include generative models of arbitrary structures, such as generative adversarial networks, variational autoencoders, diffusion models, and diffusion transformation models (DiT) that combine diffusion models and Transformer architectures.

[0096] In other examples, the data generation module 510 may include a first submodule 511 and a second submodule 512, wherein the first submodule can be used to determine the temporal information of multiple first data frames in the first sensing data, and the second submodule can be used to generate spatial information of various target environment objects in the first data frames. Optionally, the data generation module may also be referred to as a temporal generation module.

[0097] Furthermore, the first and second submodules can be cascaded and stacked N times, where N is any positive integer greater than or equal to 1. Thus, the data generation module can include N first submodules and N second submodules cascaded and stacked.

[0098] For example, the first submodule can be a Spatial-Temporal Diffusion Transformer (STDiT), such as STDiT3-t; the second submodule can be a Multi-View Diffusion Transformer (MVDiT), such as MVDiT3. The first submodule can be used to determine temporal information, and the second submodule can be used to determine the spatial information of different target sensors. It should be noted that the specific implementation methods of STDiT and MVDiT can be found in the descriptions in related technologies, and will not be repeated here. The first and second submodules can also be other types of network structures, such as ordinary diffusion models or diffusion transform models (DiT), which are not limited in this embodiment.

[0099] Furthermore, the spatial information generated by the second submodule can be spatial information at a preset resolution, so the spatial resolution of the obtained first perceptual data can also be a preset resolution. This preset resolution can be a fixed resolution pre-set by the user, for example, the minimum resolution among multiple target resolutions pre-set by the user. In this way, the data generation module only needs to generate spatial information at the preset resolution, allowing it to focus on generating temporal information, reducing the complexity of the data generation module, and improving the efficiency of its training and operation.

[0100] In some examples, the aforementioned data description information may further include device description information for describing the device information of the mobile device and / or environmental description information for describing the environment in which the mobile device is located. The aforementioned data generation model may also include a second encoding module 540, which can be used to encode the device description information and / or environmental description information to obtain a second encoded feature.

[0101] In this step, the first and second coding features can be input into the data generation module to obtain the first perceptual data output by the data generation module.

[0102] For example, the second encoded feature can be injected into the data generation module in a manner based on an attention mechanism to control the generation of the first perceived data.

[0103] Step S433: The spatial resolution of the first sensing data is adjusted by the data adjustment module at least in part according to the first coding feature to obtain the second sensing data.

[0104] In some examples, the spatial resolution of the second sensed data differs from that of the first sensed data. For instance, the spatial resolution of the second sensed data may be greater than that of the first sensed data. For example, the spatial resolution of the first sensed data may be 720P, while the spatial resolution of the second sensed data may be 1080P and / or 2K.

[0105] In some examples, the second sensing data may include sensing data at different resolutions, thereby enabling dynamic adjustment of the resolution. For instance, the input to the data adjustment module may also include one or more target resolutions. By adjusting the spatial resolution of the first sensing data according to the first coding features, one or more second sensing data corresponding to the target resolution can be obtained.

[0106] In some examples, the data adjustment module 520 may include a third submodule 521, which can be used to adjust the spatial resolution of different target environment objects in the first data frame. Optionally, the third submodule and the second submodule may be neural network models with the same structure. For example, both the third submodule and the second submodule may be multi-view diffusion transform models.

[0107] For example, the third submodule can correct and upsample the first perceptual data according to the first encoded features. The correction restores the image distortion in the output data, and the upsampling expands the resolution of the output data, thereby obtaining the second perceptual data with higher clarity.

[0108] It should be noted that inputting the first encoding feature into the data adjustment module enables the data adjustment module to support fine-grained and controllable data generation. This allows the resolution of the first data to be adjusted based on the first encoding feature of the original target environment image during the resolution adjustment process, avoiding distortion of the target environment image caused by the data generation module's processing, and resulting in higher accuracy of the second perceptual data after the resolution is improved.

[0109] Step S434: Generate virtual sensing data based on the first sensing data and / or the second sensing data.

[0110] For example, the first sensing data can be used as virtual sensing data, the second sensing data can be used as virtual sensing data, or both the first and second sensing data can be used as virtual sensing data. The virtual sensing data (e.g., the first sensing data and / or the second sensing data) may include visual sensing data (e.g., video sequences) simulating observations from multiple different target camera perspectives, and / or ultrasonic data (e.g., ultrasonic point cloud image sequences) simulating ultrasonic radar measurements at different locations.

[0111] In some examples, the virtual sensing data (e.g., first sensing data and / or second sensing data) may include virtual environment objects and corresponding label information. This label information can characterize the virtual environment object's position, type, shape, and other information. This label information can be used as ground truth for training the target control model, enabling the trained model to accurately identify the virtual environment object and reliably control the motion of the mobile device based on it.

[0112] In some examples, the data generation model described above can be pre-trained, where the data generation module and the data adjustment module can be trained independently or jointly. Using independent training allows for parallel training of the data generation and data adjustment modules, further improving model training efficiency.

[0113] By employing the method described in this embodiment, the separation of the data generation module and the data adjustment module achieves decoupling of the spatiotemporal dimension, thereby reducing the computational resource requirements and time costs for running the data generation model, improving the image distortion problem in complex scenes, and thus improving the training efficiency and running efficiency of the data generation model. This enables the automatic and efficient generation of a large amount of high-quality virtual perception data for training the target control model.

[0114] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. For example... Figure 6 As shown, the electronic device 1000 may include a memory 1010 and a processor 1020. The memory 1010 may be used to store computer instructions, and the processor 1020 may be used to retrieve computer instructions from the memory 1010 to execute all or part of the steps of any of the methods in the foregoing embodiments of this disclosure. The processor may be one or more processors, which may execute instructions individually or jointly. Similarly, the memory may be one or more memories, which may store the aforementioned computer instructions individually or jointly.

[0115] In some examples, the electronic device can be Figure 1 The electronic device is a server and / or a mobile device. In other examples, the electronic device may be any electronic device, such as a controller for a mobile device or an electronic device for performing all or part of the steps of any of the methods in the foregoing embodiments of this disclosure.

[0116] This disclosure also provides a mobile device that may include a memory and a processor. The memory may be used to store computer instructions, and the processor may be used to retrieve the computer instructions from the memory to perform all or part of the steps of any of the methods in the foregoing embodiments of this disclosure. The processor may be one or more processors, which may execute the instructions individually or jointly. Similarly, the memory may be one or more memories, which may store the aforementioned computer instructions individually or jointly.

[0117] The mobile device provided in this embodiment can be... Figure 1 or Figure 2The mobile device shown is, in some examples, a vehicle that can be an electric vehicle, a hybrid vehicle, a fuel cell vehicle, or another type of vehicle. For example, the vehicle could be one equipped with autonomous driving features.

[0118] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods in the foregoing embodiments of this disclosure. Optionally, the computer-readable storage medium may be a non-transitory storage medium, but is not limited thereto, and may also be a temporary storage medium.

[0119] This disclosure also provides a chip that may include a processing unit, which can be used to execute all or part of the steps of any of the methods in the foregoing embodiments of this disclosure. The chip may be in the form of an Application-Specific Integrated Circuit (ASIC), a System-on-Chip (SOC), a Field-Programmable Gate Array (FPGA), etc., and this embodiment is not limited to this. Optionally, the chip may further include a storage unit, which can be used to store computer instructions. The processing unit can be used to retrieve the computer instructions from the storage unit to execute all or part of the steps of any of the methods in the foregoing embodiments of this disclosure.

[0120] This disclosure also provides a computer program product that may include a computer program that, when executed by a processor, can implement any of the methods described in the foregoing embodiments of this disclosure.

[0121] This disclosure may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement any of the methods in the foregoing embodiments of this disclosure.

[0122] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media may include, for example, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), compact disc-read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any combination thereof. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0123] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0124] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Smalltalk, C++, etc.) and conventional procedural programming languages ​​(e.g., the "C" language or similar programming languages). The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network (e.g., a local area network or a wide area network), or it may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays, or programmable logic arrays, may execute computer-readable program instructions to implement various aspects of the embodiments of this disclosure by utilizing state information from the computer-readable program instructions.

[0125] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0126] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0127] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It should be noted that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are all equivalent.

[0129] Various embodiments of the present disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. A control method, characterized in that, The method includes: Acquire target environment data perceived by multiple target sensors of a mobile device; The target environment data is input into a pre-generated target control model, and the motion control of the mobile device is performed through the target control model. The data used to train the target control model includes virtual perception data generated by a data generation model. The data generation model generates the virtual perception data at least in part based on the target environment image corresponding to the target sensor. The target environment image is an image generated based on the sensor parameters and object description information of the target sensor. The object description information is used to describe the target environment objects that the target sensor can perceive.

2. The method according to claim 1, characterized in that, The virtual sensing data is generated in the following way: Obtain data description information used to generate the virtual perception data; wherein, the data description information includes the object description information; For each target sensor of the mobile device, a target environment image corresponding to the target sensor is generated based on the sensor parameters of the target sensor and the object description information; The target environment image is input into the data generation model to obtain virtual perception data output by the data generation model.

3. The method according to claim 2, characterized in that, The data generation model includes a first encoding module, a data generation module, and a data adjustment module; the step of inputting the target environment image into the data generation model to obtain the virtual perception data output by the data generation model includes: The first encoding feature is obtained by encoding the target environment images corresponding to the multiple target sensors based on the first encoding module. The data generation module generates first perceptual data based at least in part on the first encoded feature; The data adjustment module adjusts the spatial resolution of the first sensing data at least in part according to the first encoding feature to obtain the second sensing data. The virtual sensing data is generated based on the first sensing data and / or the second sensing data.

4. The method according to claim 3, characterized in that, The data generation module includes a first submodule and a second submodule. The first submodule is used to determine the temporal information of a first data frame at multiple times in the first sensed data, and the second submodule is used to generate the spatial information of each different target environment object in the first data frame.

5. The method according to claim 4, characterized in that, The data adjustment module includes a third sub-module, which is used to adjust the spatial resolution of each different target environment object in the first data frame to obtain the second data frame at multiple times in the second perception data. The third submodule and the second submodule are neural network models with the same structure.

6. The method according to claim 3, characterized in that, The data description information further includes device description information for describing the device information of the mobile device and / or environmental description information for describing the environment in which the mobile device is located; the data generation model further includes a second encoding module, which is used to encode the device description information and / or the environmental description information to obtain a second encoding feature; The generation of first perceptual data based on the data generation module, at least in part, the first encoded feature, includes: The first encoded feature and the second encoded feature are input into the data generation module to obtain the first perceived data output by the data generation module.

7. The method according to any one of claims 3 to 6, characterized in that, The data generation module and the data adjustment module are modules that are trained independently. The target environment object includes static environment objects and / or dynamic environment objects.

8. A data generation method, characterized in that, The method includes: Acquire data description information for generating virtual sensing data; wherein, the virtual sensing data is used to train a target control model, the target control model is used to perform motion control on a mobile device, the mobile device is configured with multiple target sensors for sensing the environment, and the data description information includes object description information describing the target environmental objects that the target sensors can sense; For each target sensor of the mobile device, a target environment image corresponding to the target sensor is generated based on the sensor parameters of the target sensor and the object description information; The target environment image is input into the data generation model to obtain virtual perception data output by the data generation model.

9. An electronic device, characterized in that, The method includes a memory and a processor, the memory being used to store computer instructions, and the processor being used to retrieve the computer instructions from the memory to perform the method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images