A data generation method, device, readable storage medium and terminal equipment

Abstract

The application belongs to the technical field of data processing, and particularly relates to a data generation method and device, a computer readable storage medium and a terminal device. The method generates multi-modal data based on a preset generative adversarial network to obtain virtual multi-modal data with spatial consistency. The multi-modal data includes images and radar point clouds. The generative adversarial network is trained in advance by using real multi-modal data with spatial consistency. According to the embodiment of the application, the generative adversarial network can be trained in advance by using real multi-modal data with spatial consistency, so that the generative adversarial network can learn the characteristics of the real multi-modal data, thereby having corresponding data generation capability to generate virtual multi-modal data with spatial consistency.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a data generation method, apparatus, computer-readable storage medium and terminal equipment. Background Technology

[0002] With the increasing number of car users, road traffic problems are becoming more and more serious, thus highlighting the importance of autonomous driving technology. Deep learning algorithms are the core technology in autonomous driving, and their essence lies in fitting the target domain distribution of multimodal data in a dataset. Multimodal data includes image data and radar point cloud data, and the size of the data plays a decisive role in the accuracy of the fitting result. However, due to the difficulty in obtaining real multimodal data, it is necessary to use network models to generate virtual multimodal data. Existing related technologies only support the generation of virtual image data and cannot simultaneously generate virtual radar point cloud data with spatial consistency with the virtual image data. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data generation method, apparatus, computer-readable storage medium, and terminal device to solve the problem that the prior art cannot generate virtual multimodal data with spatial consistency.

[0004] A first aspect of this application provides a data generation method, which may include: Multimodal data is generated based on a pre-defined generative adversarial network to obtain virtual multimodal data with spatial consistency. The multimodal data includes images and radar point clouds; the generative adversarial network is pre-trained on real multimodal data with spatial consistency.

[0005] In one specific implementation of the first aspect, before generating multimodal data based on a preset generative adversarial network to obtain spatially consistent virtual multimodal data, the method further includes: Acquire real images and real radar point clouds acquired simultaneously; The real radar point cloud is mapped onto the real image according to a preset data mapping relationship to obtain the real multimodal data; The generative adversarial network is trained using the real multimodal data to obtain the trained generative adversarial network.

[0006] In one specific implementation of the first aspect, acquiring the synchronously acquired real images and real radar point clouds includes: A preset synchronization trigger sends a synchronized acquisition signal to the camera and radar, enabling the camera and radar to acquire data synchronously. Acquire the real image captured by the camera in response to the acquisition signal; Obtain the radar response to the real radar point cloud acquired by the acquired signal.

[0007] In one specific implementation of the first aspect, the data mapping relationship includes a translation matrix and a rotation matrix from the radar's coordinate system to the camera's coordinate system; The step of mapping the real radar point cloud onto the real image according to a preset data mapping relationship to obtain the real multimodal data includes: The real radar point cloud is mapped onto the real image according to the translation matrix and the rotation matrix to obtain the real multimodal data.

[0008] In one specific implementation of the first aspect, training the generative adversarial network based on the real multimodal data to obtain a trained generative adversarial network includes: The real multimodal data and random mask are input into the generative adversarial network (GAN), and the output data of the GAN is obtained. The training loss is calculated based on the real multimodal data and the output data; The parameters of the generative adversarial network are adjusted according to the training loss until the preset convergence condition is met, thus obtaining the trained generative adversarial network.

[0009] In one specific implementation of the first aspect, the generation of multimodal data based on a preset generative adversarial network to obtain spatially consistent virtual multimodal data includes: A preset specific noise signal is input into the generative adversarial network to generate multimodal data, and the virtual multimodal data output by the generative adversarial network is obtained.

[0010] In one specific implementation of the first aspect, after generating multimodal data based on a preset generative adversarial network to obtain spatially consistent virtual multimodal data, the method further includes: The virtual multimodal data is decomposed into virtual images and virtual radar point clouds.

[0011] A second aspect of the embodiments of this application provides a data generation apparatus, which may include: The data generation module is used to generate multimodal data based on a preset generative adversarial network to obtain virtual multimodal data with spatial consistency. The multimodal data includes images and radar point clouds; the generative adversarial network is pre-trained on real multimodal data with spatial consistency.

[0012] In one specific implementation of the second aspect, the data generation device further includes: The synchronous acquisition module is used to acquire synchronously collected real images and real radar point clouds; The mapping module is used to map the real radar point cloud onto the real image according to a preset data mapping relationship to obtain the real multimodal data; The training module is used to train the generative adversarial network based on the real multimodal data to obtain the trained generative adversarial network.

[0013] In one specific implementation of the second aspect, the synchronization acquisition module includes: The triggering submodule is used to send a synchronized acquisition signal to the camera and the radar through a preset synchronization trigger, so that the camera and the radar can perform synchronized data acquisition. The first acquisition submodule is used to acquire the real image captured by the camera in response to the acquisition signal; The second acquisition submodule is used to acquire the real radar point cloud acquired by the radar in response to the acquisition signal.

[0014] In one specific implementation of the second aspect, the data mapping relationship includes a translation matrix and a rotation matrix from the radar's coordinate system to the camera's coordinate system; The mapping module includes: The point cloud mapping submodule is used to map the real radar point cloud onto the real image according to the translation matrix and the rotation matrix to obtain the real multimodal data.

[0015] In one specific implementation of the second aspect, the training module includes: The input submodule is used to input the real multimodal data and random mask into the generative adversarial network and obtain the output data of the generative adversarial network. The calculation submodule is used to calculate the training loss based on the real multimodal data and the output data; The adjustment submodule is used to adjust the parameters of the generative adversarial network according to the training loss until the preset convergence condition is met, so as to obtain the trained generative adversarial network.

[0016] In one specific implementation of the second aspect, the data generation module includes: A preset specific noise signal is input into the generative adversarial network to generate multimodal data, and the virtual multimodal data output by the generative adversarial network is obtained.

[0017] In one specific implementation of the second aspect, the data generation device further includes: The decomposition module is used to decompose the virtual multimodal data to obtain virtual images and virtual radar point clouds.

[0018] A third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described data generation methods.

[0019] A fourth aspect of this application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described data generation methods.

[0020] A fifth aspect of this application provides a computer program product that, when run on a terminal device, causes the terminal device to execute the steps of any of the above-described data generation methods.

[0021] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment generates multimodal data based on a preset generative adversarial network (GAN) to obtain spatially consistent virtual multimodal data; wherein, the multimodal data includes images and radar point clouds; the GAN is pre-trained on spatially consistent real multimodal data. Through this application embodiment, the GAN can be pre-trained using spatially consistent real multimodal data, enabling the GAN to learn the characteristics of real multimodal data, thereby possessing the corresponding data generation capability to generate spatially consistent virtual multimodal data. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic flowchart for training a generative adversarial network; Figure 2 A schematic diagram for external parameter calibration; Figure 3 This is a flowchart illustrating the training of a generative adversarial network based on real multimodal data. Figure 4 A schematic diagram of a generative adversarial network model; Figure 5 A schematic diagram for training a generative adversarial network; Figure 6 This is a schematic structural diagram of a data generation device according to an embodiment of this application; Figure 7 This is a schematic block diagram of a terminal device in an embodiment of this application. Detailed Implementation

[0024] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0026] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0027] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0029] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] Autonomous driving is a new-era technology combining vehicle-to-everything (V2X) communication and artificial intelligence. It can coordinate travel routes and planning times, thereby improving travel efficiency and reducing energy consumption to some extent. The core technology of autonomous driving is deep learning algorithms. Obtaining highly accurate deep learning algorithms requires massive amounts of data. However, datasets for autonomous driving include both radar point cloud data and image data, making data collection a major challenge. Therefore, generative adversarial networks (GANs) are used to generate data for deep learning algorithms. However, current GANs only support the generation of virtual image data and cannot simultaneously generate spatially consistent virtual radar point cloud data. Based on this, this application provides a method for simultaneously generating spatially consistent virtual image data and virtual radar point cloud data.

[0031] In this embodiment, the generative adversarial network can be trained in advance using real multimodal data with spatial consistency, so that the generative adversarial network can learn the characteristics of real multimodal data and thus have the corresponding data generation capability to generate virtual multimodal data with spatial consistency.

[0032] like Figure 1 As shown, the training process of a generative adversarial network can specifically include the following steps: Step S101: Acquire the synchronously acquired real images and real radar point clouds.

[0033] In one specific implementation of this application, a preset synchronization trigger can send a synchronized acquisition signal to the camera and radar, enabling the camera and radar to perform synchronized data acquisition, thereby obtaining the real image data and real radar point cloud data acquired by the camera and radar in response to the acquisition signal, respectively. It should be noted that the camera and radar can acquire data over the same spatial area, thus ensuring spatial consistency between the acquired real image data and real radar point cloud data.

[0034] Step S102: Map the real radar point cloud onto the real image according to the preset data mapping relationship to obtain real multimodal data.

[0035] The data mapping relationship can include translation and rotation matrices from the radar coordinate system to the camera coordinate system. Based on the translation and rotation matrices, real radar point cloud data can be mapped to real image data, thereby obtaining real multimodal data.

[0036] In one specific implementation of this application, extrinsic parameter calibration can be performed in advance to obtain translation and rotation matrices. Figure 2The diagram shows the external parameter calibration. The camera can calculate the representation of the calibration plane in the camera coordinate system by using the QR code or checkerboard of the calibration plane. At the same time, the beam emitted by the radar falls on the calibration plane (black dot in the figure). Using the coordinates of the radar point in the radar coordinate system and the coordinates of the plane equation in the camera coordinate system, the constraints of the point on the plane are constructed, thereby solving for the translation matrix and rotation matrix.

[0037] After obtaining the translation and rotation matrices, the actual radar point cloud can be mapped according to the following formula:

[0038] in, For the mapped radar point cloud data, The original radar point cloud data is represented by I, which is the identity matrix, t, which is the translation matrix, and R, which is the rotation matrix.

[0039] The rotation matrix R can be expressed as: R = R x ·R y ·R z

[0040]

[0041]

[0042] in, To bypass x The rotation matrix of the axis of rotation. To bypass y The rotation matrix of the axis of rotation. To bypass z The rotation matrix of the axis of rotation. To bypass x Euler angles of rotation of the axis, To bypass y Euler angles of rotation of the axis, To bypass z Euler angles for rotation of the axis.

[0043] In one specific implementation of this application, the image data can be expressed in the form of [C, H, W], where C is the number of channels. In this application embodiment, the acquired image may include three channels: red channel (R), green channel (G), and blue channel (B). H is the height of the image, and W is the width of the image.

[0044] Based on the existing channels of the image data, a new range channel (D) can be added to store the mapped radar point cloud data. Specifically, the original radar point cloud data... After mapping, the mapped radar point cloud data is obtained. That is, the corresponding distance in the channel is The location, and the value at that location, is from the original radar point cloud data. The distance value between the location and the camera is represented. Generally, the original radar point cloud data is only mapped to a few locations in the distance channel (D), with the other locations having a value of 0. Therefore, the data matrix in the distance channel (D) is generally a sparse matrix.

[0045] After the mapping process is completed, four-channel (R, G, B, D) data can be obtained and used as real multimodal data. Step S103: Train the generative adversarial network based on real multimodal data to obtain the trained generative adversarial network.

[0046] In one specific implementation of this application embodiment, step S103 may specifically include, as follows: Figure 3 The process shown: Step S301: Input real multimodal data and random mask into the generative adversarial network and obtain the output data of the generative adversarial network.

[0047] Figure 4 The diagram shows a model of a Generative Adversarial Network (GAN). A GAN consists of two models: a generator model G and a discriminator model D. The purpose of the generator model G is to generate outputs that are as similar as possible to the real data in order to deceive the discriminator model D. The purpose of the discriminator model D is to determine whether the data generated by the generator model G is similar to the real data.

[0048] In one specific implementation of this application, real multimodal data can be masked using random masking, and the generative adversarial network can repair the masked portion and use the repaired data as output data.

[0049] Step S302: Calculate the training loss based on the real multimodal data and output data.

[0050] In one specific implementation of this application, real multimodal data can be used as a supervision signal and compared with the output data of the generative adversarial network to calculate the training loss. The specific calculation method of the training loss can be set according to the actual situation. For example, any loss function in the prior art can be used to calculate the training loss, and this application does not specifically limit it.

[0051] Step S303: Adjust the parameters of the generative adversarial network according to the training loss until the preset convergence condition is met, and obtain the trained generative adversarial network.

[0052] Since generative adversarial networks have a generative model G and a discriminative model D, parameter adjustment involves two models. In this embodiment, the parameters of the generative model G are fixed and the parameters of the discriminative model D are adjusted, and then the parameters of the generative model G are fixed and the parameters of the discriminative model D are adjusted as one parameter adjustment.

[0053] After calculating the training loss, the parameters of the generative adversarial network (GAN) can be adjusted based on the training loss. In this embodiment, assuming that in the initial state, the parameters of the generative model G and the discriminative model D of the GAN are W1 and W2 respectively, the training loss is backpropagated to modify the parameters W1 and W2 of the GAN and the discriminative model D, resulting in modified parameters W3 and W4. After modifying the parameters, the next training process is executed. In this training process, the training loss is recalculated, and the training loss is backpropagated to modify the parameters W3 and W4 of the GAN, resulting in modified parameters W5 and W6, and so on. This process is repeated continuously, and the parameters can be modified in each training process until a preset convergence condition is met. The convergence condition can be set according to the actual situation. For example, the convergence condition can be set so that the generative model G can generate "indistinguishable" virtual multimodal data, and the discriminative model D has difficulty determining whether the virtual multimodal data generated by the generative model G is real multimodal data.

[0054] Figure 5 The diagram shown illustrates the training of the generative adversarial network (GAN) in this embodiment. A synchronization trigger generates a synchronous acquisition signal and sends it to the camera and radar, enabling the camera and radar to synchronously acquire real image data and real radar point cloud data. Based on a preset data mapping relationship, the real radar point cloud data is mapped to the real image data to obtain real multimodal data. The real multimodal data and a random mask are input into the GAN for training to obtain virtual multimodal data generated by the GAN. The real multimodal data and the virtual multimodal data generated by the GAN are compared and the training loss is calculated. Based on the training loss, the parameters of the GAN are adjusted using the backpropagation method until the preset convergence condition is met, thus obtaining the trained GAN.

[0055] After training the generative adversarial network (GAN), multimodal data can be generated based on the trained GAN. Since the GAN in this embodiment is trained on real multimodal data with spatial consistency, the generated virtual multimodal data also has spatial consistency.

[0056] In one specific implementation of this application, a preset specific noise signal can be input into a generative adversarial network to generate multimodal data, and the virtual multimodal data output by the generative adversarial network can be obtained.

[0057] The specific signal selected as the noise signal can be set according to the actual situation. For example, it may include, but is not limited to, sine waves, triangle waves, square waves or other controllable signals. This application embodiment does not make specific limitations in this regard.

[0058] In one specific implementation of this application, N different specific noise signals (N is a positive integer) can be preset, and the corresponding virtual multimodal data can be generated by using different types of specific noise signals, so that the generation result is controllable.

[0059] In one specific implementation of this application, after generating virtual multimodal data based on a generative adversarial network, the virtual multimodal data can be further decomposed into virtual images and virtual radar point clouds.

[0060] The process of multimodal data decomposition is the inverse process of step S102. The virtual multimodal data consists of four channels (R, G, B, D). Three of the channels (R, G, B) can be used as virtual images, and the data in the distance channel (D) can be inversely mapped according to the data mapping relationship to obtain the corresponding virtual radar point cloud.

[0061] The virtual images and virtual radar point clouds obtained from the decomposition have spatial consistency, which can be added to the dataset for autonomous driving to further improve the performance of autonomous driving.

[0062] In summary, the embodiments of this application generate multimodal data based on a pre-set generative adversarial network (GAN) to obtain spatially consistent virtual multimodal data. The multimodal data includes images and radar point clouds. The GAN is pre-trained with spatially consistent real multimodal data. Through these embodiments, the GAN can be pre-trained with spatially consistent real multimodal data, enabling it to learn the characteristics of the real multimodal data and thus acquire the corresponding data generation capability to generate spatially consistent virtual multimodal data.

[0063] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0064] Corresponding to the data generation method described in the above embodiments, Figure 6This illustration shows a structural diagram of one embodiment of a data generation apparatus provided in this application.

[0065] In this embodiment, a data generation device may include: The data generation module 601 is used to generate multimodal data based on a preset generative adversarial network to obtain virtual multimodal data with spatial consistency. The multimodal data includes images and radar point clouds; the generative adversarial network is pre-trained on real multimodal data with spatial consistency.

[0066] In one specific implementation of this application embodiment, the data generation device may further include: The synchronous acquisition module 602 is used to acquire synchronously collected real images and real radar point clouds; The mapping module 603 is used to map the real radar point cloud onto the real image according to a preset data mapping relationship to obtain the real multimodal data; The training module 604 is used to train the generative adversarial network based on the real multimodal data to obtain the trained generative adversarial network.

[0067] In one specific implementation of this application embodiment, the synchronization acquisition module includes: The triggering submodule is used to send a synchronized acquisition signal to the camera and the radar through a preset synchronization trigger, so that the camera and the radar can perform synchronized data acquisition. The first acquisition submodule is used to acquire the real image captured by the camera in response to the acquisition signal; The second acquisition submodule is used to acquire the real radar point cloud acquired by the radar in response to the acquisition signal.

[0068] In one specific implementation of this application embodiment, the data mapping relationship includes a translation matrix and a rotation matrix from the radar's coordinate system to the camera's coordinate system; The mapping module includes: The point cloud mapping submodule is used to map the real radar point cloud onto the real image according to the translation matrix and the rotation matrix to obtain the real multimodal data.

[0069] In one specific implementation of this application embodiment, the training module includes: The input submodule is used to input the real multimodal data and random mask into the generative adversarial network and obtain the output data of the generative adversarial network. The calculation submodule is used to calculate the training loss based on the real multimodal data and the output data; The adjustment submodule is used to adjust the parameters of the generative adversarial network according to the training loss until the preset convergence condition is met, so as to obtain the trained generative adversarial network.

[0070] In one specific implementation of this application embodiment, the data generation module includes: A preset specific noise signal is input into the generative adversarial network to generate multimodal data, and the virtual multimodal data output by the generative adversarial network is obtained.

[0071] In one specific implementation of this application embodiment, the data generation device further includes: The decomposition module is used to decompose the virtual multimodal data to obtain virtual images and virtual radar point clouds.

[0072] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0073] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0074] Figure 7 A schematic block diagram of a terminal device provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0075] like Figure 7 As shown, the terminal device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, it implements the steps in the above-described data generation method embodiments. Alternatively, when the processor 70 executes the computer program 72, it implements the functions of each module / unit in the above-described device embodiments.

[0076] For example, the computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 72 in the terminal device 7.

[0077] The terminal device 7 can be a computing device such as a mobile phone, tablet computer, desktop computer, laptop, handheld computer, or robot. Those skilled in the art will understand that... Figure 7This is merely an example of terminal device 7 and does not constitute a limitation on terminal device 7. It may include more or fewer components than shown, or combine certain components, or different components. For example, terminal device 7 may also include input / output devices, network access devices, buses, etc.

[0078] The processor 70 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0079] The memory 71 can be an internal storage unit of the terminal device 7, such as a hard disk or memory of the terminal device 7. The memory 71 can also be an external storage device of the terminal device 7, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device 7. Furthermore, the memory 71 can include both internal and external storage units of the terminal device 7. The memory 71 is used to store the computer program and other programs and data required by the terminal device 7. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0080] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0081] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0082] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0083] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0084] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0086] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0087] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data generation method, characterized in that, include: Acquire real images and real radar point clouds acquired simultaneously; The real radar point cloud is mapped onto the real image according to a preset data mapping relationship to obtain real multimodal data; wherein, the data mapping relationship includes a translation matrix and a rotation matrix from the radar coordinate system to the camera coordinate system; The real multimodal data and random mask are input into the generative adversarial network (GAN), and the output data of the GAN is obtained. The training loss is calculated based on the real multimodal data and the output data. The parameters of the GAN are adjusted based on the training loss until the preset convergence condition is met, and the trained GAN is obtained. A pre-set specific noise signal is input into a trained generative adversarial network to generate multimodal data, resulting in virtual multimodal data with spatial consistency. The virtual multimodal data is decomposed into virtual images and virtual radar point clouds.

2. The data generation method according to claim 1, characterized in that, The acquisition of synchronously acquired real images and real radar point clouds includes: A preset synchronization trigger sends a synchronized acquisition signal to the camera and the radar, so that the camera and the radar can perform synchronized data acquisition. Acquire the real image captured by the camera in response to the acquisition signal; Obtain the radar response to the real radar point cloud acquired by the acquired signal.

3. A data generation device, characterized in that, include: The synchronous acquisition module is used to acquire synchronously collected real images and real radar point clouds; The mapping module is used to map the real radar point cloud onto the real image according to a preset data mapping relationship to obtain real multimodal data; wherein, the data mapping relationship includes a translation matrix and a rotation matrix from the radar coordinate system to the camera coordinate system; The training module is used to input the real multimodal data and random mask into the generative adversarial network (GAN) and obtain the output data of the GAN; calculate the training loss based on the real multimodal data and the output data; and adjust the parameters of the GAN based on the training loss until a preset convergence condition is met, thereby obtaining the trained GAN. The data generation module is used to input a preset specific noise signal into a trained generative adversarial network to generate multimodal data, thereby obtaining virtual multimodal data with spatial consistency. The decomposition module is used to decompose the virtual multimodal data to obtain virtual images and virtual radar point clouds.

4. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data generation method as described in any one of claims 1 to 2.

5. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the data generation method as described in any one of claims 1 to 2.

Images