Data processing method and device, electronic equipment, storage medium and program product

Abstract

This application discloses a data processing method, apparatus, electronic device, storage medium, and program product. The data processing method includes: acquiring first multi-view synthesized data; and using a pre-trained harmonization model, performing multi-view consistency processing and long-term temporal consistency processing on the first multi-view synthesized data to obtain first multi-view harmonized data corresponding to the first multi-view synthesized data. This application introduces multi-view consistency processing and long-term temporal consistency processing through a harmonization model. Multi-view consistency processing strengthens the semantic correlation between foreground assets and background scenes under different perspectives, ensuring a high degree of uniformity of foreground assets across different perspectives. Long-term temporal consistency processing guarantees the smoothness of inter-frame motion and long-term dynamic stability of foreground assets. Therefore, the obtained multi-view harmonized data visually closely resembles the acquired data, significantly reducing inconsistency and enhancing the realism of the multi-view harmonized data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the deepening application of artificial intelligence technology in fields such as autonomous driving, virtual reality, and augmented reality, the demand for large-scale, high-quality 3D scene data is becoming increasingly urgent. Traditional real-vehicle data acquisition methods have significant limitations, such as high cost, low efficiency, and poor safety, when covering dangerous and long-tail scenarios.

[0003] Therefore, AIGC (Artificial Intelligence Generated Content) based synthetic data technology has emerged as an effective alternative. One feasible approach is to use external 3DGS (3D Gaussian Scattering) assets as foreground assets, rendered and composited into a real background scene based on the collected data, to generate synthetic data.

[0004] However, due to differences in lighting, color tone, shadows, and other factors between foreground assets and background scenes, there is a noticeable incongruity between the foreground assets and background scenes in the synthetic data, resulting in insufficient realism in the synthetic data. Summary of the Invention

[0005] The purpose of this application is to provide a data processing method, apparatus, electronic device, storage medium, and program product that can solve the problem of insufficient realism in synthetic data.

[0006] In a first aspect, embodiments of this application provide a data processing method, the method comprising: Acquire first-view composite data; Using a pre-trained harmonization model, multi-view consistency processing and long-term time-series consistency processing are performed on the first multi-view synthetic data to obtain the first multi-view harmonized data corresponding to the first multi-view synthetic data.

[0007] Optionally, the multi-view consistency processing includes: performing a multi-view cross-attention operation based on the first multi-view synthesized data.

[0008] Optionally, after obtaining the first multi-view synthetic data, the method further includes: obtaining the first multi-view ray pose representation corresponding to the first multi-view synthetic data; the multi-view consistency processing includes: performing a multi-view cross-attention operation based on the first multi-view synthetic data and the first multi-view ray pose representation.

[0009] Optionally, the long-term consistency processing includes: performing dynamic correction based on the first multi-view synthesized data.

[0010] Optionally, after acquiring the first multi-view composite data, the method further includes: acquiring the first multi-view background image and the first multi-view foreground rendering image corresponding to the first multi-view composite data; the long-term consistency processing includes: performing dynamic correction based on the first multi-view composite data, the first multi-view background image and the first multi-view foreground rendering image.

[0011] Optionally, the harmonization model is trained in the following manner: Acquire multi-view dissonance data and the second multi-view harmony data corresponding to the multi-view dissonance data; The multi-view inconsistency data is used as the input of the first initial model. In the first initial model, multi-view consistency processing and long-term consistency processing are performed based on the multi-view inconsistency data. The second multi-view harmonization data is used as the training target of the first initial model. The first initial model is trained and the trained first initial model is used as the harmonization model.

[0012] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising: The first acquisition module is used to acquire the first multi-view composite data; The processing module is used to perform multi-view consistency processing and long-term consistency processing on the first multi-view synthetic data using a pre-trained harmonization model, so as to obtain the first multi-view harmonized data corresponding to the first multi-view synthetic data.

[0013] Optionally, the processing module includes: a first processing unit, configured to perform multi-view cross-attention operation based on the first multi-view synthetic data.

[0014] Optionally, the first acquisition module is further configured to acquire the first multi-view ray pose representation corresponding to the first multi-view synthetic data; the processing module includes: a second processing unit, configured to perform multi-view cross-attention operation based on the first multi-view synthetic data and the first multi-view ray pose representation.

[0015] Optionally, the processing module includes a third processing unit for performing dynamic correction based on the first multi-view synthesized data.

[0016] Optionally, the first acquisition module is further configured to acquire a first multi-view background image and a first multi-view foreground rendering image corresponding to the first multi-view composite data; the processing module includes: a fourth processing unit, configured to perform dynamic correction based on the first multi-view composite data, the first multi-view background image and the first multi-view foreground rendering image.

[0017] Optionally, the harmonization model is trained using the following modules: The second acquisition module is used to acquire multi-view disharmony data and the second multi-view harmony data corresponding to the multi-view disharmony data. The first training module is used to take the multi-view inconsistency data as input to the first initial model, perform multi-view consistency processing and long-term consistency processing on the multi-view inconsistency data in the first initial model, use the second multi-view harmonization data as the training target of the first initial model, train the first initial model, and use the trained first initial model as the harmonization model.

[0018] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the data processing method as described in any of the preceding claims.

[0019] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the data processing method as described in any of the preceding claims.

[0020] Fifthly, embodiments of this application provide a computer program product, characterized in that the computer program product stores a program or instructions, which, when executed by a processor, implement the data processing method as described in any of the preceding claims.

[0021] In this embodiment, a harmonization model is introduced to introduce multi-view consistency processing and long-term consistency processing. Multi-view consistency processing strengthens the semantic correlation between foreground assets and background scenes under different views, ensuring a high degree of uniformity of foreground assets under each view. Long-term consistency processing ensures the smoothness of inter-frame motion and long-term dynamic stability of foreground assets. Therefore, the obtained multi-view harmonized data is visually very close to the collected data, significantly reducing the sense of incongruity and improving the realism of multi-view harmonized data.

[0022] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some drawings of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a data processing method according to an embodiment of this application; Figure 2 This is a flowchart of a model training method according to an embodiment of this application; Figure 3 This is a schematic diagram of a harmonization model according to an embodiment of this application; Figure 4 This is a flowchart of another model training method according to an embodiment of this application; Figure 5 This is a structural block diagram of a data processing apparatus according to an embodiment of this application; Figure 6 This is a structural block diagram of a model training device according to an embodiment of this application; Figure 7 This is a structural block diagram of another model training device according to an embodiment of this application; Figure 8 This is a structural block diagram of an electronic device according to an embodiment of this application; Figure 9 This is a structural block diagram of a readable storage medium according to an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0027] In this embodiment of the application, to address the problem of insufficient realism in synthetic data, a harmonization technique is proposed to process the synthetic data.

[0028] The following methods were considered in implementing the harmonization technology: Image level: A statistical matching-based approach: achieving harmony by matching the color statistics of foreground assets and background scenes; Early convolutional networks: directly generated harmonized images using an encoder-decoder architecture; Generative Adversarial Networks (GANs): The generator reconstructs the foreground region, the discriminator evaluates the harmony, and the system is optimized through adversarial learning. Diffusion model: A harmonized image is gradually generated through an iterative denoising process.

[0029] Video aspect: CO2Net (Color Consistency Optimization Network): It utilizes color mapping consistency, integrates color transformation parameters of adjacent frames through LUT (Look-Up Table), and then optimizes the result of the current frame through a lightweight fine-tuning module; Optical flow estimation + spatiotemporal correction: Combine optical flow method to estimate inter-frame motion, and enhance spatiotemporal consistency through motion compensation distortion and edge feathering.

[0030] However, the above method has the following problems: Regarding images: timing consistency cannot be guaranteed.

[0031] Statistical matching-based methods rely on manually designed features, making them difficult to handle complex scenarios. Early convolutional networks relied on paired data within the domain, which was very difficult to obtain; Generative Adversarial Networks (GANs) are prone to training instability and pattern collapse. Diffusion model: High inference cost, strong dependence on paired data within the domain.

[0032] Regarding the video: consistency across multiple perspectives cannot be guaranteed.

[0033] CO2Net: High inference cost, heavily reliant on domain-specific paired data; Optical flow estimation + spatiotemporal correction: relies on short-term motion estimation, which is difficult to handle occlusion or complex deformation.

[0034] To address the aforementioned issues, this application proposes a joint mechanism of multi-view consistency processing and long-term consistency processing in its embodiments, thereby improving data authenticity and simplifying the processing procedure. Details are described below.

[0035] Reference Figure 1 The diagram shows a flowchart of a data processing method according to an embodiment of this application.

[0036] like Figure 1 As shown, the data processing method may include the following steps: Step 101: Obtain the first multi-view composite data.

[0037] In this embodiment, data collected from different perspectives within the same scene is used as first multi-view acquisition data. Based on this first multi-view acquisition data, for each perspective, 3DGS assets (such as vehicles and pedestrians constructed using 3DGS technology) are rendered as foreground assets and composited into the real background scene of that perspective, thus obtaining first multi-view composite data. However, this first multi-view composite data suffers from a mismatch between the foreground assets and the background scene. Specifically, the first multi-view acquisition data includes the acquisition data for each perspective, and correspondingly, the first multi-view composite data includes the composite data for each perspective.

[0038] For example, the data can be an image sequence, video, etc. Accordingly, the first multi-view acquisition data can be a first multi-view acquired image sequence, a first multi-view acquired video, etc., and the first multi-view synthesized data can be a first multi-view synthesized image sequence, a first multi-view synthesized video, etc.

[0039] Step 102: Using a pre-trained harmonization model, perform multi-view consistency processing and long-term consistency processing on the first multi-view synthetic data to obtain the first multi-view harmonized data corresponding to the first multi-view synthetic data.

[0040] In this embodiment of the application, the first multi-view synthesized data is input into a pre-trained harmonization model. In the harmonization model, multi-view consistency processing and long-term consistency processing are performed on the first multi-view synthesized data to obtain the first multi-view harmonized data corresponding to the first multi-view synthesized data. The harmonization model outputs the first multi-view harmonized data.

[0041] In one optional implementation, the multi-view consistency processing may include performing a multi-view cross-attention operation based on the first multi-view synthesized data. This approach introduces a multi-view cross-attention operation, which can strengthen the semantic correlation between multiple views.

[0042] For example, the harmonization model may include a cross-view attention module, which performs multi-view cross-attention operations based on the first multi-view synthetic data. For instance, foreground and background features (including semantic features and visual features such as color and texture) of the synthetic data from different viewpoints are extracted, the similarity between features from different viewpoints is calculated to obtain attention weights, and the features from different viewpoints are weighted and fused according to the attention weights, so that the features from different viewpoints complement each other, thereby strengthening the semantic correlation between multiple viewpoints and ensuring that the same foreground asset exhibits consistent performance in terms of form, lighting, etc., under different viewpoints.

[0043] In one optional implementation, after acquiring the first multi-view synthetic data, the method may further include: acquiring the first multi-view ray pose representation corresponding to the first multi-view synthetic data. Accordingly, the multi-view consistency processing may include: performing a multi-view cross-attention operation based on the first multi-view synthetic data and the first multi-view ray pose representation. This approach replaces the traditional pose matrix with ray-map-based pose representation (RPR), explicitly modeling relative pose relationships through ray space mapping, thereby improving the robustness of viewpoint transformations.

[0044] For example, the input trajectory containing camera parameters corresponding to the first multi-view synthesized data can be obtained, and the input trajectory containing camera parameters can be encoded using a normalized Plück encoder to obtain the first multi-view ray pose representation corresponding to the first multi-view synthesized data. The first multi-view ray pose representation uses Plück coordinates to represent rays, containing information needed to describe the position and direction of a ray in three-dimensional space. For a multi-view synthesized data set, its multi-view ray pose representation contains multiple sets of data, each set corresponding to a set of rays (including the ray origin and ray direction) of pixels from one viewpoint.

[0045] For example, the harmonization model includes a cross-view attention module. This module performs multi-view cross-attention operations based on the first multi-view synthetic data and the first multi-view ray pose representation. For instance, it extracts foreground and background features (including semantic features and visual features such as color and texture) from the synthetic data at different viewpoints. During the process of calculating the similarity between features from different viewpoints to obtain attention weights, it calculates the spatial geometric bias based on the geometric relationship between rays from different viewpoints in space and adds it to the similarity. This ensures that the attention weights are based not only on semantic similarity but also on geometric consistency in space. Then, it weights and fuses the features from different viewpoints based on the attention weights, allowing the features from different viewpoints to complement each other, thereby strengthening the semantic correlation between multiple viewpoints and ensuring that the same foreground asset exhibits consistent form, lighting, and other aspects across different viewpoints.

[0046] In one optional implementation, the long-term temporal consistency processing may include: performing dynamic correction based on the first multi-view synthesized data. This approach can perform dynamic correction based on a dynamic correction learning strategy, thereby achieving inter-frame motion smoothness and long-term dynamic stability.

[0047] For example, the harmonization model may include a corrected stream video generation module, which performs dynamic correction based on the first multi-view synthesized data. For instance, based on the first multi-view synthesized data, a dynamic correction learning strategy is used to calculate predicted motion features, and these features are then used to correct information in adjacent frames to constrain the generation of the current frame. The dynamic correction learning strategy of the corrected stream video generation module learns the dynamic velocity field of the frame sequence and optimizes the generation trajectory to solve problems such as error accumulation and image drift in long video generation, ensuring motion coherence and visual consistency. This can be achieved through various methods such as scrolling windows and neural differential equations. Specific processing steps can be implemented based on practical experience, and will not be discussed in detail in this embodiment.

[0048] In one optional implementation, after acquiring the first multi-view composite data, the process may further include: acquiring a first multi-view background image and a first multi-view foreground rendering image corresponding to the first multi-view composite data. Accordingly, the long-term consistency processing may include: performing dynamic correction based on the first multi-view composite data, the first multi-view background image, and the first multi-view foreground rendering image.

[0049] For example, a masking method can be used to obtain the first multi-view background image and the first multi-view foreground rendering image corresponding to the first multi-view composite data. Specifically, for each viewpoint, the composite data for that viewpoint corresponds to both a background image and a foreground rendering image. For instance, the foreground assets of the composite data can be masked to obtain the background image, and the background scene of the composite data can be masked to obtain the foreground rendering image, and so on.

[0050] For example, based on the first multi-view synthetic data, the first multi-view background image, and the first multi-view foreground rendering image, dynamic correction is performed under the dual guidance of the first multi-view background image and the first multi-view foreground rendering image, thereby constraining the structure and spatiotemporal consistency of the generated content.

[0051] In this embodiment, a harmonization model is introduced to introduce multi-view consistency processing and long-term consistency processing. Multi-view consistency processing strengthens the semantic correlation between foreground assets and background scenes under different views, ensuring a high degree of uniformity of foreground assets under each view. Long-term consistency processing ensures the smoothness of inter-frame motion and long-term dynamic stability of foreground assets. Therefore, the obtained multi-view harmonized data is visually very close to the collected data, significantly reducing the sense of incongruity and improving the realism of multi-view harmonized data.

[0052] Reference Figure 2 The diagram shows a flowchart of a model training method according to an embodiment of this application.

[0053] like Figure 2 As shown, the model training method may include the following steps: Step 201: Obtain multi-view dissonance data and the second multi-view harmony data corresponding to the multi-view dissonance data.

[0054] Among them, the multi-view dissonance data and the second multi-view harmony data are paired, that is, one multi-view dissonance data corresponds to one second multi-view harmony data.

[0055] In one optional implementation, the process of acquiring multi-view inconsistency data and the corresponding second multi-view harmonization data may include: acquiring target multi-view acquisition data, applying perturbation to the foreground of the target multi-view acquisition data to obtain multi-view perturbation data, using the multi-view perturbation data as the multi-view inconsistency data, and using the target multi-view acquisition data as the corresponding second multi-view harmonization data; and / or, acquiring second multi-view composite data, adjusting the parameters of the second multi-view composite data to obtain multi-view adjustment data, using the second multi-view composite data as the multi-view inconsistency data, and using the multi-view adjustment data as the corresponding second multi-view harmonization data.

[0056] In this embodiment, lightweight pairing data is constructed as training data for the harmonization model.

[0057] For example, data collected from different perspectives within the same scene can be used as target multi-view acquisition data. Perturbation is applied to the foreground of the target multi-view acquisition data to obtain multi-view perturbation data. For instance, random lighting perturbation, texture perturbation, etc., can be applied to the foreground of the target multi-view acquisition data, or a large visual generation model can be used to locally redraw the foreground of the target multi-view acquisition data to obtain multi-view perturbation data. This multi-view perturbation data is used as the multi-view inconsistency data, and the target multi-view acquisition data is used as the second multi-view harmonization data corresponding to the multi-view inconsistency data.

[0058] For example, data collected from different perspectives within the same scene can be used as second multi-view acquisition data. Based on this second multi-view acquisition data, for each perspective, 3DGS assets are rendered as foreground assets and composited into the real background scene of that perspective, thus obtaining second multi-view composite data. This second multi-view composite data can be selected high-quality composite data (e.g., meeting conditions such as clear foreground outlines, complete background scene, and no obvious rendering defects). Parameter adjustments are made to the second multi-view composite data to obtain multi-view adjustment data. For example, the foreground-background blending parameters of the second multi-view composite data (such as transparency, shadow intensity, and lighting matching degree) can be finely adjusted to obtain the multi-view adjustment data. The second multi-view composite data is used as the multi-view inconsistency data, and the multi-view adjustment data is used as the second multi-view harmonization data corresponding to the multi-view inconsistency data.

[0059] For example, the pairing of the multi-view disturbance data and the target multi-view acquisition data, as well as the pairing of the second multi-view synthetic data and the multi-view adjustment data, can be selected according to a set ratio, and together they can serve as multi-view inconsistency data and the second multi-view harmonization data corresponding to the multi-view inconsistency data.

[0060] Step 202: Using the multi-view inconsistency data as input to the first initial model, perform multi-view consistency processing and long-term consistency processing on the multi-view inconsistency data in the first initial model, use the second multi-view harmonization data as the training target of the first initial model, train the first initial model, and use the trained first initial model as the harmonization model.

[0061] In this embodiment of the application, the first initial model refers to the harmonized model to be trained. Each weight parameter in the first initial model is an initial value, and each weight parameter is iteratively adjusted during the model training process.

[0062] For each pair of multi-view discrepancies data and second multi-view harmonies data, the multi-view discrepancies data is used as the input of the first initial model. In the first initial model, multi-view consistency processing and long-term consistency processing are performed based on the multi-view discrepancies data. The second multi-view harmonies data is used as the training target of the first initial model. The first initial model is trained and the trained first initial model is used as the harmonization model.

[0063] The specific processes for multi-view consistency processing and long-term consistency processing can be referred to the relevant descriptions above, and will not be discussed in detail here. The specific training process can be handled based on practical experience. For example, during training, the weight parameters can be iteratively adjusted using a first loss function and gradient descent strategy. Training is considered complete when a first condition is met. The first condition could be that the first loss function is less than a set threshold or the number of iterations reaches a preset number. This will not be discussed in detail here.

[0064] In one alternative implementation, the harmonization model may be based on a diffusion model architecture.

[0065] Accordingly, the data processing method may include: acquiring first multi-view synthesized data and first noise data; using a pre-trained harmonization model, performing multi-view consistency processing and long-term time-series consistency processing on the first multi-view synthesized data and the first noise data to obtain first multi-view harmonized data corresponding to the first multi-view synthesized data.

[0066] Accordingly, the multi-view consistency processing includes performing a multi-view cross-attention operation based on the first multi-view synthesized data and the first noise data.

[0067] Accordingly, after acquiring the first multi-view synthesized data and the first noise data, the method further includes: acquiring the first multi-view ray pose representation corresponding to the first multi-view synthesized data. The multi-view consistency processing includes: performing a multi-view cross-attention operation based on the first multi-view synthesized data, the first noise data, and the first multi-view ray pose representation.

[0068] Accordingly, the long-term consistency processing includes: performing dynamic correction based on the first multi-view synthesized data and the first noise data.

[0069] Accordingly, after acquiring the first multi-view synthesized data and the first noise data, the method further includes: acquiring the first multi-view background image and the first multi-view foreground rendering image corresponding to the first multi-view synthesized data. The long-term consistency processing includes: performing dynamic correction based on the first multi-view synthesized data, the first noise data, the first multi-view background image, and the first multi-view foreground rendering image.

[0070] Accordingly, the model training method may include: acquiring multi-view inconsistency data and second multi-view harmonization data corresponding to the multi-view inconsistency data, and adding second noise data to the second multi-view harmonization data to obtain sample multi-view noise data; using the multi-view inconsistency data and the sample multi-view noise data as input to a first initial model, performing multi-view consistency processing and long-term consistency processing in the first initial model based on the multi-view inconsistency data and the sample multi-view noise data, using the second noise data as the target of the output of the first initial model, training the first initial model, and using the trained first initial model as the harmonization model.

[0071] For example, during training, the multi-view mismatch data and the sample multi-view noise data are used as inputs to the first initial model. In the first initial model, multi-view consistency processing and long-term consistency processing are performed based on the multi-view mismatch data and the sample multi-view noise data to obtain predicted noise data. A first loss function is calculated based on the predicted noise data and the second noise data. When the first condition is not met, the weight parameters of the first initial model are adjusted and training continues. When the first condition is met, training is determined to be complete. The first condition can be that the first loss function is less than a set threshold or the number of iterations reaches a preset number.

[0072] Reference Figure 3 The diagram shows a schematic of a harmonization model according to an embodiment of this application.

[0073] like Figure 3 As shown, when training the harmonicization model, multi-view acquisition data of the target is obtained. Random lighting / texture perturbation or local redrawing is applied to the multi-view acquisition data to obtain multi-view perturbation data. Paired data is constructed using the target multi-view acquisition data and the multi-view perturbation data as training data, and the harmonicization model is trained using the training data. Alternatively, the input trajectory with camera parameters corresponding to the target multi-view acquisition data can be obtained. A normalized Plucker encoder is used to encode the input trajectory with camera parameters to obtain the second multi-view ray pose representation (i.e., camera encoding) corresponding to the target multi-view acquisition data. This second multi-view ray pose representation is then input into the harmonicization model. Second noise data, specifically a multi-level noise latent representation, can also be input into the harmonicization model. Furthermore, the second multi-view background image and the second multi-view foreground rendering image corresponding to the multi-view perturbation data can also be input into the harmonicization model.

[0074] When processing data using the harmonization model, the first multi-view synthetic data and the first noise data (specifically, a multi-scale noise latent representation) are obtained. The input trajectory containing camera parameters corresponding to the first multi-view synthetic data is obtained. The input trajectory containing camera parameters is encoded using a normalized Plücker encoder to obtain the first multi-view ray pose representation corresponding to the first multi-view synthetic data. The first multi-view background image and the first multi-view foreground rendering image corresponding to the first multi-view synthetic data are obtained. The first multi-view synthetic data, the first noise data, the first multi-view ray pose representation, the first multi-view background image, and the first multi-view foreground rendering image are input into the harmonization model. After performing multi-view consistency processing and long-term temporal consistency processing within the harmonization model, the first multi-view harmonized data corresponding to the first multi-view synthetic data is obtained. The harmonization model outputs the first multi-view harmonized data.

[0075] The harmonization model can include an encoding module, a panoramic diffusion transformer (PDiT), and a decoding module. The panoramic diffusion transformer can include multiple sets of 3D full attention modules, point-wise feed forward networks, and cross-view attention modules. It can also include a corrected streaming video generation module. Figure 3(Not shown in the image) The harmonization model uses these modules to perform multi-perspective consistency processing and long-term time-series consistency processing on the input data.

[0076] Reference Figure 4 The flowchart illustrates another model training method according to an embodiment of this application.

[0077] like Figure 4 As shown, the model training method may include the following steps: Step 401: Obtain the first multi-view composite data.

[0078] Step 402: Using a pre-trained harmonization model, perform multi-view consistency processing and long-term consistency processing on the first multi-view synthetic data to obtain the first multi-view harmonized data corresponding to the first multi-view synthetic data.

[0079] Steps 401 and 402 are the same as steps 101 and 102 above. For the specific process, please refer to the relevant descriptions of steps 101 and 102 above. This embodiment will not be discussed in detail.

[0080] Step 403: Use the first multi-view harmonized data to train the second initial model, and use the trained second initial model as the intelligent driving model.

[0081] In this embodiment of the application, the second initial model refers to the intelligent driving model to be trained. Each weight parameter in the second initial model is an initial value, and each weight parameter is iteratively adjusted during the model training process.

[0082] The specific process of training the second initial model using the first multi-view harmonized data can be handled according to different business needs and practical experience.

[0083] For example, the first multi-view harmonized data is labeled to obtain corresponding label information, such as actual bounding box information and actual lane line information from the first multi-view harmonized data. The first multi-view harmonized data is used as input to the second initial model. After internal processing by the second initial model, the predicted information corresponding to the first multi-view harmonized data is output by the second initial model. For example, the predicted information can be predicted bounding box information and predicted lane line information from the first multi-view harmonized data. A second loss function is calculated based on the label information and the predicted information. If a second condition is not met, the weight parameters of the second initial model are adjusted and training continues. When the second condition is met, training is considered complete, and the trained second initial model is used as the intelligent driving model. The second condition can be that the second loss function is less than a set threshold or the number of iterations reaches a preset number.

[0084] In this embodiment, since the first multi-view harmonized data is data with a stronger sense of realism obtained through the harmonization model, training the intelligent driving model using the first multi-view harmonized data can effectively improve the generalization ability and overall performance of the intelligent driving model.

[0085] Reference Figure 5 The diagram shows a structural block diagram of a data processing apparatus according to an embodiment of this application.

[0086] like Figure 5 As shown, the data processing device may include the following modules: The first acquisition module 501 is used to acquire the first multi-view composite data; The processing module 502 is used to perform multi-view consistency processing and long-term consistency processing on the first multi-view synthetic data using a pre-trained harmonization model, so as to obtain the first multi-view harmonized data corresponding to the first multi-view synthetic data.

[0087] Optionally, the processing module 502 includes: a first processing unit, configured to perform multi-view cross-attention operation based on the first multi-view synthetic data.

[0088] Optionally, the first acquisition module 501 is further configured to acquire the first multi-view ray pose representation corresponding to the first multi-view synthetic data; the processing module 502 includes: a second processing unit, configured to perform multi-view cross-attention operation based on the first multi-view synthetic data and the first multi-view ray pose representation.

[0089] Optionally, the processing module 502 includes a third processing unit for performing dynamic correction based on the first multi-view synthetic data.

[0090] Optionally, the first acquisition module 501 is further configured to acquire the first multi-view background image and the first multi-view foreground rendering image corresponding to the first multi-view composite data; the processing module 502 includes: a fourth processing unit, configured to perform dynamic correction based on the first multi-view composite data, the first multi-view background image and the first multi-view foreground rendering image.

[0091] Reference Figure 6 The diagram shows a structural block diagram of a model training device according to an embodiment of this application.

[0092] like Figure 6 As shown, the model training device may include the following modules: The second acquisition module 601 is used to acquire multi-view incongruity data and the second multi-view harmony data corresponding to the multi-view incongruity data. The first training module 602 is used to take the multi-view inconsistency data as input to the first initial model, perform multi-view consistency processing and long-term consistency processing on the multi-view inconsistency data in the first initial model, use the second multi-view harmonization data as the training target of the first initial model, train the first initial model, and use the trained first initial model as the harmonization model.

[0093] Optionally, the second acquisition module 602 includes: The first data acquisition unit is used to acquire target multi-view acquisition data, apply perturbation to the foreground of the target multi-view acquisition data to obtain multi-view perturbation data, use the multi-view perturbation data as the multi-view incongruity data, and use the target multi-view acquisition data as the second multi-view harmonious data corresponding to the multi-view incongruity data. And / or, The second data acquisition unit is used to acquire second multi-view composite data, adjust the parameters of the second multi-view composite data to obtain multi-view adjustment data, use the second multi-view composite data as the multi-view disharmony data, and use the multi-view adjustment data as the second multi-view harmony data corresponding to the multi-view disharmony data.

[0094] Reference Figure 7 The diagram shows a structural block diagram of another model training device according to an embodiment of this application.

[0095] like Figure 7 As shown, the model training device may include the following modules: The third acquisition module 701 is used to acquire the first multi-view synthetic data; using a pre-trained harmonization model, multi-view consistency processing and long-term consistency processing are performed on the first multi-view synthetic data to obtain the first multi-view harmonized data corresponding to the first multi-view synthetic data. The second training module 702 is used to train the second initial model using the first multi-view harmonized data, and to use the trained second initial model as the intelligent driving model.

[0096] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0097] The data processing device and model training device in the embodiments of this application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. The embodiments of this application do not impose specific limitations.

[0098] The data processing device and model training device in the embodiments of this application can be devices with an operating system. The operating system can be Android, iOS, or other possible operating systems, and this application does not specifically limit them.

[0099] Optionally, embodiments of this application also provide an electronic device.

[0100] Reference Figure 8 The diagram shows a structural block diagram of an electronic device according to an embodiment of this application.

[0101] like Figure 8 As shown, the electronic device 11 includes a processor 111, a memory 112, and a program or instruction 1121 stored in the memory and executable on the processor. When the program or instruction 1121 is executed by the processor 111, it implements the various processes of any of the above-described data processing method embodiments or model training method embodiments, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0102] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0103] Optionally, embodiments of this application also provide a readable storage medium.

[0104] Reference Figure 9 The diagram shows a structural block diagram of a readable storage medium according to an embodiment of this application.

[0105] like Figure 9As shown, the readable storage medium 21 stores a program or instruction 211. When the program or instruction 211 is executed by the processor, it implements the various processes of any of the above data processing method embodiments or any model training method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0106] The processor may be the processor in the electronic device described in the above embodiments.

[0107] Optionally, this application embodiment also provides a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement each process of any of the above data processing method embodiments or any model training method embodiments, and to achieve the same technical effect, so to avoid repetition, it will not be described again here.

[0108] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0109] The processors mentioned above may include, but are not limited to: Central Processing Unit (CPU), Network Processor (NP), Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0110] The aforementioned readable storage media include computer-readable storage media, which may include, but are not limited to: read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), electronically erasable programmable read-only memory (EEPROM), hard disk, floppy disk, flash memory, etc.

[0111] The various embodiments in this specification are related to each other and are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

[0112] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and regulations of the locality and with authorization from the owner of the relevant device.

[0113] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0115] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms. The functional units in the various embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0116] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0117] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Acquire first-view composite data; Using a pre-trained harmonization model, multi-view consistency processing and long-term time-series consistency processing are performed on the first multi-view synthetic data to obtain the first multi-view harmonized data corresponding to the first multi-view synthetic data.

2. The method according to claim 1, characterized in that, The multi-view consistency processing includes: Perform multi-view cross-attention operation based on the first multi-view synthetic data.

3. The method according to claim 1, characterized in that, After acquiring the first multi-view synthetic data, the method further includes: acquiring the first multi-view ray pose representation corresponding to the first multi-view synthetic data; The multi-view consistency processing includes performing a multi-view cross-attention operation based on the first multi-view synthetic data and the first multi-view ray pose representation.

4. The method according to claim 1, characterized in that, The long-time consistency processing includes: Dynamic correction is performed based on the first multi-view synthetic data.

5. The method according to claim 1, characterized in that, After acquiring the first multi-view composite data, the method further includes: acquiring the first multi-view background image and the first multi-view foreground rendering image corresponding to the first multi-view composite data; The long-term consistency processing includes: performing dynamic correction based on the first multi-view synthetic data, the first multi-view background image, and the first multi-view foreground rendering image.

6. The method according to claim 1, characterized in that, The harmonization model is trained in the following manner: Acquire multi-view dissonance data and the second multi-view harmony data corresponding to the multi-view dissonance data; The multi-view inconsistency data is used as the input of the first initial model. In the first initial model, multi-view consistency processing and long-term consistency processing are performed based on the multi-view inconsistency data. The second multi-view harmonization data is used as the training target of the first initial model. The first initial model is trained and the trained first initial model is used as the harmonization model.

7. A data processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire the first multi-view composite data; The processing module is used to perform multi-view consistency processing and long-term consistency processing on the first multi-view synthetic data using a pre-trained harmonization model, so as to obtain the first multi-view harmonized data corresponding to the first multi-view synthetic data.

8. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the data processing method as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the data processing method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product stores a program or instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1 to 6.

Images