Data processing method and apparatus

Abstract

This application discloses a data processing method and apparatus. The method includes: a first electronic device acquiring N multi-angle views captured by a second electronic device; inputting the N multi-angle views into a feedforward reconstruction network for 3D reconstruction processing to obtain original teacher scene data and predicted camera poses for each of the N multi-angle views; the original teacher scene data including attribute parameters of multiple Gaussian points; based on the predicted camera poses and the calibration camera poses corresponding to the second electronic device, performing coordinate system alignment processing on the original teacher scene data, and performing parameter format conversion processing on the aligned teacher scene data to obtain intermediate teacher scene data matching the coordinate system of the second electronic device; generating first and second student scene data based on the intermediate teacher scene data; the first student scene data being used for previewing on the second electronic device; and the second student scene data being used to replace the first student scene data after generation.

Description

Technical Field

[0001] This application belongs to the field of three-dimensional reconstruction technology, specifically relating to a data processing method and apparatus. Background Technology

[0002] Real-time 3D reconstruction and rendering of realistic scenes on devices such as mobile phones and tablets is a problem that fields such as virtual reality, e-commerce display, and digital cultural tourism are trying to solve. For example, a user can take a picture of an object by walking around it with their mobile phone and see a freely rotating and scaled 3D model on their phone. Currently, the mainstream 3D reconstruction technology is 3D Gaussian scattering, which represents a scene as a large number of colored ellipsoids with attributes (called Gaussian points), and generates an image by superimposing these ellipsoids. A scene usually contains hundreds of thousands to millions of ellipsoids, resulting in a large amount of data that mobile phones cannot directly handle. Therefore, the industry usually adopts a "teacher-student" compression framework: first, a high-quality, large-scale scene data (called teacher scene data) is built, and then a smaller-scale scene data (called student scene data) is generated based on this data and deployed on the mobile phone.

[0003] In related technologies, the construction of teacher scene data mainly adopts a scene-by-scene iterative optimization approach. Starting from sparse point clouds, dense scene data is generated after tens of thousands of repeated adjustments, and then student scene data is generated. The whole process takes a long time, and users have to wait a long time to see the 3D scene after uploading photos. Summary of the Invention

[0004] The purpose of this application is to provide a data processing method and apparatus that can solve the problem that users need to wait a long time to see 3D scene images.

[0005] In a first aspect, embodiments of this application provide a data processing method applied to a first electronic device, the method comprising: Obtain N multi-angle views captured by the second electronic device; where N is an integer greater than 1; The N multi-angle views are input into a feedforward reconstruction network for 3D reconstruction processing to obtain the original teacher scene data and the predicted camera poses of the N multi-angle views respectively; wherein, the original teacher scene data includes attribute parameters of multiple Gaussian points; Based on the predicted camera pose and the calibrated camera pose corresponding to the second electronic device, the original teacher scene data is subjected to coordinate system alignment processing, and the aligned teacher scene data is subjected to parameter format conversion processing to obtain intermediate teacher scene data that matches the coordinate system of the second electronic device. Based on the intermediate teacher scenario data, first student scenario data and second student scenario data are generated; The first student scene data is generated at a faster rate than the second student scene data, and the second student scene data has a better image quality than the first student scene data. The first student scene data is used for previewing on the second electronic device. The second student scene data is used to replace the first student scene data after it has been generated.

[0006] Secondly, embodiments of this application provide a data processing method applied to a second electronic device, the method comprising: Receive the first student scene data transmitted by the first electronic device; Based on the first student scene data, display the preview screen corresponding to the first student scene data; Receive the second student scenario data transmitted by the first electronic device; Based on the second student scene data, replace the first student scene data and display the screen corresponding to the second student scene data; The first student scene data and the second student scene data are both scene description data describing a 3D scene. The generation speed of the first student scene data is greater than that of the second student scene data, and the image quality of the second student scene data is better than that of the first student scene data.

[0007] Thirdly, embodiments of this application provide a data processing apparatus applied to a first electronic device, the apparatus comprising: The acquisition module is used to acquire N multi-angle views captured by the second electronic device; where N is an integer greater than 1. The reconstruction module is used to input the N multi-angle views into the feedforward reconstruction network for 3D reconstruction processing to obtain the original teacher scene data and the predicted camera poses of the N multi-angle views respectively; wherein, the original teacher scene data includes attribute parameters of multiple Gaussian points; The conversion module is used to perform coordinate system alignment processing on the original teacher scene data based on the predicted camera pose and the calibration camera pose corresponding to the second electronic device, and to perform parameter format conversion processing on the aligned teacher scene data to obtain intermediate teacher scene data that matches the coordinate system of the second electronic device. The generation module is used to generate first student scenario data and second student scenario data based on the intermediate teacher scenario data; The first student scene data is generated at a faster rate than the second student scene data, and the second student scene data has a better image quality than the first student scene data. The first student scene data is used for previewing on the second electronic device. The second student scene data is used to replace the first student scene data after it has been generated.

[0008] Fourthly, embodiments of this application provide a data processing apparatus applied to a second electronic device, the apparatus comprising: The receiving module is used to receive the first student scene data transmitted by the first electronic device; The display module is used to display a preview screen corresponding to the first student scene data based on the first student scene data; The receiving module is also used to receive second student scenario data transmitted by the first electronic device; The display module is further configured to replace the first student scene data based on the second student scene data, and display the screen corresponding to the second student scene data; The first student scene data and the second student scene data are both scene description data describing a 3D scene. The generation speed of the first student scene data is greater than that of the second student scene data, and the image quality of the second student scene data is better than that of the first student scene data.

[0009] Fifthly, embodiments of this application provide an electronic device, which includes a processor and a memory. The memory stores a program or instructions that can run on the processor. When the program or instructions are executed by the processor, they implement the steps of the data processing method as described in the first aspect, or implement the steps of the data processing method as described in the second aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the data processing method as described in the first aspect, and the steps of the data processing method as described in the second aspect.

[0011] In a seventh aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the data processing method as described in the first aspect, or to implement the data processing method as described in the second aspect.

[0012] Eighthly, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the data processing method as described in the first aspect, or to implement the data processing method as described in the second aspect.

[0013] In this embodiment, the first electronic device generates teacher scene data through a feedforward reconstruction network, replacing the iterative optimization method of scene-by-scene in related technologies. This shortens the construction time of teacher scene data and improves the construction efficiency. Furthermore, through coordinate alignment and parameter format conversion, the output of the feedforward reconstruction network can smoothly enter subsequent processes, truly translating the speed advantage into usability. Further, by generating first and second student scene data, the first student scene data is generated quickly and can be rapidly transmitted to the second electronic device for user preview, avoiding long waiting times and solving the problem of users having to wait a long time to see the 3D scene. The second student scene data automatically replaces the first student scene data after generation, ensuring both preview speed and final display quality, thus improving the user experience. Attached Figure Description

[0014] Figure 1 This is one of the flowcharts of a data processing method provided in some embodiments of this application; Figure 2 This is a flowchart of one implementation of step 104 provided in some embodiments of this application; Figure 3 This is a second flowchart of a data processing method provided in some embodiments of this application; Figure 4 This is an architecture diagram of a data processing system provided by some embodiments of this application; Figure 5 This is one of the structural block diagrams of a data processing apparatus provided in some embodiments of this application; Figure 6 This is a second structural block diagram of a data processing apparatus provided in some embodiments of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in some embodiments of this application; Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0016] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0017] To facilitate understanding, some related concepts and application scenarios involved in the embodiments of this application will be explained below.

[0018] I. Related Concepts 3D Gaussian scattering is a point cloud-based technique for representing and rendering 3D scenes. This technique represents a 3D scene as a large number of colored ellipsoids with attributes, called Gaussian points. Each Gaussian point contains attribute parameters such as position, color (spherical harmonic coefficient), size (scale), orientation (rotation), and transparency (opacity). During rendering, these Gaussian points are projected from 3D space onto a 2D image plane using a differentiable rasterizer, and the contributions of each Gaussian point to each pixel are accumulated according to their weights to generate the final image.

[0019] Feedforward reconstruction networks are pre-trained 3D reconstruction networks that perform forward inference on multiple input views from various angles, directly outputting a dense 3D Gaussian point cloud corresponding to the scene and the camera poses of each view. Unlike traditional scene-by-scene iterative optimization methods, feedforward reconstruction networks do not require repeated iterative adjustments for each scene during the inference phase. Instead, they utilize pre-learned mapping relationships from 2D images to 3D structures to generate 3D scene data through a single forward computation, resulting in a short inference time, typically in the order of seconds.

[0020] The teacher-student framework is a technical framework for model compression. In a 3D Gaussian scattering edge deployment scenario, a high-quality, large-scale scene dataset is first constructed as the teacher scene dataset. Then, using this teacher scene dataset as a supervision signal, a smaller-scale scene dataset is generated as the student scene dataset. The student scene dataset is then deployed to a mobile terminal for execution.

[0021] It should be noted that the teacher scene data and student scene data in the embodiments of this application are both three-dimensional Gaussian scene data, that is, a data set that describes a three-dimensional scene through a large number of Gaussian points with attribute parameters (including position, spherical harmonic coefficient, scale, rotation and opacity, etc.). Its essence is scene description data, rather than neural network model or machine learning model.

[0022] II. Application Scenarios Currently, while the teacher-student framework can reduce the storage and rendering pressure on mobile terminals to some extent, its overall process is time-consuming. Among them, the construction of teacher scenario data is one of the most time-consuming steps.

[0023] In related technologies, the construction of teacher scene data mainly adopts a scene-by-scene iterative optimization approach: First, the Structure for Motion Recovery algorithm is used to extract sparse point clouds from multiple input multi-angle views. Then, Gaussian points are placed at the initial positions using this sparse point cloud. Through multiple iterations, the position, color, size, orientation, and transparency of the Gaussian points are gradually adjusted. During this process, each iteration requires rendering based on the current Gaussian point cloud, comparing the rendered result with the real view, calculating the gradient based on the differences, and updating the attribute parameters of each Gaussian point in reverse. This process typically requires tens of thousands of iterations to generate dense and high-quality teacher scene data, and the construction time is relatively long, usually ranging from tens of minutes to an hour.

[0024] Furthermore, in the student scene data generation stage, the relevant technology employs a single-path distillation design. This means that after the teacher scene data is constructed, it needs to undergo a complete distillation process to generate the student scene data. This entire process is quite lengthy, typically taking several hours. Before distillation is complete, users cannot see any 3D scene effects and can only view the results after the entire process is finished, further increasing the user's waiting time and resulting in a poor user experience.

[0025] To address the aforementioned technical problems, embodiments of this application provide a data processing method and apparatus. The data processing method provided in this application will be described in detail below with reference to the accompanying drawings.

[0026] The data processing method provided in this application can be applied to cloud-terminal collaborative scenarios. The first electronic device can be a cloud device (such as a server), and the second electronic device can be a terminal device (such as a mobile phone, tablet, or other mobile terminal). Data transmission between the first and second electronic devices can occur via a network (such as a wired network or wireless network). A user can take multiple multi-angle views using the second electronic device, which then sends these views to the first electronic device. The first electronic device executes the data processing method provided in this application to generate first student scene data and second student scene data, and sends both data to the second electronic device. The second electronic device previews the scene based on the first student scene data and automatically replaces it after the second student scene data is generated, thus achieving a progressive experience from quick preview to high-quality display on the second electronic device.

[0027] The data processing method provided in this application embodiment can also be applied to local processing scenarios on terminal devices. In this scenario, the first electronic device and the second electronic device can be the same device, that is, the same mobile phone or other terminal device is responsible for both capturing multiple multi-angle views and performing 3D reconstruction and data processing, and can directly preview and display the data on the local device.

[0028] In practical applications, either of the two scenarios can be flexibly selected based on specific needs. If users have high requirements for computing power and storage space, or need multiple users and devices to share 3D scene data, they can choose the cloud-terminal collaborative scenario, where the cloud server completes the computationally intensive reconstruction and compression tasks, and the terminal device is only responsible for shooting and displaying. If users want to reduce data transmission, reduce network dependence, or have high requirements for data privacy, they can choose the terminal device local processing scenario, where the terminal device independently completes the entire process. Both scenarios can achieve a rapid preview and high-quality progressive delivery of the data processing method provided in this application embodiment.

[0029] Figure 1 This is one of the flowcharts of a data processing method provided in some embodiments of this application, the method being applied to a first electronic device, such as... Figure 1 As shown, the method may include the following steps: step 101, step 102, step 103 and step 104.

[0030] In step 101, N multi-angle views are acquired by the second electronic device; where N is an integer greater than 1.

[0031] In this embodiment, a multi-angle view refers to multiple images of the same target scene or object captured from different shooting positions and / or different shooting angles. These views have overlapping fields of view, providing appearance information of the target scene or object from different perspectives, thus providing a data foundation for subsequent 3D reconstruction.

[0032] Considering that if N is too small (e.g., 1 image), insufficient viewpoint information will be provided for 3D reconstruction, resulting in poor reconstruction quality; conversely, if N is too large (e.g., hundreds of images), it will increase the computational load and inference time of the feedforward reconstruction network, reducing processing efficiency. Therefore, in this embodiment, the value of N can be set to a range of 8 to 32 images, which provides sufficient viewpoint information to ensure reconstruction quality while controlling the computational load for fast processing. The value of N can be determined according to actual needs, and this application does not impose specific limitations on it.

[0033] In step 102, N multi-angle views are input into the feedforward reconstruction network for 3D reconstruction processing to obtain the original teacher scene data and the predicted camera poses of each of the N multi-angle views; wherein, the original teacher scene data includes attribute parameters of multiple Gaussian points.

[0034] In this embodiment, the feedforward reconstruction network can be a neural network based on the Transformer architecture. Because the Transformer architecture has powerful feature extraction and correlation modeling capabilities, it can effectively establish pixel-level correspondences between multiple views, thereby improving the accuracy and efficiency of 3D reconstruction.

[0035] In this embodiment, the original teacher scene data is 3D scene data directly output by the feedforward reconstruction network, including attribute parameters of multiple Gaussian points. The attribute parameters of each Gaussian point include position (3D coordinates), spherical harmonic coefficients (used to encode viewpoint-related colors), scale (size in 3D space), rotation (quaternion representing orientation), and opacity (transparency information). The original teacher scene data is large in volume (typically containing hundreds of thousands to millions of Gaussian points), representing high-quality, large-scale scene data, suitable for use as teacher data in subsequent processing.

[0036] In this embodiment, predicted camera pose refers to the pose information obtained by the feedforward reconstruction network predicting the shooting position and orientation of each of the N input multi-angle views while outputting 3D scene data. Each predicted camera pose includes the 3D coordinates of the camera center and the camera's orientation information. The predicted camera pose is used for subsequent coordinate system alignment processing.

[0037] In related technologies, the construction of teacher scene data mainly adopts a scene-by-scene iterative optimization approach, requiring tens of thousands of iterations and taking tens of minutes to an hour. However, the embodiment of this application can output the original teacher scene data by performing a single forward inference through a feedforward reconstruction network, with inference time only in the seconds, thus shortening the construction time of teacher scene data and improving processing efficiency.

[0038] In step 103, based on the predicted camera pose and the calibration camera pose corresponding to the second electronic device, the original teacher scene data is subjected to coordinate system alignment processing, and the aligned teacher scene data is subjected to parameter format conversion processing to obtain intermediate teacher scene data that matches the coordinate system of the second electronic device.

[0039] In this embodiment, the feedforward reconstruction network generates the original teacher scene data in its own predicted coordinate system, which is formed autonomously by the network during training and is inconsistent with the coordinate system of the real world (i.e., the coordinate system of the second electronic device). If the original teacher scene data is directly used for subsequent processing, the pose of the rendered 3D scene will deviate from the real pose.

[0040] In this embodiment, the calibrated camera pose refers to the actual camera pose corresponding to each of the N multi-angle views captured by the second electronic device. This calibrated camera pose can be obtained through pre-calibration of the camera system of the second electronic device, or it can be acquired in real time through a positioning system (such as visual inertial odometry, global positioning system, etc.) that works with the second electronic device.

[0041] In this embodiment of the application, the coordinate system alignment process utilizes the correspondence between the predicted camera pose and the calibrated camera pose to calculate the transformation parameters from the predicted coordinate system to the coordinate system of the second electronic device. Then, each Gaussian point in the original teacher scene data is transformed from the predicted coordinate system to the coordinate system of the second electronic device, so that the transformed Gaussian point cloud is consistent with the pose of the real world.

[0042] In this embodiment, the data format of the original teacher scene data output by the feedforward reconstruction network is inconsistent with the data format requirements of the subsequent processing flow. Therefore, it is necessary to perform parameter format conversion processing on each Gaussian point in the aligned teacher scene data so that the converted data format is compatible with the subsequent processing flow.

[0043] In this embodiment, the intermediate teacher scene data refers to the 3D scene data obtained by aligning the coordinate system and converting the parameter format of the original teacher scene data, which matches the coordinate system of the second electronic device and is compatible with the format of subsequent processing procedures. The intermediate teacher scene data still includes attribute parameters of multiple Gaussian points, used to generate the first student scene data and the second student scene data.

[0044] In this embodiment, by aligning the coordinate systems, the scene data generated by the feedforward reconstruction network in the prediction coordinate system is transformed into the real coordinate system of the second electronic device, ensuring that the pose of the 3D scene in subsequent processing is consistent with that of the real world. Through parameter format conversion, the output data format of the feedforward reconstruction network meets the format requirements of subsequent processing flows, enabling the teacher scene data generated by the feedforward reconstruction network to be smoothly integrated into subsequent processing flows, thereby truly translating the speed advantage of the feedforward reconstruction network into usability.

[0045] In step 104, based on the intermediate teacher scene data, first student scene data and second student scene data are generated; wherein, the generation speed of the first student scene data is greater than that of the second student scene data, and the image quality of the second student scene data is better than that of the first student scene data; the first student scene data is used for previewing on the second electronic device; the second student scene data is used to replace the first student scene data after generation is completed.

[0046] In this embodiment, the first student scene data and the second student scene data are two sets of scene description data with different granularities and purposes, both describing the same 3D scene. The first student scene data is generated faster than the second student scene data, meaning the first student scene data can be generated in a shorter time to quickly provide a preview to the user. The second student scene data has better image quality than the first student scene data, meaning it has higher rendering quality and is suitable as the scene data for final display. The relationship between the two is as follows: the first student scene data is used for previewing on a second electronic device, allowing the user to see the 3D scene effect in a shorter time; the second student scene data is used to replace the first student scene data after generation, ensuring both preview speed and final display image quality.

[0047] In this embodiment, a dual-path distillation design is employed after the teacher scene data is constructed. The first path is a fast preview path, used to generate the first student scene data in a short time (e.g., tens of seconds). This first student scene data is large in size (e.g., tens of MB) and has relatively low image quality, but it can be quickly provided to the user for preview, allowing the user to see the 3D scene effect without waiting for hours. The second path is a high-quality refinement path, used to generate the second student scene data over a longer period (e.g., several hours). This second student scene data is smaller in size (e.g., a few MB) and has higher image quality. After generation, it automatically replaces the first student scene data, allowing the user to seamlessly switch to a high-quality display effect. This dual-path distillation design solves the problem of users having to wait a long time to see the 3D scene in related technologies, while also ensuring preview speed and final display quality, thus improving the user experience.

[0048] For example, the first electronic device is a cloud server, and the second electronic device is a mobile phone. User A uses the mobile phone to take multiple multi-angle views of a sculpture from multiple different angles, and the mobile phone sends the multiple multi-angle views to the cloud server.

[0049] The cloud server acquires N multi-angle views (e.g., 16 images) captured by the mobile phone. These N multi-angle views are then input into a feedforward reconstruction network for 3D reconstruction, yielding the original teacher scene data and the predicted camera poses for each of the N multi-angle views. The original teacher scene data includes attribute parameters for multiple Gaussian points, each of which includes attributes such as position, spherical harmonic coefficients, scale, rotation, and opacity.

[0050] The cloud server performs coordinate system alignment processing on the original teacher scene data based on the predicted camera pose and the corresponding calibration camera pose of the mobile phone, transforming the original teacher scene data from the predicted coordinate system of the feedforward reconstruction network to the coordinate system of the mobile phone. Subsequently, the cloud server performs parameter format conversion processing on the aligned teacher scene data to obtain intermediate teacher scene data that matches the coordinate system of the mobile phone.

[0051] Finally, the cloud server generates first and second student scene data based on the intermediate teacher scene data. The first student scene data is generated quickly, within tens of seconds, and after being transmitted to the user's mobile phone, the user can see a 3D preview of the sculpture; the second student scene data is generated slowly, several hours later, and after being transmitted to the mobile phone, it automatically replaces the first student scene data, allowing the user to see a high-quality, detailed image of the sculpture.

[0052] As can be seen from the above embodiments, in this embodiment, the first electronic device generates teacher scene data through a feedforward reconstruction network, replacing the scene-by-scene iterative optimization method in related technologies, shortening the construction time of teacher scene data and improving the construction efficiency of teacher scene data; on this basis, through coordinate alignment processing and parameter format conversion processing, the output of the feedforward reconstruction network can smoothly enter the subsequent process, truly transforming the speed advantage into usability; furthermore, by generating first student scene data and second student scene data, the first student scene data is generated quickly and can be quickly transmitted to the second electronic device for user preview, avoiding long waiting times for users and solving the problem that users need to wait a long time to see the 3D scene; the second student scene data automatically replaces the first student scene data after it is generated, ensuring preview speed while taking into account the final display quality, thus improving the user experience.

[0053] In some embodiments provided in this application, applicable to multi-view input scenarios, step 101 may specifically include the following steps: step 1011, step 1012 and step 1013.

[0054] In step 1011, M multi-angle views captured by the second electronic device are received; where M is an integer greater than 1 and N ≤ M.

[0055] In this embodiment, a second electronic device (such as a mobile phone) responds to a user's shooting operation by taking multiple pictures of the target scene or object from multiple different angles, obtaining M multi-angle views. The second electronic device sends the M multi-angle views to a first electronic device, and correspondingly, the first electronic device receives the M multi-angle views sent by the second electronic device.

[0056] In this embodiment, after the first electronic device acquires M multi-angle views, it determines N multi-angle views from the M multi-angle views and inputs them into the feedforward reconstruction network. Optionally, if M is greater than a preset threshold, step 1012 is executed; if M is less than or equal to the preset threshold, step 1013 is executed.

[0057] In step 1012, when M is greater than a preset quantity threshold, N multi-angle views are uniformly selected from the M multi-angle views according to the spatial distribution of the viewpoints; wherein N is less than or equal to the preset quantity threshold.

[0058] In this embodiment of the application, the preset quantity threshold can be 8 to 32 images, for example, the preset quantity threshold can be 8, 16 or 32 images.

[0059] In this embodiment, N multi-angle views are uniformly selected from M multi-angle views according to the spatial distribution of the viewpoints. This ensures that the N views input to the feedforward reconstruction network are uniformly covered in terms of spatial viewpoints, avoiding voids or distortions in the reconstruction results caused by the viewpoints being concentrated in a certain local area, thereby improving the integrity and accuracy of the 3D reconstruction. For example, if all the selected views are concentrated on the front of the target scene, the geometry of the back and sides of the scene cannot be effectively reconstructed; however, by uniformly selecting the views according to the spatial distribution of the viewpoints, the input views can cover all directions of the target scene, allowing the feedforward reconstruction network to obtain more comprehensive scene information, thereby outputting more complete and accurate 3D scene data.

[0060] In step 1013, if M is less than or equal to a preset quantity threshold, the M multi-angle views are determined as N multi-angle views.

[0061] For example, the first electronic device is a server, the second electronic device is a mobile phone, M is 100, and the preset data threshold and N are both 16. User A uses the mobile phone to take 100 multi-angle views around a sculpture, and the mobile phone sends the 100 multi-angle views to the server. After receiving the 100 multi-angle views, the server determines that the preset number threshold is 16. Since 100 is greater than 16, the server selects 16 multi-angle views evenly from the 100 multi-angle views according to the spatial distribution of the viewpoints, and inputs these 16 multi-angle views into the feedforward reconstruction network for 3D reconstruction processing.

[0062] In this embodiment of the application, by setting a preset quantity threshold, N views are selected evenly when M is greater than the preset quantity threshold, and all views are used when M is less than or equal to the preset quantity threshold.

[0063] As can be seen, in this embodiment of the application, the above method ensures that the number of input views is sufficient to support 3D reconstruction, while controlling the computational load of the feedforward reconstruction network, thus achieving a balance between reconstruction quality and processing efficiency.

[0064] In some embodiments provided in this application, when performing coordinate system alignment processing, step 103 may specifically include the following steps: step 1031, step 1032, step 1033 and step 1034.

[0065] In step 1031, the predicted camera center coordinates of N multi-angle views are extracted from the predicted camera pose to obtain N predicted camera center coordinates.

[0066] In this embodiment, when the feedforward reconstruction network performs 3D reconstruction processing on N multi-angle views, in addition to outputting the original teacher scene data, it also outputs the predicted camera pose corresponding to each view. Each predicted camera pose includes the camera's position and orientation information, where the camera position is usually represented by a 3D coordinate, which is the camera center coordinate. This step extracts the camera center coordinates from the N predicted camera poses to obtain N predicted camera center coordinates, which are used for subsequent correspondence and comparison with the calibrated camera center coordinates.

[0067] In step 1032, the center coordinates of the calibration camera are extracted from the pose of N multi-angle views to obtain N center coordinates of the calibration camera.

[0068] In this embodiment, the calibrated camera pose refers to the actual camera pose corresponding to each of the N multi-angle views captured by the second electronic device. This calibrated camera pose can be obtained through pre-calibration of the camera system of the second electronic device, or it can be acquired in real time through a positioning system (such as visual inertial odometry, global positioning system, etc.) that works with the second electronic device. Each calibrated camera pose also includes the camera's position and orientation information. This step extracts the camera center coordinates from the N calibrated camera poses to obtain N calibrated camera center coordinates.

[0069] In this embodiment of the application, since the N multi-angle views are the same set of views, there is a one-to-one correspondence between the predicted camera center coordinates and the calibrated camera center coordinates, that is, the predicted camera center coordinates of the c-th view correspond to the calibrated camera center coordinates of the c-th view, where 1≤c≤N.

[0070] In step 1033, similarity transformation parameters are determined based on N predicted camera center coordinates and N calibrated camera center coordinates.

[0071] In this embodiment, the predicted camera center coordinates are the camera position output by the feedforward reconstruction network in the predicted coordinate system, while the calibrated camera center coordinates are the camera position of the second electronic device in the real coordinate system. There is a fixed spatial transformation relationship between the two sets of coordinates, which can be described by similarity transformation parameters.

[0072] In this embodiment, the similarity transformation is a shape-preserving transformation. The similarity transformation parameters include three components: scaling, rotation, and translation, corresponding to the scaling factor s, the rotation matrix R, and the translation vector t, respectively. The similarity transformation can preserve the shape of the Gaussian point, only transforming the position, scale, and orientation of the Gaussian point as a whole, thereby ensuring that the geometric structure of the scene is not destroyed. The mathematical form of the similarity transformation is as follows: Formula (1): (1) Where x is the coordinate before the transformation. These are the transformed coordinates.

[0073] In this embodiment, N pairs of corresponding points (each pair including a predicted camera center coordinate and a calibration camera center coordinate) are used to solve for the optimal similarity transformation parameters through Singular Value Decomposition (SVD), minimizing the overall error between the transformed predicted camera center coordinates and the calibration camera center coordinates. SVD is a commonly used matrix factorization method that can effectively solve for the optimal transformation in the least squares sense.

[0074] In step 1034, each Gaussian point in the original teacher scene data is transformed from the prediction coordinate system of the feedforward reconstruction network to the coordinate system of the second electronic device according to the similarity transformation parameters, so as to obtain the aligned teacher scene data.

[0075] In this embodiment, after determining the similarity transformation parameters, the position, scale, and rotation of each Gaussian point in the original teacher scene data are transformed according to the similarity transformation parameters. Specifically, the position transformation uses formula (1) to transform the position of the Gaussian point from the prediction coordinate system to the coordinate system of the second electronic device; the scale transformation uses the following formula (2), and the scale of the Gaussian point changes synchronously with the overall scaling ratio; the rotation transformation uses the following formula (3), and the orientation of the Gaussian point changes synchronously with the overall rotation.

[0076] (2) (3) in, This represents the scale vector of the Gaussian point before transformation in the predicted coordinate system, containing scale values ​​along three axes; q represents the scale vector of the Gaussian point in the true coordinate system after transformation; q represents the rotation quaternion of the Gaussian point in the predicted coordinate system before transformation, which is used to indicate the orientation of the Gaussian ellipsoid. This represents the quaternion multiplication operation, used to combine two rotations into one rotation; This represents the rotation quaternion of the Gaussian point in the real coordinate system after the transformation.

[0077] As can be seen, in this embodiment of the application, through coordinate alignment processing, all Gaussian points in the original teacher scene data are migrated from the prediction coordinate system to the coordinate system of the second electronic device, eliminating the deviation between the prediction coordinate system of the feedforward reconstruction network and the real-world coordinate system, making the Gaussian point cloud consistent with the pose of the real world, and ensuring that the 3D scene in subsequent processing matches the real pose.

[0078] In some embodiments provided in this application, when performing parameter format conversion processing, step 103 above may specifically include the following steps: step 1035.

[0079] In step 1035, for each Gaussian point in the aligned teacher scene data, perform at least one of the following parameter format conversions: convert the spherical harmonic coefficients of the Gaussian point from channel-first format to order-first format, and truncate or fill with zeros according to the target order; convert the scale parameter of the Gaussian point from the linear domain to the logarithmic domain; convert the opacity parameter of the Gaussian point from the probability domain to the log-odds domain; and convert the component order of the rotation quaternion of the Gaussian point from (x,y,z,w) to (w,x,y,z).

[0080] In this embodiment, spherical harmonic coefficients are used to encode the view-related color information of Gaussian points. Feedforward reconstruction networks typically store spherical harmonic coefficients in a channel-first format, meaning the coefficients of the same color channel for all Gaussian points are stored consecutively; for example, the R channel coefficients of all Gaussian points are stored consecutively, followed by the G channel, and then the B channel. However, subsequent processing requires an order-first format, meaning the coefficients of the same order for all Gaussian points are stored consecutively. This difference in formats means the output of the feedforward reconstruction network cannot be directly read by subsequent processing. Therefore, it is necessary to transpose the spherical harmonic coefficients of each Gaussian point, converting the channel-first format to an order-first format. Furthermore, different application scenarios have different requirements for color accuracy. This embodiment can also truncate the spherical harmonic coefficients according to the target order (retaining lower-order coefficients and discarding higher-order coefficients) or fill them with zeros (adding zeros when the target order is higher than the current order) to adapt to the balance between color accuracy and storage overhead in different scenarios.

[0081] For example, taking the truncation of a third-order spherical harmonic (16 coefficients per channel) to a first-order (4 coefficients per channel) as an example, the coefficients of the R channel of a certain Gaussian point are r0~r15, the G channel is g0~g15, and the B channel is b0~b15. During truncation, only the first 4 coefficients of each channel are retained: r0~r3, g0~g3, and b0~b3, while higher-order coefficients such as r4~r15 are discarded. At the same time, the coefficient arrangement format is changed from channel-first format to order-first format. Before the conversion, it is r0,r1,...,r15,g0,g1,...,g15,b0,b1,...,b15, and after the conversion, it is r0,g0,b0,r1,g1,b1,...,r3,g3,b3. Through truncation and format transposition, the number of color parameters for each Gaussian point is reduced from 48 to 12, and the arrangement format is compatible with subsequent processing, reducing storage overhead and computational load. Among them, r0, g0, and b0 are zero-order coefficients, representing the base color of the Gaussian point (the diffuse component independent of the viewing angle); r1~r3, g1~g3, and b1~b3 are first-order coefficients, representing the low-frequency color components that vary with the viewing angle. Subsequent processing uses only these first-order coefficients for rendering. Although this sacrifices fine color variation details such as highlights and reflections, it is sufficient to meet the visual requirements of the preview scene.

[0082] In this embodiment, the scale parameter represents the size of the Gaussian point along three axes. The scale parameter output by the feedforward reconstruction network is located in the linear domain, for example, with scale values ​​of 0.01, 0.1, 1.0, etc.; while subsequent processing requires the scale parameter to be located in the logarithmic domain. The reason for applying the logarithmic domain to the scale parameter is that the scale of different Gaussian points may span multiple orders of magnitude, for example, from 0.001 to 10. In the linear domain, the gradient difference between large and small scale parameters is large during optimization, leading to unstable optimization; while the logarithmic domain can map scales of different orders of magnitude to a relatively balanced numerical range, making the optimization process more stable and efficient. Therefore, it is necessary to perform a logarithmic transformation on the scale parameter of each Gaussian point, converting it from the linear domain to the logarithmic domain.

[0083] In this embodiment, the opacity parameter represents the transparency of the Gaussian point, ranging from 0 (completely transparent) to 1 (completely opaque). The opacity parameter output by the feedforward reconstruction network is located in the probability domain (between 0 and 1); however, subsequent processing requires the opacity parameter to be located in the logit domain (range from negative infinity to positive infinity). The reason for applying the logit domain to the opacity parameter is that during optimization, the attribute parameters of the Gaussian point need to be updated via gradient descent. If the parameter is located in the probability domain (0 to 1), the parameter value is restricted to a finite interval, and it is easy to reach the boundary during optimization (e.g., exceeding the range of 0 to 1 after updating), leading to gradient vanishing or parameter out-of-bounds errors. The logit domain, however, has no boundary restrictions, allowing the parameter to be freely updated across the entire real number range, avoiding out-of-bounds errors during optimization and improving the stability and effectiveness of the optimization. In practical applications, the opacity parameter can be converted from the probability domain to the logit domain using the inverse sigmoid function (logit function).

[0084] In this embodiment, a quaternion is a mathematical tool used to represent rotation in three-dimensional space, containing four components. Different algorithms or processing flows have different conventions regarding the order of quaternion components. The quaternion component order output by the feedforward reconstruction network is (x, y, z, w), while subsequent processing flows require the component order to be (w, x, y, z). If the component order is inconsistent, subsequent flows will be unable to correctly parse the orientation information of Gaussian points, resulting in incorrect orientation of the Gaussian ellipsoid during rendering. Therefore, it is necessary to rearrange the components of the rotation quaternion for each Gaussian point, transforming the component order from (x, y, z, w) to (w, x, y, z) to ensure that the orientation information is correctly parsed.

[0085] As can be seen, in this embodiment of the application, through parameter format conversion processing, the original teacher scene data output by the feedforward reconstruction network is fully compatible with the subsequent processing flow in terms of data format, so that the teacher scene data generated by the feedforward reconstruction network in seconds can be smoothly connected to the subsequent processing flow, thereby truly transforming the speed advantage of the feedforward reconstruction network into usability.

[0086] In some embodiments provided in this application, the intermediate teacher scene data can be optimized first, and then the first student scene data and the second student scene data can be generated based on the optimized teacher scene data, such as... Figure 2 As shown, step 104 above may specifically include the following steps: step 1041, step 1042 and step 1043.

[0087] In step 1041, the intermediate teacher scene data undergoes a first iteration of optimization processing for T rounds, and Gaussian points in areas missing due to lack of view coverage are added during the first iteration of optimization processing for T rounds, resulting in filled teacher scene data; where T is an integer greater than 1, the first iteration of optimization processing includes: rendering the intermediate teacher scene data based on at least one of N multi-angle views to obtain a first rendered image, and adjusting the attribute parameters of Gaussian points in the intermediate teacher scene data according to the difference between the first rendered image and at least one view.

[0088] In this embodiment, the first iterative optimization process is a rendering-comparison-adjustment loop: in each iteration, at least one view is selected from N multi-angle views, and an image (i.e., the first rendered image) is rendered using the current Gaussian point cloud in the camera pose of that view. The rendered image is compared with the real photo of that view, and the differences between the two (such as pixel color differences, structural differences, etc.) are calculated. Based on the difference, the gradient is backpropagated, and the attribute parameters such as the position, color, scale, rotation, and opacity of the Gaussian points are adjusted so that the rendered image is closer to the real view in the next iteration.

[0089] In this embodiment, although the intermediate teacher scene data is already dense 3D scene data, the number of views input to the feedforward reconstruction network is limited (usually 8 to 32). Some areas not covered by the viewpoint may lack Gaussian points, resulting in holes in the scene. To fill these holes, this embodiment introduces densification processing during the first iterative optimization process. Densification processing is interspersed throughout the iteration process. For example, after every f iterations, it checks whether there are any hole areas in the scene lacking Gaussian points. If so, new Gaussian points are generated in the hole areas or surrounding Gaussian points are copied to the hole areas, allowing the newly added Gaussian points to participate in subsequent iterative optimization. This not only fills the holes in the areas not covered by the viewpoint but also allows the newly added Gaussian points to gradually adapt to the surrounding environment through subsequent iterations, ensuring the integrity of the scene.

[0090] In some embodiments, the number of iterations in the first iteration of the T-round optimization process can be 3,000 to 10,000, which is about one-tenth to one-third of the number of iterations in traditional optimization (30,000 to 50,000). Since the intermediate teacher scene data has already been provided with dense and relatively accurate initial scene data by the feedforward reconstruction network, it is not necessary to grow it step by step from a sparse point cloud as in traditional methods. Therefore, optimization can be completed with fewer iterations. In other embodiments, the number of iterations T can also be 50 to 100, and users can flexibly choose whether to enable this short-round quality recovery according to actual needs.

[0091] In step 1042, the filled teacher scene data is rendered on N multi-angle views respectively. The opacity contribution of each Gaussian point on the N multi-angle views is accumulated. The points are sorted from high to low according to their contribution, and the top K Gaussian points are retained. The remaining Gaussian points are removed to obtain the optimized teacher scene data. Here, K is an integer greater than 1.

[0092] In this embodiment of the application, after iterative optimization processing, the number of Gaussian points may increase significantly (due to the densification process supplementing new Gaussian points). Some of these Gaussian points may contribute very little to the rendered image. These points are either very transparent (almost invisible), or very small in scale (extremely narrow coverage), or completely occluded by other Gaussian points. These low-contribution Gaussian points not only increase the amount of data, but also increase the storage and computational overhead of subsequent processing.

[0093] To identify and remove these low-contribution Gaussian points, in this embodiment, the filled teacher scene data is rendered on N multi-angle views. During rendering, each Gaussian point contributes a certain opacity value to the pixels in the view. The contribution value of the Gaussian point to all pixels is accumulated on each view to obtain the opacity contribution of the Gaussian point on that view. Then, the opacity contribution of the Gaussian point on all views is added together to obtain the total opacity contribution of the Gaussian point. All Gaussian points are sorted from high to low according to the total opacity contribution, and the top K Gaussian points (K is a preset retention number) are retained, while the remaining Gaussian points are removed.

[0094] In this embodiment of the application, by using importance culling, Gaussian points that contribute very little to the rendering can be removed, effectively controlling the number of Gaussian points while ensuring image quality, thereby reducing the amount of data and the computational overhead of subsequent processing.

[0095] For example, the filled teacher scene data contains 1 million Gaussian points. After rendering on N multi-angle views and accumulating the opacity contribution of each Gaussian point, the data is sorted from highest to lowest contribution. The top K=800,000 Gaussian points are retained, and the 200,000 Gaussian points with the lowest contribution are removed. These 200,000 removed Gaussian points contribute very little to the rendered image, and the image quality is basically unaffected after removal. However, the data volume is reduced from 1 million points to 800,000 points, reducing storage and computational overhead by 20%.

[0096] In this embodiment of the application, through the above optimization process, the quality of teacher scene data is improved, gaps are filled, and low-contribution Gaussian points are removed, so that the subsequently generated student scene data has higher quality and a more reasonable volume, thereby improving processing efficiency.

[0097] In step 1043, based on the optimized teacher scenario data, first student scenario data and second student scenario data are generated.

[0098] In this embodiment, the optimized teacher scenario data has undergone iterative optimization (filling gaps and optimizing Gaussian point attribute parameters) and importance removal (removing Gaussian points with low contribution), resulting in high quality and controllable data volume. It is suitable as teacher data for generating the first student scenario data and the second student scenario data.

[0099] In this embodiment, the first student scene data is used for rapid preview on a second electronic device. It is generated quickly, has a large data volume, and relatively low image quality. The second student scene data is used to replace the first student scene data after generation. It is generated more slowly, has a smaller data volume, and higher image quality. Both share the same optimized teacher scene data as a supervision signal and can be generated in parallel.

[0100] As can be seen, in this embodiment, by iteratively optimizing and densifying the intermediate teacher scene data, areas missing due to incomplete viewpoint coverage are filled, improving the scene's completeness. Through importance culling, low-contribution Gaussian points are removed, controlling the data volume while maintaining image quality, thus improving the quality and making the volume of the teacher scene data more reasonable. Based on this, first and second student scene data are generated in parallel through dual pathways. The first student scene data is generated quickly for rapid preview, avoiding long user wait times. The second student scene data automatically replaces the first student scene data after generation for high-quality display, thereby shortening user waiting time and improving user experience while ensuring final image quality.

[0101] In some embodiments provided in this application, when generating the first student scene data based on the optimized teacher scene data, the above step 1043 may specifically include the following steps: step 10431, step 10432, step 10433 and step 10434.

[0102] In step 10431, the spherical harmonic coefficients of each Gaussian point in the optimized teacher scene data are reduced in order to obtain the reduced-order teacher scene data.

[0103] In this embodiment, each Gaussian point in the optimized teacher scene data carries spherical harmonic coefficients to encode viewpoint-related color information. Higher-order spherical harmonic coefficients (e.g., third-order, 16 coefficients per channel) can finely represent color details such as highlights and reflections that change with viewpoint, but the data volume is large. Order reduction processing refers to truncating the spherical harmonic coefficients from higher to lower orders, for example, from third-order (16 coefficients per channel) to first-order (4 coefficients per channel), retaining only the zero-order DC component and lower-order AC components. Simultaneously, the format transpose of the channel dimension and order dimension is completed, making the reduced-order teacher scene data compatible with mobile renderers and reducing processing time. After order reduction, the number of color parameters per Gaussian point is reduced from 48 to 12, significantly reducing the data volume.

[0104] In this embodiment, although the reduction processing loses some color details (such as subtle changes in highlights and reflections), it preserves the basic colors and main viewpoint-related changes in the first-order spherical harmonic coefficients, which can meet the visual requirements of the preview scene. Through the reduction processing, the generation speed of the first student scene data is improved, the data volume is controlled, and it is compatible with the data format of the mobile renderer, ensuring that the first student scene data can be correctly rendered on the mobile device.

[0105] In step 10432, the importance score of each Gaussian point in the reduced-order teacher scene data is calculated based on the opacity parameter and scale parameter of the Gaussian point.

[0106] In this embodiment, the importance score is used to measure the importance of each Gaussian point in a 3D scene. The opacity and scale parameters of a Gaussian point reflect its contribution to the rendered image. The opacity parameter reflects the transparency of the Gaussian point; the higher the opacity, the more prominent the point is during rendering, and the higher its importance. The scale parameter reflects the size of the Gaussian point in 3D space; the larger the scale, the wider the area covered by the point, and the higher its importance. A comprehensive importance score is calculated based on these two parameters. Specifically, the opacity parameter after sigmoid activation is used as the opacity score, and the scale parameter (e.g., the ratio of the product of the three axes to the 90th percentile volume) after a power transformation is used as the volume-aware score. The two are multiplied to obtain the comprehensive importance score. A higher importance score indicates a greater contribution of the Gaussian point to the scene, making it more worthy of retention.

[0107] In step 10433, the data is sorted from high to low according to importance score, and the top Gaussian points of the first preset percentage are retained. The remaining Gaussian points are removed to obtain lightweight teacher scenario data.

[0108] In this embodiment, after calculating the importance score of each Gaussian point, all Gaussian points are sorted from high to low according to their importance scores. The top-ranked Gaussian points (i.e., those with higher importance) are retained, while the lower-ranked Gaussian points (i.e., those with lower importance) are removed. The first preset percentage can be set according to actual needs, for example, it can be set to 40% to 60%.

[0109] In this embodiment, lightweight teacher scene data refers to scene data obtained by reducing the order of spherical harmonic coefficients and pruning based on attribute importance of optimized teacher scene data. Compared with the optimized teacher scene data, the spherical harmonic coefficients of each Gaussian point are reduced from high order to low order (e.g., from third order to first order), and the number of color parameters is reduced from 48 to 12; at the same time, only the Gaussian points with the highest importance scores in the first preset percentage are retained, and the rest are removed. Through the above processing, the amount of lightweight teacher scene data is significantly reduced, the generation speed is greatly improved, and the needs of rapid preview are met. At the same time, the basic colors and main viewpoint-related changes in the first-order spherical harmonic coefficients, as well as the Gaussian points with high importance, are retained, so that the preview screen can reflect the overall structure and main color information of the 3D scene. Lightweight teacher scene data can be used directly as the first student scene data, or it can be used as the first student scene data after a short round of iterative optimization.

[0110] In this embodiment, lightweight cropping reduces the number of Gaussian points and decreases data volume while maintaining image quality. Unlike the importance elimination in step 1042, which is based on rendering contribution (actual rendering result) and involves more computation but is more accurate, this step scores and crops based on the Gaussian points' own properties (opacity and scale), without requiring a training camera or rendering, resulting in faster computation and suitability for use in scenarios requiring rapid preview paths.

[0111] In step 10434, the first student scenario data is determined based on the lightweight teacher scenario data.

[0112] In some embodiments, step 10434 may specifically include the following steps: step 104341 or step 104342.

[0113] In step 104341, the lightweight teacher scenario data is determined as the first student scenario data. In step 104342, the lightweight teacher scenario data is subjected to Z rounds of second iteration optimization to obtain the first student scenario data; Where Z is an integer greater than 1, the second iteration optimization process includes: rendering the lightweight teacher scene data based on at least one of the N multi-angle views to obtain a second rendered image, and adjusting the attribute parameters of the Gaussian points in the lightweight teacher scene data according to the difference between the second rendered image and at least one view.

[0114] In this embodiment of the application, the method for determining the first student scenario data may include the following two: The first method directly uses lightweight teacher scenario data as the primary student scenario data. This method requires no additional iterative optimization and has the fastest generation speed, making it suitable for scenarios requiring extremely fast previewing.

[0115] The second approach involves performing a Z-round second-iteration optimization on the lightweight teacher scene data to obtain the first student scene data. This method fine-tunes the attribute parameters of the Gaussian points through a small number of iterations, restoring the image quality to a certain extent, and is suitable for scenarios that require better preview quality. This second-iteration optimization does not introduce densification operations and takes approximately tens of seconds to several minutes.

[0116] In this embodiment, the second electronic device can flexibly choose either of the two methods described above according to actual needs. If the user prioritizes ultra-fast preview, they can choose the first method, directly using lightweight teacher scene data as the first student scene data, skipping short-round iteration optimization, and seeing the 3D scene effect as quickly as possible. If the user has higher requirements for preview image quality, they can choose the second method, enabling short-round iteration optimization, so that the image quality can be improved to a certain extent within an acceptable time range. By using the two optional methods described above, different needs for preview speed and preview image quality in different scenarios are taken into account.

[0117] As can be seen, in this embodiment of the application, the optimized teacher scene data is subjected to spherical harmonic coefficient reduction processing and lightweight pruning based on attribute importance. While retaining the main color information and key Gaussian points, the data volume of the first student scene data is reduced, the generation speed is improved, and the first student scene data can be generated and transmitted to the second electronic device for user preview in a short time, thereby avoiding long waiting time for users.

[0118] In some embodiments provided in this application, when generating second student scene data based on optimized teacher scene data, the above step 1043 may specifically include the following steps: step 10435, step 10436, step 10437 and step 10438.

[0119] In step 10435, the optimized teacher scene data undergoes P rounds of third iteration optimization processing. During the P rounds of third iteration optimization processing, every i rounds, a Gaussian point culling judgment is performed based on the rendering opacity and maximum scale of the Gaussian points. Gaussian points with rendering opacity lower than the first threshold and maximum scale lower than the second threshold are marked as to be culled. For Gaussian points marked as to be culled in j consecutive judgments, a culling operation is performed. Here, P, i, and j are all integers greater than 1, P > i ≥ j. The third iteration optimization processing includes: rendering the teacher scene data in the current iteration based on at least one of N multi-angle views to obtain a third rendered image, and adjusting the attribute parameters of the Gaussian points in the teacher scene data in the current iteration according to the difference between the third rendered image and at least one view.

[0120] In this embodiment, the third iteration optimization process is similar to the first iteration optimization process, which is a rendering-comparison-adjustment loop. In each iteration, at least one view is selected from N multi-angle views, and a third rendered image is rendered using the current Gaussian point cloud at the camera pose of that view. The rendered image is compared with the real photo of that view, and the position, color, scale, rotation, and opacity of the Gaussian points are adjusted according to the backpropagation gradient based on the difference, so that the rendered image gradually approximates the real photo.

[0121] In this embodiment, during the third iteration of optimization, a Gaussian point culling decision is performed every i rounds. For each Gaussian point, two metrics are calculated: rendering opacity and maximum scale. Rendering opacity reflects the visibility of the Gaussian point's contribution to the rendered image across all training views; a low value indicates the Gaussian point is almost invisible during rendering. Maximum scale refers to the maximum size of the Gaussian point along the three axes; a low value indicates the Gaussian point is small and has limited coverage. If the rendering opacity of a Gaussian point is below a first threshold and its maximum scale is below a second threshold, the Gaussian point is marked for culling.

[0122] In this embodiment, i represents the number of rounds between two elimination decisions, meaning that an elimination decision is performed every i rounds. For example, if i=50, then an elimination decision is performed in round 50, round 100, round 150, and so on. In practical applications, the value of i can be set according to the total number of iterations P and the required elimination frequency, for example, it can be any integer value between 50 and 200.

[0123] To prevent accidental deletion, this embodiment employs a voting-based removal mechanism. A Gaussian point needs to be marked as pending removal j times consecutively before the removal operation is performed. For example, if j is 3, a Gaussian point needs to meet the removal criteria in 3 consecutive checks before it is actually deleted. This voting-based removal mechanism avoids the accidental deletion of contributing Gaussian points due to a single abnormal check, ensuring the accuracy of the removal process.

[0124] In this embodiment of the application, through the above-mentioned joint elimination operation, Gaussian points that contribute little to rendering are gradually removed during the iterative optimization process, thereby continuously reducing the amount of data while ensuring image quality, and providing a foundation for subsequent network replacement and compression.

[0125] In step 10436, when the third iteration of optimization in round P is executed to round a, a lightweight neural network is constructed. The positional encoding in the Gaussian point attribute parameters is used as the input of the lightweight neural network, and the appearance attribute in the Gaussian point attribute parameters is used as the output of the lightweight neural network. The attribute parameters of each Gaussian point are replaced with the weight parameters of the lightweight neural network. Here, a is an integer greater than 1, and P > a > i.

[0126] In this embodiment, when the third iteration of optimization processing reaches the a-th round (e.g., about one-third of the way through training), a lightweight neural network (such as a multilayer perceptron, MLP) is constructed. The input of this network is the positional encoding of Gaussian points (mapping three-dimensional coordinates to a high-dimensional space), and the output is the appearance attributes of Gaussian points (including spherical harmonic coefficients, scale, rotation, and opacity). By training this network, it learns the mapping relationship from position to appearance attributes, and then the attribute parameters stored independently for each Gaussian point are replaced with the weight parameters of this network.

[0127] In this embodiment, since the attribute parameters of Gaussian points in a 3D scene change smoothly with position, and the attribute parameters of adjacent Gaussian points have a strong correlation, the lightweight neural network can utilize this spatial correlation to learn the mapping pattern from position to attribute, thereby replacing the parameters stored independently point-by-point with shared network weights. After the network parameter replacement, each Gaussian point only needs to store its position coordinates, and the appearance attributes are calculated in real time by the neural network based on the position during rendering, reducing storage space.

[0128] In this embodiment of the application, attribute parameters stored point by point are converted into shared network weights by network parameter replacement, thereby reducing the amount of data.

[0129] In step 10437, when the third iteration of optimization in round P is executed to round b, the weight parameters of the lightweight neural network are clustered to obtain multiple cluster centers. A codebook is constructed based on the multiple cluster centers, and the weight parameters of the lightweight neural network are replaced with the codebook index; where b is an integer greater than 1, and P > b > a.

[0130] In this embodiment, when the third iteration of optimization is executed to the b-th round (e.g., near the end of training), the weight parameters of the lightweight neural network have essentially converged. At this point, clustering is performed on the weight parameters, for example, using the K-means clustering algorithm, to cluster all weight parameters into multiple categories. The center of each category serves as a cluster center, and multiple cluster centers constitute a codebook. Subsequently, the original weight parameter values ​​are replaced with codebook indices; that is, each weight parameter no longer stores its original floating-point value, but instead stores the index number of the category to which the parameter belongs.

[0131] In this embodiment, since many parameter values ​​in the neural network weight parameters are similar or identical, clustering can be used to group these similar parameter values ​​into one category. A cluster center value is used to approximate all parameter values ​​in this category, and an index number is used to represent the category. In this way, what used to require storing a floating-point number (e.g., 32 bits) now only requires storing an index number (e.g., 10 bits), significantly reducing storage space.

[0132] In step 10438, after the third iteration of optimization in round P, the parameters after codebook index replacement are subjected to entropy encoding compression to obtain the second student scenario data.

[0133] Entropy coding is a lossless compression method that assigns different length codes based on the frequency of a symbol's occurrence: shorter codes for frequently occurring symbols and longer codes for less frequently occurring symbols, thereby reducing the average code length of the data overall. In this embodiment, after steps 10436 and 10437, the data exists in the form of a codebook index. The distribution of codebook index values ​​is usually uneven, with some index values ​​occurring more frequently than others. Entropy coding (e.g., Huffman coding) assigns short codes to high-frequency index values ​​and long codes to low-frequency index values, further removing data redundancy and improving compression efficiency.

[0134] As can be seen, in this embodiment, the optimized teacher scene data undergoes five progressive compression processes, including iterative optimization, joint elimination, network parameter replacement, vector quantization, and entropy encoding compression, ultimately resulting in second student scene data with small data volume and high image quality, which is suitable for deployment on mobile terminals for high-quality rendering.

[0135] In some embodiments provided in this application, the data processing method provided is described above. Figure 1 After step 104 in the illustrated embodiment, the following steps may be added: step 105 and step 106.

[0136] In step 105, the first student scene data is transmitted to the second electronic device so that a preview screen corresponding to the first student scene data is displayed on the second electronic device.

[0137] In this embodiment, after the first student scenario data is generated, the first electronic device transmits the data to the second electronic device. Data transmission between the first and second electronic devices can be conducted via wired or wireless networks, such as Wi-Fi, mobile networks, or Bluetooth.

[0138] In this embodiment, after receiving the first student scene data, the second electronic device renders the data and displays a corresponding preview on its screen. Users can interact with the preview using the second electronic device, such as rotating the viewpoint and zooming in, to browse the approximate shape and color of the 3D scene from different angles. The first student scene data is generated quickly (approximately tens of seconds), and transmission and display can also be completed in a short time, allowing users to see the 3D scene effect quickly without waiting for hours.

[0139] In step 106, after the second student scene data is generated, the second student scene data is transmitted to the second electronic device to replace the first student scene data, so that the screen corresponding to the second student scene data is displayed on the second electronic device.

[0140] In this embodiment, the generation of the second student scene data takes a considerable amount of time. During this generation period, the user has already browsed a preview of the 3D scene on the second electronic device using the first student scene data. Once the second student scene data is generated in the background, the first electronic device transmits the second student scene data to the second electronic device.

[0141] In this embodiment, after receiving the second student scene data, the second electronic device automatically replaces the currently displayed first student scene data with the second student scene data and displays the corresponding image on the screen. Since the image quality of the second student scene data is superior to that of the first student scene data, the user can see a more detailed 3D scene effect after the replacement. The replacement process is completed automatically, requiring no manual operation from the user, nor reloading or restarting the application, achieving a seamless switch from quick preview to high-quality display.

[0142] As can be seen, in this embodiment of the application, the user first sees a preview of the rapidly generated 3D scene on the second electronic device, and then it is automatically replaced after the high-quality second student scene data is generated. This avoids long waiting times, ensures the final display quality, and improves the user experience.

[0143] Figure 3 This is a second flowchart of a data processing method provided in some embodiments of this application. The method is applied to a second electronic device, such as... Figure 3 As shown, the method may include the following steps: step 301 and step 302.

[0144] In step 301, the first student scene data transmitted by the first electronic device is received, and a preview screen corresponding to the first student scene data is displayed based on the first student scene data.

[0145] In this embodiment, a second electronic device (such as a mobile terminal like a mobile phone or tablet) receives first student scene data transmitted by a first electronic device (such as a cloud server). The first student scene data is generated on the first electronic device side based on intermediate teacher scene data. It is generated relatively quickly (approximately tens of seconds), has a large data volume (approximately tens of MB), and has relatively low image quality, but it can reflect the overall structure and main color information of the three-dimensional scene.

[0146] In this embodiment, after receiving the first student scene data, the second electronic device can directly perform real-time rendering through a rasterization renderer and display the corresponding preview image on the screen. The renderer performs sampling scheduling based on the importance weight of each Gaussian point: Gaussian points with high importance weights contribute more to the rendered image and are processed first during rendering; Gaussian points with low importance weights contribute less to the rendered image and their sampling frequency can be appropriately reduced or some sampling can be skipped. Through this importance-based sampling rendering strategy, unnecessary computational overhead is reduced while ensuring rendering quality, enabling the mobile terminal to achieve smooth real-time rendering even with limited computing power. Users can interact with the preview image through the second electronic device, such as rotating the viewpoint and zooming, browsing the approximate shape and color of the 3D scene from different angles. The 3D scene effect can be seen immediately after the first student scene data transmission is completed, without waiting for several hours.

[0147] In step 302, the second student scene data transmitted by the first electronic device is received, the first student scene data is replaced based on the second student scene data, and the screen corresponding to the second student scene data is displayed; wherein, the first student scene data and the second student scene data are both scene description data describing the three-dimensional scene, the generation speed of the first student scene data is greater than the generation speed of the second student scene data, and the image quality of the second student scene data is better than that of the first student scene data.

[0148] In this embodiment, while the user previews the 3D scene using the first student scene data, the first electronic device continues to generate the second student scene data in the background. The generation speed of the second student scene data is slower (approximately several hours), but the image quality is higher and the data size is smaller (approximately a few MB), making it suitable for high-quality display on mobile terminals.

[0149] In this embodiment, after the second student scene data is generated, the first electronic device transmits it to the second electronic device. The second electronic device continuously listens for push notifications from the first electronic device. Upon receiving the second student scene data, it performs entropy decoding and parameter dequantization operations to restore the complete parameters of the second student scene data. Subsequently, it automatically replaces the currently displayed first student scene data with the second student scene data and displays a higher-quality 3D scene on the screen. The replacement process is completed automatically in the background, requiring no manual operation from the user, nor interruption of the current browsing or reloading of the application. This achieves a seamless switch from preview to high-quality display, providing the user with a higher-quality 3D scene display effect almost imperceptibly.

[0150] As can be seen from the above embodiments, in this embodiment, the user first sees a preview of the rapidly generated 3D scene on the second electronic device, and it is automatically replaced after the high-quality second student scene data is generated. This not only avoids the user waiting for a long time, but also ensures the final display quality and improves the user experience.

[0151] This application also provides a data processing system. Figure 4 This is an architecture diagram of a data processing system provided by some embodiments of this application. In this system, the first electronic device is a server, and the second electronic device is a mobile terminal, such as... Figure 4 As shown, the system consists of two parts: a server and a mobile terminal. The computationally intensive 3D reconstruction and distillation tasks are completed on the server side, while the mobile terminal is only responsible for lightweight decoding and real-time rendering.

[0152] The server includes the following modules: feedforward reconstruction module, coordinate system alignment module, parameter format conversion module, teacher post-optimization module, dual-path distillation module, and data transmission module.

[0153] The feedforward reconstruction module receives multi-viewpoint views uploaded by the user, samples them uniformly, and inputs them into a pre-trained feedforward reconstruction network to perform a single inference, outputting an initial 3D Gaussian point cloud and predicted camera poses for each view. This module corresponds to steps 101 to 102 mentioned above.

[0154] The coordinate system alignment module receives the predicted camera pose from the feedforward network and the camera pose from the calibration system, solves for the similarity transformation parameters through singular value decomposition, and aligns the Gaussian point cloud to the world coordinate system. This module corresponds to the coordinate system alignment process in step 103 above.

[0155] The parameter format conversion module sequentially performs spherical harmonic coefficient format conversion, scale domain conversion, opacity domain conversion, and quaternion sequence conversion, outputting standardized teacher scene data compatible with the distillation pipeline. This module corresponds to the parameter format conversion processing in step 103 above.

[0156] The teacher post-optimization module is used to perform short-round iterative optimization and importance removal on standardized teacher scenario data to improve reconstruction quality and control the Gaussian point size. This module corresponds to steps 1041 to 1042 mentioned above.

[0157] The dual-path distillation module comprises a fast preview distillation unit and a full distillation compression unit. Both use refined teacher scene data as a shared supervision signal and can run in parallel on the same or different computing nodes. The fast preview distillation unit completes first and triggers the preview scene data transmission; the full distillation compression unit triggers the deployment scene data transmission upon completion. This module corresponds to the dual-path generation process in step 1043 above.

[0158] The data transmission module manages the network transmission scheduling of scene data files, ensuring that preview-level scene data is transmitted first, and that deployment-level scene data is automatically pushed to the mobile terminal upon completion. This module corresponds to steps 105 to 106 mentioned above.

[0159] The mobile terminal includes the following modules: a data receiving and decoding module, a rasterization rendering engine, and a seamless data replacement and update module.

[0160] The data receiving and decoding module is used to receive compressed scene data files pushed by the server, perform entropy decoding and parameter dequantization, and restore the complete parameters of the student scene data.

[0161] The rasterization rendering engine is used for a differentiable rasterization renderer based on importance sampling. It schedules sampling according to the importance weight of each Gaussian point to achieve smooth real-time rendering under the limited computing power of mobile terminals.

[0162] The seamless data replacement and update module is used to monitor the deployment scenario data push on the server and complete the seamless switch from preview-level scenario data to deployment-level scenario data during the user's browsing process, without the user noticing any interruption.

[0163] As can be seen, the data processing system provided in this application concentrates the computationally intensive 3D reconstruction and distillation tasks on the server side, while the mobile terminal is only responsible for lightweight decoding and real-time rendering. This fully utilizes the computing power advantages of both the server and the mobile terminal, enabling the mobile terminal to run smoothly even with limited computing power. At the same time, the responsibilities of each module are clear and the interfaces are standardized. The feedforward reconstruction network, compression strategy, etc., can be independently replaced and upgraded, which facilitates subsequent iterative optimization and functional expansion. Furthermore, preview-level scene data is prioritized to reach the mobile terminal, and the dual-path distillation enables a progressive delivery from rapid preview to high-quality deployment, improving the user experience.

[0164] To facilitate understanding, a complete example is provided below to illustrate the data processing method provided in the embodiments of this application.

[0165] For example, user Xiao Wang used his mobile phone to take 100 photos around his living room and uploaded them to a cloud server.

[0166] The cloud server selects 16 images from 100 (ensuring uniform spatial distribution of viewpoints) and inputs them into the feedforward reconstruction network. The feedforward reconstruction network performs one forward inference, taking approximately 3 seconds, and outputs 1 million Gaussian points (each Gaussian point containing attributes such as position, color, size, rotation, and transparency) and the predicted camera poses for each of the 16 images. This stage does not involve iteration; the feedforward reconstruction network only performs the calculation once.

[0167] After the cloud server obtains the predicted camera pose, it combines it with the poses of the 16 calibrated cameras corresponding to the mobile phone and uses SVD to solve for the similarity transformation parameters, transforming 1 million Gaussian points from the predicted coordinate system of the feedforward reconstruction network to the coordinate system of the mobile phone. This stage does not involve iteration; SVD is solved directly.

[0168] The cloud server performs four parameter format conversions on each Gaussian point after transformation: converting the spherical harmonic coefficients from channel-first format to order-first format, converting the scaling parameter from the linear domain to the logarithmic domain, converting the opacity parameter from the probability domain to the log-odds domain, and converting the component order of the rotation quaternion from (x, y, z, w) to (w, x, y, z), resulting in intermediate teacher scene data compatible with subsequent processing. This stage does not involve iteration and is a batch data format conversion.

[0169] The cloud server iteratively optimizes the intermediate teacher scene data. For example, a total of 5000 iterations are performed, taking approximately 5 minutes. In each iteration, one image is randomly selected from 16 photos, and an image of that viewpoint is rendered using the current Gaussian point cloud. The difference is calculated by comparing it with the real photo, and the attribute parameters of all Gaussian points are fine-tuned based on the difference. Every 100 iterations, it checks whether there are any missing areas in the scene due to the viewpoint not being covered. If so, new Gaussian points are added. After 5000 iterations, the opacity contribution of each Gaussian point is calculated on all 16 photos. The data is sorted by contribution, and the top 80% (800,000 points) are retained, while the bottom 20% (200,000 points) are deleted, resulting in the optimized teacher scene data, approximately 80MB.

[0170] After the optimized teacher scenario data is generated, the cloud server simultaneously starts two processing pathways.

[0171] The first path (quick preview path): The cloud server downgrades the spherical harmonic coefficients of each Gaussian point in the optimized teacher scene data from third order (16 coefficients per channel) to first order (4 coefficients per channel), retaining only the basic colors and major perspective variations. Then, it calculates an importance score based on the opacity and scale parameters of each Gaussian point, sorts them from highest to lowest score, retains the top 50% of Gaussian points (approximately 400,000), and deletes the remaining 50%. Xiao Wang chooses to enable quality restoration, and the cloud server performs 500 short iterations of optimization on the cropped 400,000 Gaussian points. Each iteration selects one photo for rendering, comparison, and parameter fine-tuning, taking approximately tens of seconds. For example, the first path takes approximately 40 seconds to generate approximately 40MB of first student scene data, which is then transmitted to Xiao Wang's phone. A 3D preview of the living room is then displayed on Xiao Wang's phone, allowing him to freely rotate and zoom to view the approximate layout and colors of the living room.

[0172] The second pathway (high-quality deployment pathway): While the user previews the data, the cloud server performs complete distillation and compression on the optimized teacher scene data in the background. A total of 3000 iterations are performed. In the first 1000 iterations, each iteration selects one photo for rendering, comparison, and parameter adjustment, while low-contribution Gaussian points are removed at regular intervals. At the 1000th iteration, the cloud server constructs a lightweight neural network (MLP), using the positional encoding of Gaussian points as input and the appearance attributes of Gaussian points as output. The weight parameters of this network replace the independently stored attribute parameters of each Gaussian point, reducing the data size from approximately 50MB to approximately 6MB. In the remaining 2000 iterations, the weights of the MLP continue to be optimized, while low-contribution Gaussian points are continuously removed. At the 2900th iteration, the weight parameters of the MLP are clustered, a codebook is constructed, and the weight parameters are replaced with the codebook index, reducing the data size from approximately 6MB to approximately 2MB. After the iterations are completed, the parameters are entropy-encoded and compressed, ultimately generating a second student scene data of approximately several MB.

[0173] After the second student scene data was generated, the cloud server pushed it to Xiao Wang's mobile phone. The phone continuously listened to the cloud server's push notifications. Upon receiving the second student scene data, it automatically replaced the currently displayed first student scene data with the second student scene data, and the living room image displayed on the screen switched to a high-quality, detailed image. The replacement process was completed automatically in the background. Xiao Wang did not need to perform any operation or interrupt his browsing, and he obtained a higher-quality 3D scene display effect without even realizing it.

[0174] In summary, this application embodiment replaces the traditional scene-by-scene iterative optimization method with a feedforward reconstruction network to construct teacher scene data, reducing the teacher subject construction time from tens of minutes to seconds. Short-round post-optimization and importance elimination are used to compensate for quality gaps. Simultaneously, a coordinate system similarity transformation alignment and a multi-dimensional parameter format unification conversion mechanism eliminate system-level incompatibility issues between the feedforward reconstruction output and subsequent processing flows, achieving seamless integration between feedforward reconstruction and the distillation pipeline. Based on this, a dual-path parallel distillation architecture is proposed. The fast preview path generates tens of MB of preview-level student scene data within tens of seconds for users to browse instantly. The high-quality refining path performs complete distillation and applies multi-level compression such as networked parameter replacement and vector quantization to generate several MB of deployment-level student scene data. Both paths share the same teacher scene data and can be executed in parallel, achieving a progressive delivery experience from fast preview to high-quality deployment.

[0175] The data processing method provided in this application can be executed by a data processing device. This application uses an example of a data processing device executing the data processing method to illustrate the data processing device provided in this application.

[0176] Figure 5This is one of the structural block diagrams of a data processing apparatus provided in some embodiments of this application, applied to a first electronic device, such as... Figure 5 As shown, the data processing device 500 may include: an acquisition module 501, a reconstruction module 502, a conversion module 503, and a generation module 504. The acquisition module 501 is used to acquire N multi-angle views captured by the second electronic device; where N is an integer greater than 1. The reconstruction module 502 is used to input the N multi-angle views into a feedforward reconstruction network for 3D reconstruction processing to obtain the original teacher scene data and the predicted camera poses of the N multi-angle views respectively; wherein, the original teacher scene data includes attribute parameters of multiple Gaussian points; The conversion module 503 is used to perform coordinate system alignment processing on the original teacher scene data based on the predicted camera pose and the calibration camera pose corresponding to the second electronic device, and to perform parameter format conversion processing on the aligned teacher scene data to obtain intermediate teacher scene data that matches the coordinate system of the second electronic device. The generation module 504 is used to generate first student scene data and second student scene data based on the intermediate teacher scene data. The first student scene data is generated at a faster rate than the second student scene data, and the second student scene data has a better image quality than the first student scene data. The first student scene data is used for previewing on the second electronic device. The second student scene data is used to replace the first student scene data after it has been generated.

[0177] As can be seen from the above embodiments, in this embodiment, the first electronic device generates teacher scene data through a feedforward reconstruction network, replacing the scene-by-scene iterative optimization method in related technologies, shortening the construction time of teacher scene data and improving the construction efficiency of teacher scene data; on this basis, through coordinate alignment processing and parameter format conversion processing, the output of the feedforward reconstruction network can smoothly enter the subsequent process, truly transforming the speed advantage into usability; furthermore, by generating first student scene data and second student scene data, the first student scene data is generated quickly and can be quickly transmitted to the second electronic device for user preview, avoiding long waiting times for users and solving the problem that users need to wait a long time to see the 3D scene; the second student scene data automatically replaces the first student scene data after it is generated, ensuring preview speed while taking into account the final display quality, thus improving the user experience.

[0178] Optionally, as an embodiment, the transformation module 503 is specifically used to extract the predicted camera center coordinates of each of the N multi-angle views from the predicted camera pose to obtain N predicted camera center coordinates; extract the calibration camera center coordinates of each of the N multi-angle views from the calibration camera pose to obtain N calibration camera center coordinates; determine similarity transformation parameters based on the N predicted camera center coordinates and the N calibration camera center coordinates; and transform each Gaussian point in the original teacher scene data from the predicted coordinate system of the feedforward reconstruction network to the coordinate system of the second electronic device based on the similarity transformation parameters to obtain aligned teacher scene data.

[0179] Optionally, as an embodiment, the conversion module 503 is specifically used to perform at least one of the following parameter format conversions on each Gaussian point in the aligned teacher scene data: converting the spherical harmonic coefficients of the Gaussian point from channel-first format to order-first format, and truncating or filling with zero values ​​according to the target order; converting the scale parameter of the Gaussian point from the linear domain to the logarithmic domain; converting the opacity parameter of the Gaussian point from the probability domain to the log-odds domain; and converting the component order of the rotation quaternion of the Gaussian point from (x,y,z,w) to (w,x,y,z).

[0180] Optionally, as an embodiment, the generation module 504 is specifically used to perform a first iteration optimization process of T rounds on the intermediate teacher scene data, and to supplement Gaussian points in areas missing due to lack of view coverage during the first iteration optimization process of T rounds, to obtain filled teacher scene data; wherein, T is an integer greater than 1, and the first iteration optimization process includes: rendering the intermediate teacher scene data based on at least one of the N multi-angle views to obtain a first rendered image, adjusting the attribute parameters of the Gaussian points in the intermediate teacher scene data according to the difference between the first rendered image and the at least one view; rendering the filled teacher scene data on the N multi-angle views respectively, accumulating the opacity contribution of each Gaussian point on the N multi-angle views respectively, sorting them from high to low according to the contribution, retaining the top K Gaussian points, and removing the remaining Gaussian points to obtain optimized teacher scene data; wherein, K is an integer greater than 1; and generating first student scene data and second student scene data based on the optimized teacher scene data.

[0181] Optionally, as an embodiment, the generation module 504 is specifically used to reduce the spherical harmonic coefficients of each Gaussian point in the optimized teacher scene data to obtain reduced-order teacher scene data; calculate the importance score of each Gaussian point in the reduced-order teacher scene data according to the opacity parameter and scale parameter of the Gaussian point; sort the Gaussian points from high to low according to their importance scores, retain the top first preset percentage of Gaussian points, and remove the remaining Gaussian points to obtain lightweight teacher scene data; and determine the first student scene data based on the lightweight teacher scene data.

[0182] Optionally, as an embodiment, the generation module 504 is specifically used to determine the lightweight teacher scene data as the first student scene data; or, to perform Z rounds of second iteration optimization processing on the lightweight teacher scene data to obtain the first student scene data; wherein, Z is an integer greater than 1, and the second iteration optimization processing includes: rendering the lightweight teacher scene data based on at least one of the N multi-angle views to obtain a second rendered image, and adjusting the attribute parameters of the Gaussian points in the lightweight teacher scene data according to the difference between the second rendered image and the at least one view.

[0183] Optionally, as an embodiment, the generation module 504 is specifically used to perform P rounds of third iteration optimization processing on the optimized teacher scene data. During the P rounds of third iteration optimization processing, every i rounds, a Gaussian point culling judgment is performed based on the rendering opacity and maximum scale of the Gaussian points. Gaussian points whose rendering opacity is lower than a first threshold and whose maximum scale is lower than a second threshold are marked as to be culled. For Gaussian points marked as to be culled in j consecutive judgments, a culling operation is performed. Wherein, P, i, and j are all integers greater than 1, P > i ≥ j. The third iteration optimization processing includes: rendering the teacher scene data in the current iteration based on at least one of the N multi-angle views to obtain a third rendered image, and adjusting the Gaussian points in the teacher scene data in the current iteration according to the difference between the third rendered image and the at least one view. Attribute parameters; when the third iteration of optimization processing in round P is executed to round a, a lightweight neural network is constructed. The positional encoding in the Gaussian point attribute parameters is used as the input of the lightweight neural network, and the appearance attribute in the Gaussian point attribute parameters is used as the output of the lightweight neural network. The attribute parameters of each Gaussian point are replaced with the weight parameters of the lightweight neural network; where a is an integer greater than 1, P > a > i; when the third iteration of optimization processing in round P is executed to round b, the weight parameters of the lightweight neural network are clustered to obtain multiple cluster centers. A codebook is constructed based on the multiple cluster centers, and the weight parameters of the lightweight neural network are replaced with the codebook index; where b is an integer greater than 1, P > b > a; after the third iteration of optimization processing in round P is completed, the parameters after the codebook index replacement are subjected to entropy encoding compression processing to obtain the second student scene data.

[0184] Optionally, as an embodiment, the data processing apparatus 500 may further include: The transmission module is used to transmit the first student scene data to the second electronic device so as to display a preview screen corresponding to the first student scene data on the second electronic device; after the second student scene data is generated, the second student scene data is transmitted to the second electronic device to replace the first student scene data so as to display the screen corresponding to the second student scene data on the second electronic device.

[0185] Figure 6 This is a second structural block diagram of a data processing device provided in some embodiments of this application, applied to a second electronic device, such as... Figure 6 As shown, the data processing device 600 may include: a receiving module 601 and a display module 602; The receiving module 601 is used to receive the first student scene data transmitted by the first electronic device; The display module 602 is used to display a preview screen corresponding to the first student scene data based on the first student scene data; The receiving module 601 is also used to receive the second student scene data transmitted by the first electronic device; The display module 602 is further configured to replace the first student scene data based on the second student scene data and display the image corresponding to the second student scene data; wherein, the first student scene data and the second student scene data are both scene description data describing a three-dimensional scene, the generation speed of the first student scene data is greater than the generation speed of the second student scene data, and the image quality of the second student scene data is better than the image quality of the first student scene data.

[0186] As can be seen from the above embodiments, in this embodiment, the user first sees a preview of the rapidly generated 3D scene on the second electronic device, and it is automatically replaced after the high-quality second student scene data is generated. This not only avoids the user waiting for a long time, but also ensures the final display quality and improves the user experience.

[0187] The data processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0188] The data processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0189] The data processing device provided in this application embodiment can achieve the above-mentioned... Figure 1or Figure 3 To avoid repetition, the various processes implemented in the method embodiment shown will not be described again here.

[0190] Optionally, such as Figure 7 As shown, this application embodiment also provides an electronic device 700, including a processor 701 and a memory 702. The memory 702 stores a program or instructions that can run on the processor 701. When the program or instructions are executed by the processor 701, they implement the various steps of the above-described data processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0191] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0192] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of this application.

[0193] The electronic device 800 includes, but is not limited to, components such as: radio frequency unit 801, network module 802, audio output unit 803, input unit 804, sensor 805, display unit 806, user input unit 807, interface unit 808, memory 809, and processor 810.

[0194] Those skilled in the art will understand that the electronic device 800 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 810 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0195] In some embodiments, when the electronic device 800 is a first electronic device, the processor 810 is configured to acquire N multi-angle views captured by the second electronic device; wherein N is an integer greater than 1; input the N multi-angle views into a feedforward reconstruction network for 3D reconstruction processing to obtain original teacher scene data and the predicted camera poses of each of the N multi-angle views; wherein the original teacher scene data includes attribute parameters of multiple Gaussian points; based on the predicted camera poses and the calibration camera poses corresponding to the second electronic device, the original teacher scene data is subjected to coordinate system alignment processing, and the aligned teacher scene data is subjected to parameter format conversion processing to obtain intermediate teacher scene data that matches the coordinate system of the second electronic device; based on the intermediate teacher scene data, first student scene data and second student scene data are generated; wherein the generation speed of the first student scene data is greater than the generation speed of the second student scene data, and the image quality of the second student scene data is better than that of the first student scene data; the first student scene data is used for previewing on the second electronic device; the second student scene data is used to replace the first student scene data after generation.

[0196] As can be seen, in this embodiment, the first electronic device generates teacher scene data through a feedforward reconstruction network, replacing the scene-by-scene iterative optimization method in related technologies, shortening the construction time of teacher scene data and improving the construction efficiency of teacher scene data. On this basis, through coordinate alignment processing and parameter format conversion processing, the output of the feedforward reconstruction network can smoothly enter the subsequent process, truly transforming the speed advantage into usability. Furthermore, by generating first student scene data and second student scene data, the first student scene data is generated quickly and can be quickly transmitted to the second electronic device for user preview, avoiding long waiting times for users and solving the problem of users having to wait a long time to see the 3D scene. After the second student scene data is generated, it automatically replaces the first student scene data, ensuring preview speed while taking into account the final display quality, thus improving the user experience.

[0197] Optionally, as an embodiment, the processor 810 is specifically configured to extract the predicted camera center coordinates of each of the N multi-angle views from the predicted camera pose to obtain N predicted camera center coordinates; extract the calibration camera center coordinates of each of the N multi-angle views from the calibration camera pose to obtain N calibration camera center coordinates; determine similarity transformation parameters based on the N predicted camera center coordinates and the N calibration camera center coordinates; and transform each Gaussian point in the original teacher scene data from the predicted coordinate system of the feedforward reconstruction network to the coordinate system of the second electronic device based on the similarity transformation parameters to obtain aligned teacher scene data.

[0198] Optionally, as an embodiment, the processor 810 is specifically configured to perform at least one of the following parameter format conversions on each Gaussian point in the aligned teacher scene data: converting the spherical harmonic coefficients of the Gaussian point from channel-first format to order-first format, and truncating or filling with zero values ​​according to the target order; converting the scale parameter of the Gaussian point from the linear domain to the logarithmic domain; converting the opacity parameter of the Gaussian point from the probability domain to the log-odds domain; and converting the component order of the rotation quaternion of the Gaussian point from (x,y,z,w) to (w,x,y,z).

[0199] Optionally, as an embodiment, the processor 810 is specifically used to perform a first iteration optimization process of T rounds on the intermediate teacher scene data, and to supplement Gaussian points in areas missing due to lack of view coverage during the first iteration optimization process of T rounds, to obtain filled teacher scene data; wherein, T is an integer greater than 1, and the first iteration optimization process includes: rendering the intermediate teacher scene data based on at least one of the N multi-angle views to obtain a first rendered image, adjusting the attribute parameters of the Gaussian points in the intermediate teacher scene data according to the difference between the first rendered image and the at least one view; rendering the filled teacher scene data on the N multi-angle views respectively, accumulating the opacity contribution of each Gaussian point on the N multi-angle views respectively, sorting them from high to low according to the contribution, retaining the top K Gaussian points, and removing the remaining Gaussian points to obtain optimized teacher scene data; wherein, K is an integer greater than 1; and generating first student scene data and second student scene data based on the optimized teacher scene data.

[0200] Optionally, as an embodiment, the processor 810 is specifically used to perform order reduction processing on the spherical harmonic coefficients of each Gaussian point in the optimized teacher scene data to obtain reduced-order teacher scene data; calculate the importance score of each Gaussian point in the reduced-order teacher scene data according to the opacity parameter and scale parameter of the Gaussian point; sort the Gaussian points from high to low according to their importance scores, retain the top first preset percentage of Gaussian points, and remove the remaining Gaussian points to obtain lightweight teacher scene data; and determine the first student scene data based on the lightweight teacher scene data.

[0201] Optionally, as an embodiment, the processor 810 is specifically configured to determine the lightweight teacher scene data as the first student scene data; or, to perform a Z-round second iteration optimization process on the lightweight teacher scene data to obtain the first student scene data; wherein Z is an integer greater than 1, and the second iteration optimization process includes: rendering the lightweight teacher scene data based on at least one of the N multi-angle views to obtain a second rendered image, and adjusting the attribute parameters of the Gaussian points in the lightweight teacher scene data according to the difference between the second rendered image and the at least one view.

[0202] Optionally, as an embodiment, the processor 810 is specifically configured to perform P rounds of third iteration optimization processing on the optimized teacher scene data, and during the P rounds of third iteration optimization processing, every i rounds, a Gaussian point culling judgment is performed based on the rendering opacity and maximum scale of the Gaussian points, marking Gaussian points whose rendering opacity is lower than a first threshold and whose maximum scale is lower than a second threshold as to be culled; for Gaussian points marked as to be culled in j consecutive judgments, a culling operation is performed; wherein, P, i, and j are all integers greater than 1, P > i ≥ j; the third iteration optimization processing includes: rendering the teacher scene data in the current iteration based on at least one of the N multi-angle views to obtain a third rendered image, and adjusting the attributes of the Gaussian points in the teacher scene data in the current iteration according to the difference between the third rendered image and the at least one view. Parameters; When the third iteration of optimization in round P is executed to round a, a lightweight neural network is constructed. The positional encoding in the Gaussian point attribute parameters is used as the input of the lightweight neural network, and the appearance attribute in the Gaussian point attribute parameters is used as the output of the lightweight neural network. The attribute parameters of each Gaussian point are replaced with the weight parameters of the lightweight neural network; where a is an integer greater than 1, P > a > i; When the third iteration of optimization in round P is executed to round b, the weight parameters of the lightweight neural network are clustered to obtain multiple cluster centers. A codebook is constructed based on the multiple cluster centers, and the weight parameters of the lightweight neural network are replaced with the codebook index; where b is an integer greater than 1, P > b > a; After the third iteration of optimization in round P is completed, the parameters after the codebook index replacement are subjected to entropy encoding compression processing to obtain the second student scene data.

[0203] Optionally, as an embodiment, the processor 810 is further configured to transmit the first student scene data to the second electronic device to display a preview screen corresponding to the first student scene data on the second electronic device; after the second student scene data is generated, the processor transmits the second student scene data to the second electronic device to replace the first student scene data, thereby displaying a screen corresponding to the second student scene data on the second electronic device.

[0204] In some embodiments, when the electronic device 800 is a second electronic device, the processor 810 is configured to receive first student scenario data transmitted by the first electronic device. Display unit 806 is used to display a preview screen corresponding to the first student scene data based on the first student scene data; The processor 810 is also used to receive second student scene data transmitted by the first electronic device; The display unit 806 is further configured to replace the first student scene data based on the second student scene data and display the image corresponding to the second student scene data; wherein the first student scene data and the second student scene data are both scene description data describing a three-dimensional scene, the generation speed of the first student scene data is greater than the generation speed of the second student scene data, and the image quality of the second student scene data is better than the image quality of the first student scene data.

[0205] As can be seen, in this embodiment of the application, the user first sees a preview of the rapidly generated 3D scene on the second electronic device, and then it is automatically replaced after the high-quality second student scene data is generated. This not only avoids the user waiting for a long time, but also ensures the final display quality and improves the user experience.

[0206] It should be understood that, in this embodiment, the input unit 804 may include a graphics processing unit (GPU) 8041 and a microphone 8042. The GPU 8041 processes image data of still images or videos obtained by an image capturing device (such as a camera) in video capture mode or image capture mode. The display unit 806 may include a display panel 8061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 may include a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0207] The memory 809 can be used to store software programs and various data. The memory 809 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 809 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM). The memory 809 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0208] Processor 810 may include one or more processing units; optionally, processor 810 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 810.

[0209] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described data processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0210] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0211] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above data processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0212] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0213] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the data processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0214] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0215] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0216] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data processing method, characterized in that, Applied to a first electronic device, the method includes: Obtain N multi-angle views captured by the second electronic device; where N is an integer greater than 1; The N multi-angle views are input into a feedforward reconstruction network for 3D reconstruction processing to obtain the original teacher scene data and the predicted camera poses of the N multi-angle views respectively; wherein, the original teacher scene data includes attribute parameters of multiple Gaussian points; Based on the predicted camera pose and the calibrated camera pose corresponding to the second electronic device, the original teacher scene data is subjected to coordinate system alignment processing, and the aligned teacher scene data is subjected to parameter format conversion processing to obtain intermediate teacher scene data that matches the coordinate system of the second electronic device. Based on the intermediate teacher scenario data, first student scenario data and second student scenario data are generated; The first student scene data is generated at a faster rate than the second student scene data, and the second student scene data has a better image quality than the first student scene data. The first student scene data is used for previewing on the second electronic device. The second student scene data is used to replace the first student scene data after it has been generated.

2. The method according to claim 1, characterized in that, The step of aligning the original teacher scene data with a coordinate system based on the predicted camera pose and the calibrated camera pose corresponding to the second electronic device includes: Extract the predicted camera center coordinates of each of the N multi-angle views from the predicted camera pose to obtain N predicted camera center coordinates; Extract the center coordinates of the calibration camera from each of the N multi-angle views from the calibration camera pose to obtain N center coordinates of the calibration camera; Based on the N predicted camera center coordinates and the N calibrated camera center coordinates, determine the similarity transformation parameters; Based on the similarity transformation parameters, each Gaussian point in the original teacher scene data is transformed from the prediction coordinate system of the feedforward reconstruction network to the coordinate system of the second electronic device to obtain aligned teacher scene data.

3. The method according to claim 1, characterized in that, The parameter format conversion process for the aligned teacher scene data includes: For each Gaussian point in the aligned teacher scene data, perform at least one of the following parameter format conversions: The spherical harmonic coefficients of the Gaussian points are converted from channel-first format to order-first format, and then truncated or filled with zeros according to the target order. The scaling parameter of the Gaussian point is transformed from the linear domain to the logarithmic domain; Transform the opacity parameter of the Gaussian point from the probability domain to the log-odds domain; The order of the components of the rotation quaternion of the Gaussian point is transformed from (x,y,z,w) to (w,x,y,z).

4. The method according to claim 1, characterized in that, The process of generating first student scenario data and second student scenario data based on the intermediate teacher scenario data includes: The intermediate teacher scene data is subjected to a first iteration of optimization processing for T rounds, and Gaussian points in areas missing due to lack of view coverage are added during the first iteration of optimization processing for T rounds, resulting in filled teacher scene data; wherein, T is an integer greater than 1, the first iteration optimization processing includes: rendering the intermediate teacher scene data based on at least one of the N multi-angle views to obtain a first rendered image, and adjusting the attribute parameters of the Gaussian points in the intermediate teacher scene data according to the difference between the first rendered image and the at least one view; The filled teacher scene data is rendered on the N multi-angle views respectively. The opacity contribution of each Gaussian point on the N multi-angle views is accumulated. The points are sorted from high to low according to the contribution, the top K Gaussian points are retained, and the remaining Gaussian points are removed to obtain the optimized teacher scene data. Here, K is an integer greater than 1. Based on the optimized teacher scenario data, first student scenario data and second student scenario data are generated.

5. The method according to claim 4, characterized in that, The process of generating first student scenario data based on the optimized teacher scenario data includes: The spherical harmonic coefficients of each Gaussian point in the optimized teacher scene data are reduced in order to obtain the reduced-order teacher scene data. The importance score of each Gaussian point in the reduced-order teacher scene data is calculated based on the opacity parameter and scale parameter of the Gaussian point. Sort the data by importance score from high to low, retain the top Gaussian points with the highest percentage, and remove the rest to obtain lightweight teacher scenario data. Based on the lightweight teacher scenario data, the first student scenario data is determined.

6. The method according to claim 5, characterized in that, The determination of the first student scenario data based on the lightweight teacher scenario data includes: The lightweight teacher scenario data is designated as the first student scenario data; or, The lightweight teacher scenario data is subjected to a second iteration of Z rounds of optimization to obtain the first student scenario data; Where Z is an integer greater than 1, the second iterative optimization process includes: rendering the lightweight teacher scene data based on at least one of the N multi-angle views to obtain a second rendered image, and adjusting the attribute parameters of the Gaussian points in the lightweight teacher scene data according to the difference between the second rendered image and the at least one view.

7. The method according to claim 4, characterized in that, The process of generating second student scenario data based on the optimized teacher scenario data includes: The optimized teacher scene data undergoes a P-round third iteration optimization process. During this P-round third iteration optimization process, every i rounds, a Gaussian point culling judgment is performed based on the rendering opacity and maximum scale of the Gaussian points. Gaussian points whose rendering opacity is lower than a first threshold and whose maximum scale is lower than a second threshold are marked as to be culled. For Gaussian points marked as to be culled in j consecutive judgments, a culling operation is performed. Wherein, P, i, and j are all integers greater than 1, and P > i ≥ j. The third iteration optimization process includes: rendering the teacher scene data in the current iteration based on at least one of the N multi-angle views to obtain a third rendered image, and adjusting the attribute parameters of the Gaussian points in the teacher scene data in the current iteration according to the difference between the third rendered image and the at least one view. When the third iteration of the P-round optimization process reaches the a-th round, a lightweight neural network is constructed. The positional encoding in the Gaussian point attribute parameters is used as the input of the lightweight neural network, and the appearance attribute in the Gaussian point attribute parameters is used as the output of the lightweight neural network. The attribute parameters of each Gaussian point are replaced with the weight parameters of the lightweight neural network; where a is an integer greater than 1, P > a > i. When the third iteration of the P-round optimization process reaches the b-th round, the weight parameters of the lightweight neural network are clustered to obtain multiple cluster centers. A codebook is constructed based on the multiple cluster centers, and the weight parameters of the lightweight neural network are replaced with the codebook index; where b is an integer greater than 1, and P > b > a. After the third iteration of optimization in round P is completed, the parameters after codebook index replacement are subjected to entropy encoding compression to obtain the second student scenario data.

8. The method according to claim 1, characterized in that, After generating the first student scenario data and the second student scenario data based on the intermediate teacher scenario data, the method further includes: The first student scene data is transmitted to the second electronic device to display a preview screen corresponding to the first student scene data on the second electronic device; After the second student scene data is generated, the second student scene data is transmitted to the second electronic device to replace the first student scene data, so that the screen corresponding to the second student scene data is displayed on the second electronic device.

9. A data processing method, characterized in that, Applied to a second electronic device, the method includes: Receive the first student scene data transmitted by the first electronic device; Based on the first student scene data, display the preview screen corresponding to the first student scene data; Receive the second student scenario data transmitted by the first electronic device; Based on the second student scene data, replace the first student scene data and display the screen corresponding to the second student scene data; The first student scene data and the second student scene data are both scene description data describing a 3D scene. The generation speed of the first student scene data is greater than that of the second student scene data, and the image quality of the second student scene data is better than that of the first student scene data.

10. A data processing apparatus, characterized in that, Applied to a first electronic device, the device includes: The acquisition module is used to acquire N multi-angle views captured by the second electronic device; where N is an integer greater than 1. The reconstruction module is used to input the N multi-angle views into the feedforward reconstruction network for 3D reconstruction processing to obtain the original teacher scene data and the predicted camera poses of the N multi-angle views respectively; wherein, the original teacher scene data includes attribute parameters of multiple Gaussian points; The conversion module is used to perform coordinate system alignment processing on the original teacher scene data based on the predicted camera pose and the calibration camera pose corresponding to the second electronic device, and to perform parameter format conversion processing on the aligned teacher scene data to obtain intermediate teacher scene data that matches the coordinate system of the second electronic device. The generation module is used to generate first student scenario data and second student scenario data based on the intermediate teacher scenario data; The first student scene data is generated at a faster rate than the second student scene data, and the second student scene data has a better image quality than the first student scene data. The first student scene data is used for previewing on the second electronic device. The second student scene data is used to replace the first student scene data after it has been generated.

Images