Three-dimensional reconstruction method and device, electronic equipment and storage medium

Abstract

The invention relates to a three-dimensional reconstruction method and device, electronic equipment and a storage medium. The method comprises the following steps: in the process of performing three-dimensional reconstruction on a target model, performing model compression and sparse processing on an initially established initial model to obtain a candidate model; performing knowledge distillation on the candidate model through the first teacher model, performing self-distillation on the candidate model and / or performing cross-modal knowledge migration on the candidate model to obtain an alternative model; and according to the real-time load of the target device, the data quality of the three-dimensional reconstruction task and / or the task target demand, carrying out adaptive adjustment on the training strategy of the alternative model, and training to obtain a target model. Therefore, by adaptively adjusting the training steps and algorithms of the alternative models, the training process can be accurately matched with the actual application scene; the obtained target model not only can meet the performance requirement of three-dimensional reconstruction, but also can flexibly adapt to different equipment resources and task scenes, and achieves the optimal balance of precision, efficiency and practicability.

Description

[0001] This application relates to the field of data processing technology, and in particular to a three-dimensional reconstruction method, apparatus, electronic device, and storage medium. Background Technology

[0002] 3D reconstruction technology has become a core technology supporting innovation in fields such as Augmented Reality (AR), Virtual Reality (VR), robot navigation, intelligent manufacturing, and the digitization of cultural heritage, with extremely broad application prospects. However, current mainstream 3D reconstruction algorithms still face significant resource adaptation bottlenecks: whether it is traditional geometry-based bundle adjustment and multi-view stereo matching methods, or technologies that have emerged in recent years based on deep learning such as neural radiation fields and point cloud processing networks, all require high-performance CPUs, GPUs, and large-capacity memory to provide computing power. This characteristic makes achieving real-time, high-precision 3D reconstruction on resource-constrained terminal devices (including mobile devices such as smartphones and tablets, small embedded systems such as consumer drones and home robots, as well as edge artificial intelligence (AI) devices and wearable devices) a huge challenge.

[0003] To adapt to low-resource environments, existing technologies typically adopt compromise solutions such as sacrificing reconstruction quality, reducing real-time processing, or relying on cloud computing. However, these solutions all have unavoidable drawbacks: sacrificing reconstruction accuracy directly limits its application in scenarios with high detail requirements, such as AR interaction and robot navigation; reducing real-time performance cannot meet the needs of users for immediate feedback and dynamic device response; relying on cloud computing not only affects processing efficiency due to network latency, but also poses the risk of privacy leaks during data transmission, and is completely ineffective in offline scenarios. Summary of the Invention

[0004] Therefore, it is necessary to provide a three-dimensional reconstruction method, apparatus, electronic device, and storage medium to address the aforementioned technical problems.

[0005] In a first aspect, this application provides a three-dimensional reconstruction method, the method comprising: During the 3D reconstruction of the target model, the initial model is compressed and sparsified to obtain candidate models. The candidate models are obtained by performing knowledge distillation on the candidate models using the first teacher model, self-distillation on the candidate models, and / or cross-modal knowledge transfer on the candidate models. The training strategy of the candidate models is adaptively adjusted based on the real-time load of the target device, the data quality of the 3D reconstruction task, and / or the task objective requirements, and the target model is trained to obtain the target model.

[0006] In one embodiment, the model compression and sparsification processing of the initially established model includes at least one of the following: The first method involves quantizing the floating-point parameters and / or activation values ​​of the initial model during training to obtain a low-bit-width fixed-point number. The second method is to prune the initial model according to the importance of its parameters, thus obtaining the pruned initial model. The third method involves performing a low-rank decomposition on the high-rank matrix in the initial model to obtain a combination of multiple smaller matrices. The fourth method is to match the corresponding model network architecture based on the hardware limitations of the target device and the task characteristics of the 3D reconstruction task.

[0007] In one embodiment, knowledge distillation is performed on the candidate model using a first teacher model to obtain alternative models, including: Transferable target knowledge is extracted from the first teacher model; wherein, the target knowledge includes intermediate layer knowledge and / or output layer knowledge; the intermediate layer knowledge includes intermediate feature maps and / or attention distributions; the output layer knowledge includes output logic, output probabilities, and / or pose estimation results; The candidate model is trained using the target knowledge to obtain the alternative model.

[0008] In one embodiment, the candidate model is self-distilled to obtain the alternative model, including: The shallow features and deep features of the candidate model are aligned using a first loss function to obtain the alternative model; And / or, When the candidate model includes multiple related subtasks, knowledge distillation is performed between the task branches corresponding to each subtask to obtain the alternative model.

[0009] In one embodiment, cross-modal knowledge transfer is performed using the candidate model to obtain alternative models, including: Acquire high-modality sample data; The second teacher model was trained using the high-modal sample data, and geometric priors were extracted. The candidate model is obtained by forcing the first output of the candidate model to be consistent with the second output of the second teacher model through a second loss function; wherein the second loss function includes soft target loss, hard target loss and / or feature alignment loss.

[0010] In one embodiment, the step of adaptively adjusting the training steps or training algorithm of the candidate models based on the real-time load of the target device, the data quality of the 3D reconstruction task, and / or the task objective requirements, to train the target model, includes: If the real-time load is less than or equal to the first threshold and / or the data quality is less than the second threshold, the training strategy of the candidate model is adaptively adjusted according to the task objective requirements and in conjunction with the basic rule engine to obtain the target model. If the real-time load is greater than the first threshold and / or the data quality is greater than or equal to the second threshold, the training strategy of the candidate model is adaptively adjusted according to the task objective requirements and in conjunction with a lightweight learning model to obtain the target model.

[0011] Secondly, this application also provides a three-dimensional reconstruction apparatus, the apparatus comprising: The sparse module is used to compress and sparsify the initial model during the 3D reconstruction of the target model to obtain candidate models. The distillation module is used to perform knowledge distillation on the candidate model through the first teacher model, self-distillation on the candidate model, and / or cross-modal knowledge transfer on the candidate model to obtain alternative models. The adjustment module is used to adaptively adjust the training strategy of the candidate models according to the real-time load of the target device, the data quality of the 3D reconstruction task and / or the task objective requirements, so as to train the target model.

[0012] Thirdly, this application also provides an electronic device, including a processor and a memory; wherein the memory is used to store a computer program; and the processor is configured to, when executing the computer program, implement the steps of the method described in any embodiment of this application.

[0013] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any embodiment of this application.

[0014] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the method described in any embodiment of this application.

[0015] In the aforementioned 3D reconstruction method, the initial model is first compressed and sparsified to obtain candidate models during the 3D reconstruction process of the target model. This effectively eliminates redundant parameters and computational modules, significantly reducing model memory usage and inference computation, laying the foundation for subsequent deployment in resource-constrained scenarios such as edge devices. Then, alternative models are obtained through multi-dimensional optimization via first-teacher model distillation, self-distillation, and / or cross-modal knowledge transfer. This fully compensates for potential performance losses caused by compression and sparsity. For example, the teacher model can transfer high-precision reconstruction experience, self-distillation can mine the potential of internal model features, and cross-modal knowledge transfer can supplement the geometric prior information required for 3D reconstruction, comprehensively ensuring the accuracy and robustness of model reconstruction. Finally, the training steps and algorithms of the alternative models are adaptively adjusted according to the real-time load of the target device, the data quality of the 3D reconstruction task, and the task objective requirements. This allows the training process to accurately match the actual application scenario. For example, when the device load is high, computational efficiency is prioritized; when the task requires high-detail reconstruction, accuracy improvement is emphasized. The final target model not only meets the performance requirements of 3D reconstruction but also flexibly adapts to different device resources and task scenarios, achieving the optimal balance between accuracy, efficiency, and practicality. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a three-dimensional reconstruction method according to an exemplary embodiment; Figure 2 This is a flowchart illustrating a three-dimensional reconstruction method according to an exemplary embodiment; Figure 3 This is a structural block diagram of a three-dimensional reconstruction apparatus according to an exemplary embodiment; Figure 4 This is an internal structural diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] The terms "first," "second," and "third" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0019] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. Those skilled in the art will understand, explicitly and implicitly, that the embodiments described herein can be combined with other embodiments.

[0020] In some embodiments, the 3D reconstruction method provided in this application can be applied to electronic devices or cloud servers. The electronic device can be any mobile terminal or fixed terminal. The terminal can be a device that provides voice and / or data connectivity to the user. For example, the terminal can be an IoT terminal, such as a sensor device, a mobile phone or so-called "cellular" phone, and a computer with an IoT terminal; for example, it can be a fixed, portable, pocket-sized, handheld, or computer-embedded device. The cloud server can be any virtualized computing resource or physical server cluster. The server can be a platform that provides on-demand, scalable computing, storage, networking, and application services to the user.

[0021] In some embodiments, such as Figure 1 As shown, a three-dimensional reconstruction method is provided, the method comprising the following steps: S101, during the three-dimensional reconstruction of the target model, the initial model is compressed and sparsified to obtain a candidate model.

[0022] In this embodiment of the application, the initial model can be a deep learning neural network model.

[0023] In this application embodiment, model compression can refer to reducing the model size through structural optimization or representation simplification. Model compression may include, but is not limited to, at least one of the following: quantizing the model, performing low-rank decomposition on high-rank matrices in the model, and allowing lightweight models to inherit the ability of large models through knowledge distillation.

[0024] In this embodiment, sparsification is implemented by preserving key information and removing redundancy to introduce sparsity. Sparsification can also instruct the model to be pruned.

[0025] S102, the candidate model is obtained by performing knowledge distillation on the candidate model through the first teacher model, self-distillation on the candidate model, and / or cross-modal knowledge transfer on the candidate model.

[0026] In this embodiment, knowledge distillation refers to a technique that uses a "teacher-student" framework to transfer knowledge (such as output distribution and intermediate features) from a complex, high-resource teacher model to a lightweight, low-resource student model, thereby approximating the performance of a high-performance model with low resource cost.

[0027] In this embodiment, self-distillation refers to a technique in which a single model mines its own information through internal knowledge flow (such as deep-to-shallow or multi-branch inter-propagation) to improve generalization and robustness.

[0028] In this application embodiment, cross-modal knowledge transfer refers to the technique of transferring unique knowledge from high-modal data (such as lidar point clouds) to low-modal models (such as RGB-only image models) to compensate for the deficiencies of low-modal information.

[0029] S103, based on the real-time load of the target device, the data quality of the 3D reconstruction task, and / or the task objective requirements, the training strategy of the alternative models is adaptively adjusted to train the target model.

[0030] In some embodiments, such as Figure 2 As shown, the step of adaptively adjusting the training strategy of the candidate models based on the real-time load of the target device, the data quality of the 3D reconstruction task, and / or the task objective requirements to train the target model includes: S1031, if the real-time load is less than or equal to the first threshold and / or the data quality is less than the second threshold, the training strategy of the candidate model is adaptively adjusted according to the task objective requirements and in combination with the basic rule engine to obtain the target model; S1032, when the real-time load is greater than the first threshold and / or the data quality is greater than or equal to the second threshold, the training strategy of the candidate model is adaptively adjusted according to the task objective requirements and in combination with the lightweight learning model to obtain the target model.

[0031] For example, if severe motion blur is detected (gradient change rate > 0.3), it can be determined that the data quality is less than the second threshold, and the electronic device is under low to medium load, less than the first threshold (e.g., CPU < 60%). Then, the basic rule engine determines that the current training is insufficient in capturing dynamic features, and decides to adjust to "increase the sampling density of the optical flow tracking module (from 100 points per frame to 200 points per frame) and increase the number of pose optimization iterations (from 5 to 8 times)" to strengthen the learning of motion trajectories. If a low-texture scene (e.g., feature point density < 5) and reprojection error > 2 pixels are detected, the scene is relatively simple, and it can be determined that the data quality is less than the second threshold. Then, the basic rule engine determines that the feature matching stability is insufficient, and decides to switch to enabling the pre-trained texture enhancement branch (lightweight style transfer module), while reducing the keyframe insertion threshold (e.g., from displacement 0.5 meters to 0.3 meters) to improve training robustness by supplementing scene feature redundancy.

[0032] For example, when an electronic device is dealing with complex and overlapping scenarios, such as simultaneously experiencing high load exceeding the first threshold (CPU > 80%), high texture (feature point density > 15), and a task objective requiring real-time reconstruction (latency requirement < 30ms), the model will combine historical data to learn that simplifying feature channels in high-texture scenarios does not affect accuracy. The decision is to "reduc the number of convolutional layer channels (from 256 to 128) and disable non-critical edge detail optimization modules," thereby compressing the training time per frame from 40ms to 25ms while ensuring the core reconstruction accuracy.

[0033] In some embodiments, the decision is also dynamically weighted according to the priority of the task objective requirement: if the task objective requirement is real-time preview on a mobile device, the real-time load of the electronic device is given a higher weight than the accuracy index, and the computational load reduction strategy is triggered first; if the task objective requirement is high-precision offline modeling, the reprojection error and feature alignment are given higher weights, allowing the training complexity to be increased within the controllable range of device load.

[0034] In some embodiments, the electronic device may determine the real-time load based on the target device's CPU utilization, GPU utilization, and NPU utilization; determine the feature matching success rate based on the ratio of the number of feature points successfully matched with the previous frame in the input image data to the number of feature points detected or to be matched in the current frame; determine the reprojection error based on the error between the predicted position of the known 3D feature points projected onto the current image plane and the actual observed 2D position; and determine the system performance of the target device based on the real-time load, feature matching success rate, and / or reprojection error.

[0035] In some embodiments, the electronic device can perform Laplacian filtering on the input image data, calculate the Laplacian variance, and perform motion blur analysis on the image data based on the Laplacian variance; wherein, the smaller the variance value, the more blurred the image data; calculate the horizontal and vertical gradients using operators such as Sobel, calculate the mean squared gradient based on the horizontal and vertical gradients, and determine the texture richness based on the mean squared gradient; wherein, the larger the mean squared gradient, the richer the texture of the image data; calculate the average brightness change rate based on the brightness of the image data at at least two time points, and perform illumination change analysis based on the average brightness change rate; wherein, the average brightness change rate indicates more stable illumination; divide the input image data into small grids, count the standard deviation of the number of feature points in the grids, and perform feature point distribution analysis based on the standard deviation; wherein, the smaller the standard deviation, the more uniform the feature point distribution. Based on motion blur analysis, texture richness analysis, illumination change analysis, and / or feature point distribution analysis, the data quality of the 3D reconstruction task is determined.

[0036] For example, the electronic device can also determine whether the image data is blurred based on motion blur analysis. If it is not blurred, it can determine whether the system performance is sufficient. If sufficient, the feature matching method can be switched from descriptor matching to the KLT (Kanade-Lucas-Tomasi) optical flow tracking method; if insufficient, the processing frame rate can be reduced. In the case of blurriness, it can further determine whether there is missing texture or a decrease in feature matching success rate. If so, the keyframe insertion threshold can be adaptively reduced or new keyframes can be forcibly inserted to enhance map redundancy. Based on the above adjustments to the training strategy, execution instructions are generated to train the target model.

[0037] In some embodiments, the electronic device can determine the decision benefit based on the average change in data quality before and after the training strategy switch (e.g., the difference between the average data quality of the 5 frames after the switch and the average data quality of the 5 frames before the switch) and the average change in system performance; determine whether the adjustment result of the current training strategy is positive based on the decision benefit; and optimize the subsequent training strategy adjustment based on the adjustment result.

[0038] In the aforementioned 3D reconstruction method, the initial model is first compressed and sparsified to obtain candidate models during the 3D reconstruction process of the target model. This effectively eliminates redundant parameters and computational modules, significantly reducing model memory usage and inference computation, laying the foundation for subsequent deployment in resource-constrained scenarios such as edge devices. Then, alternative models are obtained through multi-dimensional optimization via first-teacher model distillation, self-distillation, and / or cross-modal knowledge transfer. This fully compensates for potential performance losses caused by compression and sparsity. For example, the teacher model can transfer high-precision reconstruction experience, self-distillation can mine the potential of internal model features, and cross-modal knowledge transfer can supplement the geometric prior information required for 3D reconstruction, comprehensively ensuring the accuracy and robustness of model reconstruction. Finally, the training steps and algorithms of the alternative models are adaptively adjusted according to the real-time load of the target device, the data quality of the 3D reconstruction task, and the task objective requirements. This allows the training process to accurately match the actual application scenario. For example, when the device load is high, computational efficiency is prioritized; when the task requires high-detail reconstruction, accuracy improvement is emphasized. The final target model not only meets the performance requirements of 3D reconstruction but also flexibly adapts to different device resources and task scenarios, achieving the optimal balance between accuracy, efficiency, and practicality.

[0039] In some embodiments, the model compression and sparsification processing of the initially established model includes at least one of the following: The first method involves quantizing the floating-point parameters and / or activation values ​​of the initial model during training to obtain a low-bit-width fixed-point number. The second method is to prune the initial model according to the importance of its parameters, thus obtaining the pruned initial model. The third method involves performing a low-rank decomposition on the high-rank matrix in the initial model to obtain a combination of multiple smaller matrices. The fourth method is to match the corresponding model network architecture based on the hardware limitations of the target device and the task characteristics of the 3D reconstruction task.

[0040] In some embodiments, electronic devices can integrate quantization operations into the training process during the initial model training process, performing low-bit-width conversion between the model's floating-point parameters (such as FP32) and the activation values ​​in the calculation process, quantizing them into fixed-point numbers such as INT8 and INT4; by reducing the parameter storage volume through quantization (for example, INT8 only occupies 1 / 4 of the storage of FP32), and because integer operations have lower hardware computing power requirements, the computational overhead during subsequent inference of the model can be significantly reduced, laying a lightweight foundation for the initial model.

[0041] Furthermore, considering that direct quantization may introduce errors due to rounding and truncation of floating-point values, thereby affecting model performance (such as the depth estimation accuracy and feature extraction robustness of 3D reconstruction), Quantization-Aware Training (QAT) is used. During the forward propagation process in the training phase, simulated quantization-dequantization operations are inserted into key layers of the model (such as convolutional layers and fully connected layers). This allows the model to actively adapt to the error characteristics brought about by quantization while learning the core knowledge of the task (such as the geometric features of the 3D scene and the matching rules of multiple views). For example, the parameter distribution is adjusted to better fit the fixed-point representation range, reducing information loss caused by activation values ​​exceeding the low bit width boundary.

[0042] In some embodiments, the electronic device can use an initial model that has been trained and meets performance standards as a basis to score and rank the importance of all parameters (including weights, filters, channels, etc.) in the model based on one or more criteria (such as absolute value of parameters, contribution, etc.); according to a preset goal (such as reducing the number of parameters by 50% or reducing FLOPs by 60%), remove the low-ranked non-critical parameters; since pruning can easily lead to a decrease in accuracy, the pruned sparse model needs to be fine-tuned, which can be done by iteratively training several rounds with a small learning rate, allowing the remaining parameters to further learn and compensate for the function of the pruned parameters; repeat the process of importance scoring-pruning-fine-tuning to gradually approach the target model size and inference speed while maintaining the stability of core performance.

[0043] In some embodiments, electronic devices can perform low-rank decomposition operations on high-rank matrices that carry key computations in the model (such as the weight matrix of a volumetric rendering MLP in a 3D related network, or the feature tensor of a point cloud feature extractor), decomposing such large high-rank structures into a product combination of multiple low-rank small matrices.

[0044] In this application embodiment, hardware limitations refer to the hardware resource constraints of the electronic device, such as limited CPU / GPU computing power and small memory.

[0045] In some embodiments, the task characteristics of a 3D reconstruction task include the task requirements of each subtask. For example, the task requirement for the depth estimation subtask may be to accurately capture geometric cues such as texture variations and object edges in the image; the task requirement for the dense reconstruction subtask may be that the network outputs high-resolution 3D details, such as the bumps and dents of the object surface.

[0046] In some embodiments, electronic devices can use Neural Architecture Search (NAS) to customize architectural details (such as prioritizing hardware-friendly, efficient convolutional modules) based on the hardware characteristics of the target device (such as supported computing types and memory limits), while optimizing the network structure for the core requirements of 3D reconstruction tasks (such as dense reconstruction design focusing on feature modules that preserve details, and pose tracking design focusing on lightweight units that are temporally correlated), and automatically search for and design the optimal lightweight network architecture.

[0047] In this embodiment, through the synergy of quantization operations and quantization-aware training, the initial model can undergo lightweight transformation during the training phase while retaining its original performance and accuracy to the greatest extent, providing a high-quality foundation model for subsequent optimization stages such as model compression, sparsification, and knowledge distillation. Alternatively, by precisely selecting and eliminating redundant parameters, a more streamlined pruned initial model can be obtained, reducing the model size while preserving performance to the greatest extent. Alternatively, by replacing the original high-rank matrix with a small matrix product, the amount of parameter storage can be significantly reduced, while the computational complexity of matrix multiplication can be lowered, effectively reducing the memory footprint and inference time of the initial model, and adapting to multi-resolution meshes, feature planes, and other data representations in 3D scenes. Alternatively, by clearly defining task requirements and hardware constraints, a reasonable parts library can be built, and then an efficient search strategy can be used for automatic trial and error to ultimately generate a lightweight architecture that is both suitable for 3D reconstruction tasks (ensuring accuracy) and can run smoothly on edge hardware (controlling resource consumption).

[0048] In some embodiments, knowledge distillation is performed on the candidate models using a first teacher model to obtain alternative models, including: Transferable target knowledge is extracted from the first teacher model; wherein, the target knowledge includes intermediate layer knowledge and / or output layer knowledge; the intermediate layer knowledge includes intermediate feature maps and / or attention distributions; the output layer knowledge includes output logic, output probabilities, and / or pose estimation results; The candidate model is trained using the target knowledge to obtain the alternative model.

[0049] In some embodiments, the electronic device can utilize the capabilities of a high-performance, high-resource-consumption first teacher model, such as a large offline 3D reconstruction model or a large deep learning network, to first extract transferable target knowledge from it. For example, intermediate layer knowledge includes intermediate feature maps and attention distributions, while output layer knowledge includes output logic, output probabilities, and pose estimation results. The target knowledge guides the training of a lightweight candidate model (the target is deployed on a low-resource electronic device), ultimately yielding a candidate model.

[0050] In this embodiment, knowledge distillation allows lightweight candidate models to efficiently inherit the core capabilities of the high-performance, high-resource-consumption first teacher model, effectively compensating for the shortcomings of candidate models, such as low accuracy and weak robustness, that may exist due to the lightweight structure. The resulting alternative model can maintain its lightweight characteristics to adapt to low-resource device deployment, while also possessing high performance close to that of the teacher model, perfectly balancing the accuracy requirements of tasks such as 3D reconstruction with device resource constraints.

[0051] In some embodiments, the candidate model is self-distilled to obtain the alternative model, including: The shallow features and deep features of the candidate model are aligned using a first loss function to obtain the alternative model; And / or, When the candidate model includes multiple related subtasks, knowledge distillation is performed between the task branches corresponding to each subtask to obtain the alternative model.

[0052] In some embodiments, during the training of the candidate model, a first loss function is used to make shallow features mimic deep features (e.g., aligning the feature maps of the second and fifth layers with the feature maps of the fifth layer through cosine similarity constraints), thereby improving the feature representation capability of the shallow layers.

[0053] In some embodiments, when a candidate model contains multiple related subtasks, such as object detection and semantic segmentation, the outputs or intermediate features of different task branches can serve as soft targets for each other. For example, object bounding box information from the detection branch can be distilled into the segmentation branch to help more accurately locate objects; conversely, semantic labels from the segmentation branch can also assist the detection branch in filtering background interference. Alternatively, motion association information from the pose tracking branch can be used to assist the depth estimation branch in reducing dynamic scene errors, and geometric information from the depth estimation branch can be used to optimize the stability of the pose tracking branch.

[0054] In this embodiment, on the one hand, by aligning shallow features with deep features through a first loss function, the richer semantic information of deep features can be used to optimize the expression of shallow features, reducing the shortcomings of lightweight models that only capture details and lack global understanding due to their small number of layers and simple structure, thus improving the completeness and accuracy of feature extraction. On the other hand, if the candidate model contains multiple related sub-tasks, the knowledge distillation between the task branches can achieve information complementarity between branches, allowing the candidate model to maintain its lightweight characteristics while having stronger robustness and adaptability to complex scenarios, making it more suitable for tasks such as 3D reconstruction on lightweight electronic devices.

[0055] In some embodiments, cross-modal knowledge transfer is performed using the candidate model to obtain alternative models, including: Acquire high-modality sample data; The second teacher model was trained using the high-modal sample data, and geometric priors were extracted. The candidate model is obtained by forcing the first output of the candidate model to be consistent with the second output of the second teacher model through a second loss function; wherein the second loss function includes soft target loss, hard target loss and / or feature alignment loss.

[0056] In some embodiments, soft target loss can capture the details of the probability distribution output of the second teacher model, hard target loss can ensure the accuracy of key results, and feature alignment loss can force the candidate model to keep consistent with the intermediate feature map of the second teacher model, etc.

[0057] In some embodiments, an electronic device can first acquire high-modality sample data (such as high-precision LiDAR point clouds, RGB-D data, etc., which contain precise geometric information lacking in low-modality data); use this high-modality sample data to train a second teacher model, allowing it to fully learn the core features of the high-modality data and extract geometric priors (such as the three-dimensional dimensions of objects, surface normals, spatial positional relationships, etc.); through a second loss function (including soft target loss, hard target loss, and / or feature alignment loss), the first output of the candidate model (such as the depth map and reconstructed point cloud output by the candidate model relying solely on RGB images) is forced to be consistent with the second output of the second teacher model (such as the precise depth map and three-dimensional structure output by the teacher model trained on high-modality data), ultimately obtaining a candidate model. This process can effectively distill the geometric priors of high-modality data into a lightweight candidate model relying solely on RGB images, compensating for the lack of modal information caused by sensor limitations (inability to carry high-cost devices such as LiDAR) in low-resource devices (such as mobile phones and embedded terminals).

[0058] In this embodiment, by transferring the geometric prior of high-modal data to a candidate model that relies solely on RGB images, the problems of low reconstruction accuracy and poor robustness of low-resource devices are solved, allowing the lightweight candidate model to learn the spatial structure rules of high-modal data. On the other hand, with the guidance of the second teacher model and the precise alignment of multiple loss functions, the candidate model can maintain its lightweight characteristics while making its performance approach that of the high-modal model. This significantly reduces the deployment threshold and cost of 3D reconstruction technology in low-resource scenarios, while taking into account both the practicality and performance of the model.

[0059] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0060] Based on the same inventive concept, this application also provides a three-dimensional reconstruction apparatus for implementing the three-dimensional reconstruction method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the three-dimensional reconstruction apparatus provided below can be found in the limitations of the three-dimensional reconstruction method described above, and will not be repeated here.

[0061] In one embodiment, such as Figure 3 As shown, a three-dimensional reconstruction device is provided, the device comprising: The sparse module 10 is used to perform model compression and sparsification on the initially established model during the 3D reconstruction of the target model to obtain candidate models. Distillation module 20 is used to perform knowledge distillation on the candidate model through the first teacher model, self-distillation on the candidate model, and / or cross-modal knowledge transfer on the candidate model to obtain alternative models; The adjustment module 30 is used to adaptively adjust the training strategy of the alternative models according to the real-time load of the target device, the data quality of the 3D reconstruction task and / or the task objective requirements, so as to train the target model.

[0062] In one embodiment, the sparse module 10 is configured to perform at least one of the following steps: The first method involves quantizing the floating-point parameters and / or activation values ​​of the initial model during training to obtain a low-bit-width fixed-point number. The second method is to prune the initial model according to the importance of its parameters, thus obtaining the pruned initial model. The third method involves performing a low-rank decomposition on the high-rank matrix in the initial model to obtain a combination of multiple smaller matrices. The fourth method is to match the corresponding model network architecture based on the hardware limitations of the target device and the task characteristics of the 3D reconstruction task.

[0063] In one embodiment, the distillation module 20 is configured to perform the following steps: Transferable target knowledge is extracted from the first teacher model; wherein, the target knowledge includes intermediate layer knowledge and / or output layer knowledge; the intermediate layer knowledge includes intermediate feature maps and / or attention distributions; the output layer knowledge includes output logic, output probabilities, and / or pose estimation results; The candidate model is trained using the target knowledge to obtain the alternative model.

[0064] In one embodiment, the distillation module 20 is configured to perform the following steps: The shallow features and deep features of the candidate model are aligned using a first loss function to obtain the alternative model; And / or, When the candidate model includes multiple related subtasks, knowledge distillation is performed between the task branches corresponding to each subtask to obtain the alternative model.

[0065] In one embodiment, the distillation module 20 is configured to perform the following steps: Acquire high-modality sample data; The second teacher model was trained using the high-modal sample data, and geometric priors were extracted. The candidate model is obtained by forcing the first output of the candidate model to be consistent with the second output of the second teacher model through a second loss function; wherein the second loss function includes soft target loss, hard target loss and / or feature alignment loss.

[0066] In one embodiment, the adjustment module 30 is configured to perform the following steps: If the real-time load is less than or equal to the first threshold and / or the data quality is less than the second threshold, the training strategy of the candidate model is adaptively adjusted according to the task objective requirements and in conjunction with the basic rule engine to obtain the target model. If the real-time load is greater than the first threshold and / or the data quality is greater than or equal to the second threshold, the training strategy of the candidate model is adaptively adjusted according to the task objective requirements and in conjunction with a lightweight learning model to obtain the target model.

[0067] Each module in the aforementioned 3D reconstruction device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in the processor of the electronic device in hardware form or independent of the processor, or it can be stored in the memory of the electronic device in software form, so that the processor can call and execute the corresponding operations of each module.

[0068] In one embodiment, an electronic device is provided, the internal structure of which can be shown as follows: Figure 4 As shown, the electronic device includes a processor, memory, communication interface, display unit, and input device connected via a method bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores operating methods and computer programs. The internal memory provides an environment for the operation of the operating methods and computer programs stored in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a three-dimensional reconstruction method. The display screen can be an LCD screen or an e-ink display screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0069] Those skilled in the art in this field can understand. Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0070] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0071] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps performed by the processor of the electronic device of any of the above.

[0072] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0073] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, compilable logic units, quantum computing-based data processing logic units, etc., and are not limited to these.

[0074] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0075] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A three-dimensional reconstruction method, characterized in that, The method includes: During the 3D reconstruction of the target model, the initial model is compressed and sparsified to obtain candidate models. The candidate models are obtained by performing knowledge distillation on the candidate models using the first teacher model, self-distillation on the candidate models, and / or cross-modal knowledge transfer on the candidate models. The training strategy of the candidate models is adaptively adjusted based on the real-time load of the target device, the data quality of the 3D reconstruction task, and / or the task objective requirements, and the target model is trained to obtain the target model.

2. The method according to claim 1, characterized in that, The initial model compression and sparsification process includes at least one of the following: The first method involves quantizing the floating-point parameters and / or activation values ​​of the initial model during training to obtain a low-bit-width fixed-point number. The second method is to prune the initial model according to the importance of its parameters, thus obtaining the pruned initial model. The third method involves performing a low-rank decomposition on the high-rank matrix in the initial model to obtain a combination of multiple smaller matrices. The fourth method is to match the corresponding model network architecture based on the hardware limitations of the target device and the task characteristics of the 3D reconstruction task.

3. The method according to claim 1, characterized in that, The candidate models are subjected to knowledge distillation using the first teacher model to obtain alternative models, including: Transferable target knowledge is extracted from the first teacher model; wherein, the target knowledge includes intermediate layer knowledge and / or output layer knowledge; the intermediate layer knowledge includes intermediate feature maps and / or attention distributions; the output layer knowledge includes output logic, output probabilities, and / or pose estimation results; The candidate model is trained using the target knowledge to obtain the alternative model.

4. The method according to claim 1, characterized in that, The candidate models are obtained by self-distillation of the candidate models, including: The shallow features and deep features of the candidate model are aligned using a first loss function to obtain the alternative model; And / or, When the candidate model includes multiple related subtasks, knowledge distillation is performed between the task branches corresponding to each subtask to obtain the alternative model.

5. The method according to claim 1, characterized in that, Cross-modal knowledge transfer is performed using the candidate models to obtain alternative models, including: Acquire high-modality sample data; The second teacher model was trained using the high-modal sample data, and geometric priors were extracted. The candidate model is obtained by forcing the first output of the candidate model to be consistent with the second output of the second teacher model through a second loss function; wherein the second loss function includes soft target loss, hard target loss and / or feature alignment loss.

6. The method according to claim 1, characterized in that, The step of adaptively adjusting the training strategy of the candidate models based on the real-time load of the target device, the data quality of the 3D reconstruction task, and / or the task objective requirements, to train the target model, includes: If the real-time load is less than or equal to the first threshold and / or the data quality is less than the second threshold, the training strategy of the candidate model is adaptively adjusted according to the task objective requirements and in conjunction with the basic rule engine to obtain the target model. If the real-time load is greater than the first threshold and / or the data quality is greater than or equal to the second threshold, the training strategy of the candidate model is adaptively adjusted according to the task objective requirements and in conjunction with a lightweight learning model to obtain the target model.

7. A three-dimensional reconstruction device, characterized in that, The device includes: The sparse module is used to compress and sparsify the initial model during the 3D reconstruction of the target model to obtain candidate models. The distillation module is used to perform knowledge distillation on the candidate model through the first teacher model, self-distillation on the candidate model, and / or cross-modal knowledge transfer on the candidate model to obtain alternative models. The adjustment module is used to adaptively adjust the training strategy of the candidate models according to the real-time load of the target device, the data quality of the 3D reconstruction task and / or the task objective requirements, so as to train the target model.

8. An electronic device, characterized in that, It includes a processor and a memory; wherein the memory is used to store a computer program; and the processor is configured to, when executing the computer program, implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program performs the steps of the method as described in any one of claims 1 to 6.

Images