Single image metric scale human scene collaborative reconstruction method and system
By utilizing a self-supervised visual representation encoder and a Transformer decoder under single-image conditions, combined with a multi-dimensional redundant loss function and a deep optimization model, the accuracy and generalization problems of human body and scene collaborative reconstruction are solved, achieving high-precision metric reconstruction results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU YILAN TECH CO LTD
- Filing Date
- 2026-01-19
- Publication Date
- 2026-06-19
AI Technical Summary
Existing technologies struggle to achieve collaborative reconstruction of the human body and scene under single-image conditions, particularly in terms of insufficient accuracy and limited generalization at the measurement scale.
By acquiring RGB images for encoding feature extraction, and utilizing a self-supervised visual representation encoder and a Transformer decoder based on a self-attention mechanism, combined with a multi-dimensional redundancy loss function and a deep optimization model, camera intrinsic parameters and human body parameters are estimated to perform collaborative reconstruction of the human body and the scene.
It achieves collaborative measurement and reconstruction of human body and scene under single image conditions, ensuring that human body pose and global position conform to the projection relationship of real camera, thus improving the accuracy and generalization ability of measurement and reconstruction.
Smart Images
Figure CN121544813B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision 3D reconstruction technology, and in particular to a method and system for collaborative reconstruction of human scenes at a single image metric scale. Background Technology
[0002] The 3D reconstruction of human bodies and scenes from single images has shown broad application prospects and important application value in the field of virtual reality.
[0003] In the field of human body reconstruction, most current methods define the camera model as a weak perspective model, while some methods customize the camera model to achieve more reasonable human body reconstruction. However, all of the above methods are different from the real camera model, thus losing depth information and making it difficult to estimate the true scale of the scene.
[0004] In recent years, some methods have emerged that use real camera intrinsic parameters as input, often focusing on reconstructed geometric features while neglecting the true metric scale. However, metric depth estimation suffers from severely limited generalization in outdoor scenes, failing to meet practical needs. Examples include TRAM (ECCV2024), but its metric accuracy remains insufficient. In 3D reconstruction, methods such as VGGT (CVPR2025) can simultaneously estimate 3D point clouds and camera parameters, but struggle to estimate the true metric scale. Unidepth (CVPR2024) can achieve metric depth estimation, but its generalization in outdoor scenes is extremely limited.
[0005] Currently, there is no method that can achieve collaborative reconstruction of human body and scene at a single image metric scale. Therefore, how to provide a method and system for collaborative reconstruction of human body and scene at a single image metric scale is an urgent problem to be solved. Summary of the Invention
[0006] This invention provides a method and system for collaborative reconstruction of human scenes at a single image metric scale, in order to solve the aforementioned technical problems in the prior art.
[0007] According to a first aspect of the present invention, a method for collaborative reconstruction of human scene at a single image metric scale is provided.
[0008] The single-image metric scale human scene collaborative reconstruction method includes:
[0009] Acquire an RGB image and extract encoded features from the RGB image to obtain image features;
[0010] Based on the image features, a pre-configured scene decoder is used to perform decoding estimation to obtain camera intrinsic parameters and the initial depth of each image pixel;
[0011] Based on the image features, the human body decoder is trained using a pre-configured multi-dimensional redundancy loss function and initial human body parameters. The trained human body decoder is then used to estimate the human body model and obtain the human body parameters.
[0012] Based on the measured human body parameters, the global translation component of the human body model in the camera coordinate system is determined, and this global translation component is used as the human body measurement depth; and based on the human body measurement depth, the initial depth is corrected to obtain the scene measurement depth;
[0013] Human scene collaborative reconstruction based on camera intrinsic parameters, human body parameters, and scene depth measurement.
[0014] Specifically, the RGB image is subjected to coded feature extraction to obtain image features including:
[0015] The RGB image is divided into different image blocks and position codes are added. The image blocks and position codes are then input into a pre-configured self-supervised visual representation encoder for feature extraction to obtain image features.
[0016] Optionally, the self-supervised visual representation encoder is a DINOv3 type self-supervised visual representation encoder.
[0017] Optionally, both the scene decoder and the human body decoder are Transformer decoders based on a self-attention mechanism. The scene decoder maps image features to camera intrinsics and depth, and uses a camera output head and a depth output head composed of fully connected layers to estimate the camera intrinsics and the initial depth of each image pixel. The human body decoder maps image features to initial human body parameters, represents the camera intrinsics as pixel-aligned rays, and uses a human body parameter output head composed of fully connected layers to estimate the initial human body parameters in the camera coordinate system.
[0018] The initial human body parameters include: human body shape parameters, human body posture parameters, and human body global translation parameters.
[0019] The formula for the multi-dimensional redundancy loss function is as follows:
[0020]
[0021] In the formula, This represents the overall multi-dimensional redundancy loss function; This represents the true value of the global translation parameters of the human body; Represents a two-dimensional loss function; This is the result of two-dimensional key point detection; This represents the estimated projection of key human body points onto a two-dimensional plane. Represents the human pose loss function; Represents the true values of human posture parameters; This represents the estimated human posture parameters; Represents the human body size loss function; Represents the true values of human body shape parameters; This represents the estimated human body shape parameters; Represents the loss function for human height; This represents the true value of human height parameters; This represents the estimated height parameters of a person based on body shape parameters; Represents the image feature loss function; Representing image features; Represents the encoded features of the human body model; , , , and These represent the weights of the two-dimensional loss function term, pose loss function term, body shape loss function term, height loss function term, and feature loss function term, respectively.
[0022] Specifically, based on the human body depth measurement, the initial depth value is corrected to obtain the scene depth measurement, including:
[0023] A depth optimization model is constructed, and an optimization objective function is constructed with the goal of minimizing the error between the human body measurement depth and the initial depth. The optimization objective function is then solved using the least squares method to obtain the optimal depth scaling scale and the optimal scene global translation amount. The optimal depth scaling scale and the optimal scene global translation amount are then substituted into the depth optimization model to calculate the scene measurement depth.
[0024] The formula for the depth optimization model is as follows:
[0025]
[0026] The objective function formula is as follows:
[0027]
[0028] In the formula, Indicates the depth of scene measurement; This represents the initial depth of each image pixel; Indicates the depth scaling scale; Indicates the global translation of the scene; Indicates depth in human body measurements; This represents the objective function to be optimized.
[0029] The collaborative reconstruction of human scenes based on camera intrinsic parameters, human body parameters, and scene depth measurements includes:
[0030] Set the pixel depth value in the scene depth measurement;
[0031] Based on camera intrinsic parameters, the three-dimensional coordinates of pixels in the camera coordinate system are calculated using the inverse perspective projection transformation formula; and based on the three-dimensional coordinates, all pixels are traversed to generate a three-dimensional point cloud of the scene.
[0032] Based on the measurement of human body parameters, a human body mesh model is used to generate a three-dimensional human body mesh that conforms to the real measurement scale.
[0033] The human body 3D mesh and the scene 3D point cloud are placed in the same camera coordinate system to overlap the human body region.
[0034] The inverse perspective projection transformation formula for three-dimensional coordinates is:
[0035]
[0036] In the formula, Represents the horizontal coordinate in the camera coordinate system; Represents the vertical coordinate in the camera coordinate system; Represents the depth coordinates in the camera coordinate system; Indicates the horizontal focal length; Indicates a vertical angle; This represents the pixel depth value in the set scene measurement depth; Represents the x-coordinate of a pixel in the scene depth map; Represents the pixel ordinate in the scene depth map; Indicates the x-coordinate of the camera's principal point; This represents the ordinate of the camera's principal point.
[0037] According to a second aspect of the present invention, a collaborative reconstruction system for human scene using a single image metric scale is provided.
[0038] The single-image metric scale human scene collaborative reconstruction system includes:
[0039] An image encoding module is used to acquire an RGB image and extract encoded features from the RGB image to obtain image features;
[0040] The scene decoding module is used to perform decoding estimation based on the image features using a pre-configured scene decoder to obtain camera intrinsic parameters and the initial depth of each image pixel;
[0041] The human body decoding module is used to train the human body decoder based on the image features, using a pre-configured multi-dimensional redundancy loss function and initial human body parameters, and to use the trained human body decoder to estimate the human body model and obtain the human body parameters.
[0042] The measurement processing module is used to determine the global translation component of the human model in the camera coordinate system based on the measured human body parameters, and use the global translation component as the human body measurement depth; and to correct the initial depth based on the human body measurement depth to obtain the scene measurement depth.
[0043] The collaborative reconstruction module is used for collaborative reconstruction of human scenes based on camera intrinsic parameters, human body parameters, and scene depth measurements.
[0044] Specifically, when the image encoding module extracts encoded features from the RGB image to obtain image features, it divides the RGB image into different image blocks and adds positional encoding. The image blocks and positional encoding are then input into a pre-configured self-supervised visual representation encoder for feature extraction to obtain image features.
[0045] Optionally, the self-supervised visual representation encoder is a DINOv3 type self-supervised visual representation encoder.
[0046] Optionally, both the scene decoder and the human body decoder are Transformer decoders based on a self-attention mechanism. The scene decoder maps image features to camera intrinsics and depth, and uses a camera output head and a depth output head composed of fully connected layers to estimate the camera intrinsics and the initial depth of each image pixel. The human body decoder maps image features to initial human body parameters, represents the camera intrinsics as pixel-aligned rays, and uses a human body parameter output head composed of fully connected layers to estimate the initial human body parameters in the camera coordinate system.
[0047] The initial human body parameters include: human body shape parameters, human body posture parameters, and human body global translation parameters.
[0048] The formula for the multi-dimensional redundancy loss function is as follows:
[0049]
[0050] In the formula, This represents the overall multi-dimensional redundancy loss function; This represents the true value of the global translation parameters of the human body; Represents a two-dimensional loss function; This is the result of two-dimensional key point detection; This represents the projection of pre-estimated key points of the human body onto a two-dimensional plane; Represents the human pose loss function; Represents the true values of human posture parameters; This represents the estimated human posture parameters; Represents the human body size loss function; Represents the true values of human body shape parameters; This represents the estimated human body shape parameters; Represents the loss function for human height; Represents the true value of human height; This indicates the estimated height of a person based on body shape parameters; Represents the image feature loss function; Representing image features; Represents the encoded features of the human body model; , , , and These represent the weights of the two-dimensional loss function term, pose loss function term, body shape loss function term, height loss function term, and feature loss function term, respectively.
[0051] Specifically, when the measurement processing module corrects the initial depth value based on the human body measurement depth to obtain the scene measurement depth, it constructs a depth optimization model and builds an optimization objective function with the goal of minimizing the error between the human body measurement depth and the initial depth. The optimization objective function is then solved using the least squares method to obtain the optimal depth scaling scale and the optimal scene global translation amount. The optimal depth scaling scale and the optimal scene global translation amount are then substituted into the depth optimization model to calculate the scene measurement depth.
[0052] The formula for the depth optimization model is as follows:
[0053]
[0054] The objective function formula is as follows:
[0055]
[0056] In the formula, Indicates the depth of scene measurement; This represents the initial depth of each image pixel; Indicates the depth scaling scale; Indicates the global translation of the scene; Indicates depth in human body measurements; This represents the objective function to be optimized.
[0057] Specifically, when the collaborative reconstruction module performs human scene collaborative reconstruction based on camera intrinsic parameters, human body measurement parameters, and scene measurement depth, it sets the pixel depth value in the scene measurement depth; calculates the three-dimensional coordinates of the pixels in the camera coordinate system using the inverse perspective projection transformation formula based on the camera intrinsic parameters; and generates a scene three-dimensional point cloud by traversing all pixels based on the three-dimensional coordinates; generates a human three-dimensional mesh that conforms to the real measurement scale through a human body mesh model based on the human body measurement parameters; and places the human three-dimensional mesh and the scene three-dimensional point cloud in the same camera coordinate system to overlap the human body region.
[0058] The inverse perspective projection transformation formula for three-dimensional coordinates is:
[0059]
[0060] In the formula, Represents the horizontal coordinate in the camera coordinate system; Represents the vertical coordinate in the camera coordinate system; Show the depth coordinates in the camera coordinate system; Indicates the horizontal focal length; Indicates a vertical angle; This represents the pixel depth value in the set scene measurement depth; Represents the x-coordinate of a pixel in the scene depth map; Represents the pixel ordinate in the scene depth map; Indicates the x-coordinate of the camera's principal point; This represents the ordinate of the camera's principal point.
[0061] The technical solution provided by this invention may include the following beneficial effects:
[0062] This invention utilizes prior information about the interrelationship between the scene and the human body to directly represent the camera model as a realistic perspective projection model. By estimating the camera focal length and principal point position, this information is encoded into pixel-aligned ray representations and used for human body reconstruction supervision. This method ensures that the human body pose and global position conform to the projection relationship of the real camera. Human body metric reconstruction is achieved through redundant representation supervision. Simultaneously, the human body metric estimation results are used as constraints for metric depth estimation to achieve metric scene reconstruction. In an experimental environment, the method's effectiveness has been verified, successfully solving the problem of collaborative metric reconstruction between the human body and the scene.
[0063] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description
[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0065] Figure 1 This is a flowchart illustrating a single-image metric scale collaborative reconstruction method for human scene, according to an exemplary embodiment.
[0066] Figure 2 This is a structural block diagram of a single-image metric scale human scene collaborative reconstruction system according to an exemplary embodiment;
[0067] Figure 3 This is a schematic diagram illustrating the principle of collaborative reconstruction of a human scene at a single image metric scale, according to an exemplary embodiment.
[0068] Figure 4 This is a schematic diagram of the structure of a computer device according to an exemplary embodiment. Detailed Implementation
[0069] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some embodiments may include or substitute parts and features of other embodiments. The scope of the embodiments herein encompasses the entire scope of the claims and all available equivalents thereof. Throughout this document, the terms “first,” “second,” etc., are used only to distinguish one element from another without requiring or implying any actual relationship or order between the elements. Indeed, a first element can also be referred to as a second element, and vice versa. Furthermore, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a structure, apparatus, or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a structure, apparatus, or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the structure, apparatus, or device that includes said element. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.
[0070] The terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer" used in this document to indicate orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings. They are used solely for the convenience of describing the document and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In the description herein, unless otherwise specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two elements; they can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.
[0071] In this document, unless otherwise stated, the term "multiple" means two or more.
[0072] In this article, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.
[0073] In this article, the term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.
[0074] It should be understood that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order constraint on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the diagram may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0075] The modules in the apparatus or system of this application can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0076] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.
[0077] Figure 1An embodiment of a single-image metric scale collaborative reconstruction method for human scene is shown.
[0078] In this optional embodiment, the single-image metric scale human scene collaborative reconstruction method includes:
[0079] Step S101: Obtain an RGB image and extract encoded features from the RGB image to obtain image features;
[0080] Step S102: Based on the image features, perform decoding estimation using a pre-configured scene decoder to obtain camera intrinsic parameters and the initial depth of each image pixel;
[0081] Step S103: Based on the image features, the human body decoder is trained using a pre-configured multi-dimensional redundancy loss function and initial human body parameters, and the trained human body decoder is used to estimate the human body model to obtain the human body parameters.
[0082] Step S104: Based on the measured human body parameters, determine the global translation component of the human body model in the camera coordinate system, and use the global translation component as the human body measurement depth; and based on the human body measurement depth, correct the initial depth to obtain the scene measurement depth.
[0083] Step S105: Perform collaborative reconstruction of the human scene based on camera intrinsic parameters, human body parameters, and scene depth.
[0084] Figure 2 An embodiment of a single-image metric scale human scene collaborative reconstruction system of the present invention is shown.
[0085] In this optional embodiment, the single-image metric scale human scene collaborative reconstruction system includes:
[0086] Image encoding module 201 is used to acquire RGB images and extract encoded features from the RGB images to obtain image features;
[0087] The scene decoding module 202 is used to perform decoding estimation based on the image features using a pre-configured scene decoder to obtain camera intrinsic parameters and the initial depth of each image pixel;
[0088] The human body decoding module 203, based on the image features, trains the human body decoder using a pre-configured multi-dimensional redundancy loss function and initial human body parameters, and uses the trained human body decoder to estimate the human body model and obtain the measured human body parameters.
[0089] The measurement processing module 204 is used to determine the global translation component of the human model in the camera coordinate system based on the measured human body parameters, and use the global translation component as the human body measurement depth; and to correct the initial depth based on the human body measurement depth to obtain the scene measurement depth.
[0090] The collaborative reconstruction module 205 is used for collaborative reconstruction of human scenes based on camera intrinsic parameters, human body parameters, and scene depth measurements.
[0091] In the above optional embodiments, when extracting image features from the RGB image, the RGB image can be divided into different image blocks and position codes can be added. The image blocks and position codes are then input into a pre-configured DINOv3 type self-supervised visual representation encoder for feature extraction to obtain image features.
[0092] Both the scene decoder and the human decoder can employ a Transformer decoder based on a self-attention mechanism. The scene decoder consists of several Transformer modules and uses a self-attention mechanism to allow information interaction between different parts of the image features, thereby understanding the meaning of each local feature within the global context. It maps image features to camera intrinsics and depth, using a camera output head and a depth output head composed of fully connected layers to estimate the camera intrinsics and initial depth. The human decoder has the same architecture as the scene decoder, but learns different information during training due to different network supervision signals. The human decoder directly constructs a mapping from image features to initial human parameters, representing camera intrinsics as pixel-aligned rays. Using a human parameter output head composed of fully connected layers, it estimates the initial human parameters in the camera coordinate system. These initial human parameters can include: human body shape parameters, human pose parameters, and human global translation parameters.
[0093] In the above optional embodiments, the multi-dimensional redundancy loss function formula is:
[0094]
[0095] In the formula, This represents the overall multi-dimensional redundancy loss function; This represents the true value of the global translation parameters of the human body; Represents a two-dimensional loss function; This is the result of two-dimensional key point detection; This represents the projection of the pre-estimated human key points onto a two-dimensional plane (estimating human two-dimensional key points is a mature technology in the field, and is generally obtained from images through open-source Openpose models, DWPose models, and RTMPose models). Represents the human pose loss function; Represents the true values of human posture parameters; This represents the estimated human posture parameters; Represents the human body size loss function; Represents the true values of human body shape parameters; This represents the estimated human body shape parameters; Represents the loss function for human height; Represents the true value of human height; This indicates the estimated height of a person based on body shape parameters; Represents the image feature loss function; Representing image features; Represents the encoded features of the human body model; , , , and These represent the weights of the two-dimensional loss function term, pose loss function term, body shape loss function term, height loss function term, and feature loss function term, respectively.
[0096] By employing redundant representation supervision, the network learns anthropometric information. Compared to existing methods that only use body shape loss function, an additional height loss function is added. This allows the network to learn more accurately real-world human body measurements, compared to existing methods that only use a two-dimensional loss function. Additional feature loss function This allows the network to learn the correspondence between the human body model and the image more accurately. This redundant representation supervision helps the network model converge faster, achieving metric human body estimation.
[0097] In the above optional embodiments, when correcting the initial depth value based on the human body measurement depth to obtain the scene measurement depth, a depth optimization model can be constructed first. An optimization objective function is constructed with the goal of minimizing the error between the human body measurement depth and the initial depth. The optimization objective function is then solved using the least squares method to obtain the optimal depth scaling scale and the optimal global scene translation. The optimal depth scaling scale and the optimal global scene translation are then substituted into the depth optimization model for calculation to obtain the scene measurement depth.
[0098] The formula for the depth optimization model is as follows:
[0099]
[0100] The objective function formula is as follows:
[0101]
[0102] In the formula, Indicates the depth of scene measurement; This represents the initial depth of each image pixel; Indicates the depth scaling scale; Indicates the global translation of the scene; Indicates depth in human body measurements; This represents the objective function to be optimized.
[0103] When performing collaborative reconstruction of human scenes based on camera intrinsics, human body parameters, and scene depth measurements, the pixel depth value in the scene depth measurement can be set; based on camera intrinsics, the three-dimensional coordinates of pixels in the camera coordinate system are calculated using the inverse perspective projection transformation formula; and all pixels are traversed based on the three-dimensional coordinates to generate a scene three-dimensional point cloud; based on human body parameters, a human body three-dimensional mesh conforming to the real measurement scale is generated through a human body mesh model (such as the SMPL model (Skinned Multi-Person Linear Model)); and the human body three-dimensional mesh and the scene three-dimensional point cloud are placed in the same camera coordinate system to overlap the human body region.
[0104] The inverse perspective projection transformation formula for three-dimensional coordinates is:
[0105]
[0106] In the formula, Represents the horizontal coordinate in the camera coordinate system; Represents the vertical coordinate in the camera coordinate system; Show the depth coordinates in the camera coordinate system; Indicates the horizontal focal length; Indicates a vertical angle; This represents the pixel depth value in the set scene measurement depth; Represents the x-coordinate of a pixel in the scene depth map; Represents the pixel ordinate in the scene depth map; Indicates the x-coordinate of the camera's principal point; This represents the ordinate of the camera's principal point.
[0107] Figure 4An embodiment of a computer device according to the present invention is shown. The computer device may be a server, and includes a processor, memory, and a network interface connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores static and dynamic information data. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the above-described method embodiment.
[0108] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0109] In addition, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.
[0110] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.
[0111] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.
[0112] This invention is not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this invention is limited only by the appended claims.
Claims
1. A single image metric scale human scene collaborative reconstruction method, characterized in that, include: Acquire an RGB image and extract encoded features from the RGB image to obtain image features; Based on the image features, a pre-configured scene decoder is used to perform decoding estimation to obtain camera intrinsic parameters and the initial depth of each image pixel; Based on the image features, the human body decoder is trained using a pre-configured multi-dimensional redundancy loss function and initial human body parameters. The trained human body decoder is then used to estimate the human body model and obtain the human body parameters. Based on the measured human body parameters, the global translation component of the human body model in the camera coordinate system is determined, and this global translation component is used as the human body measurement depth; and based on the human body measurement depth, the initial depth is corrected to obtain the scene measurement depth; Human scene collaborative reconstruction based on camera intrinsic parameters, human body parameters, and scene depth measurement; Both the scene decoder and the human body decoder are Transformer decoders based on a self-attention mechanism. The scene decoder maps image features to camera intrinsics and depth, and uses a camera output head and a depth output head composed of fully connected layers to estimate the camera intrinsics and the initial depth of each image pixel. The human body decoder maps image features to initial human body parameters, represents the camera intrinsics as pixel-aligned rays, and uses a human body parameter output head composed of fully connected layers to estimate the initial human body parameters in the camera coordinate system. The formula for the multi-dimensional redundancy loss function is as follows: In the formula, This represents the overall multi-dimensional redundancy loss function; This represents the true value of the global translation parameters of the human body; Represents a two-dimensional loss function; This is the result of two-dimensional key point detection; This represents the projection of pre-estimated key points of the human body onto a two-dimensional plane; Represents the human pose loss function; Represents the true values of human posture parameters; This represents the estimated human posture parameters; Represents the human body size loss function; Represents the true values of human body shape parameters; This represents the estimated human body shape parameters; Represents the loss function for human height; Represents the true value of human height; This indicates the estimated height of a person based on body shape parameters; Represents the image feature loss function; Representing image features; Represents the encoded features of the human body model; , , , and These represent the weights of the two-dimensional loss function term, pose loss function term, body shape loss function term, height loss function term, and feature loss function term, respectively.
2. The single image metric scale human scene co-reconstruction method according to claim 1, characterized in that, The RGB image is subjected to coded feature extraction to obtain image features including: The RGB image is divided into different image blocks and position codes are added. The image blocks and position codes are then input into a pre-configured self-supervised visual representation encoder for feature extraction to obtain image features.
3. The single image metric scale human scene co-reconstruction method according to claim 2, characterized in that, The self-supervised visual representation encoder is a DINOv3 type self-supervised visual representation encoder.
4. The single image metric scale human scene co-reconstruction method according to claim 1, characterized in that, The initial parameters of the human body include: human body shape parameters, human body posture parameters, and human body global translation parameters.
5. The single image metric scale human scene co-reconstruction method according to claim 1, characterized in that, Human scene collaborative reconstruction based on camera intrinsic parameters, human body parameters, and scene depth measurements includes: Set the pixel depth value in the scene depth measurement; Based on camera intrinsic parameters, the three-dimensional coordinates of pixels in the camera coordinate system are calculated using the inverse perspective projection transformation formula; and based on the three-dimensional coordinates, all pixels are traversed to generate a three-dimensional point cloud of the scene. Based on the measurement of human body parameters, a human body mesh model is used to generate a three-dimensional human body mesh that conforms to the real measurement scale. The human body 3D mesh and the scene 3D point cloud are placed in the same camera coordinate system to overlap the human body region.
6. The single image metric scale human scene co-reconstruction method according to claim 5, characterized in that, The formula for the inverse perspective projection transformation in three-dimensional coordinates is: In the formula, Represents the horizontal coordinate in the camera coordinate system; Represents the vertical coordinate in the camera coordinate system; Represents the depth coordinates in the camera coordinate system; Indicates the horizontal focal length; Indicates a vertical angle; This represents the pixel depth value in the set scene measurement depth; Represents the x-coordinate of a pixel in the scene depth map; Represents the pixel ordinate in the scene depth map; Indicates the x-coordinate of the camera's principal point; This represents the ordinate of the camera's principal point.
7. A single image metric scale human scene collaborative reconstruction system, characterized in that, include: An image encoding module is used to acquire an RGB image and extract encoded features from the RGB image to obtain image features; The scene decoding module is used to perform decoding estimation based on the image features using a pre-configured scene decoder to obtain camera intrinsic parameters and the initial depth of each image pixel; The human body decoding module is used to train the human body decoder based on the image features, using a pre-configured multi-dimensional redundancy loss function and initial human body parameters, and to use the trained human body decoder to estimate the human body model and obtain the human body parameters. The measurement processing module is used to determine the global translation component of the human model in the camera coordinate system based on the measured human body parameters, and use the global translation component as the human body measurement depth; and to correct the initial depth based on the human body measurement depth to obtain the scene measurement depth. The collaborative reconstruction module is used for collaborative reconstruction of human scenes based on camera intrinsic parameters, human body parameters, and scene depth measurements. Both the scene decoder and the human body decoder are Transformer decoders based on a self-attention mechanism. The scene decoder maps image features to camera intrinsics and depth, and uses a camera output head and a depth output head composed of fully connected layers to estimate the camera intrinsics and the initial depth of each image pixel. The human body decoder maps image features to initial human body parameters, represents the camera intrinsics as pixel-aligned rays, and uses a human body parameter output head composed of fully connected layers to estimate the initial human body parameters in the camera coordinate system. The formula for the multi-dimensional redundancy loss function is as follows: In the formula, This represents the overall multi-dimensional redundancy loss function; This represents the true value of the global translation parameters of the human body; Represents a two-dimensional loss function; This is the result of two-dimensional key point detection; This represents the projection of pre-estimated key points of the human body onto a two-dimensional plane; Represents the human pose loss function; Represents the true values of human posture parameters; This represents the estimated human posture parameters; Represents the human body size loss function; Represents the true values of human body shape parameters; This represents the estimated human body shape parameters; Represents the loss function for human height; Represents the true value of human height; This indicates the estimated height of a person based on body shape parameters; Represents the image feature loss function; Representing image features; Represents the encoded features of the human body model; , , , and These represent the weights of the two-dimensional loss function term, pose loss function term, body shape loss function term, height loss function term, and feature loss function term, respectively.