Method and apparatus for constructing a three-dimensional scanned body model

Abstract

An electronic device for object construction includes a processor. The processor is configured to: obtain a 3D scan of the object. The processor is further configured to: match the established parametric model to the 3D scan by minimizing the surface distance between the established parametric model and the 3D scan, and by minimizing 3D joint errors. The processor is further configured to: identify the correspondence between the established parametric model and the 3D scan. Furthermore, the processor is configured to: transfer attributes of the established parametric model to the 3D scan based on the correspondence to generate the constructed 3D scan. The processor is further configured to: apply animation to the constructed 3D scan. The constructed 3D scan with the applied animation is displayed on a screen.

Description

Technical Field

[0001] This disclosure generally relates to computer-generated graphics. More specifically, this disclosure relates to animate static scans of three-dimensional objects. Background Technology

[0002] 3D content, such as animated 3D objects, is becoming increasingly popular. For example, 3D content is frequently found in movies and games and can be entirely computer-generated or generated from an object's set image or a combination thereof. Animating static objects involves constructing the object using both layered sets of surface representations (such as skin) and interconnected parts (such as bones). Constructing static objects into animations is complex and time-consuming, which hinders the widespread adoption of creating fully animated objects. Summary of the Invention

[0003] Technical issues

[0004] This disclosure provides methods and apparatus for constructing 3D scanned human body models.

[0005] Technical solution

[0006] In one embodiment, an electronic device for object construction is provided. The electronic device includes a processor. The processor is configured to: obtain a 3D scan of the object. The processor is further configured to: match the established parametric model to the 3D scan by minimizing the surface distance between the established parametric model and the 3D scan, and by minimizing 3D joint errors. The processor is further configured to: identify the correspondence between the established parametric model and the 3D scan. Furthermore, the processor is configured to: transfer attributes of the established parametric model to the 3D scan based on the correspondence to generate the constructed 3D scan. The processor is further configured to: apply animation to the constructed 3D scan. The constructed 3D scan with the applied animation is displayed on a monitor.

[0007] In another embodiment, a method for object construction is provided. The method includes obtaining a 3D scan of the object. The method further includes matching the established parametric model to the 3D scan by minimizing the surface distance between the established parametric model and the 3D scan, and minimizing 3D joint errors. The method also includes identifying the correspondence between the established parametric model and the 3D scan. Furthermore, the method includes transferring attributes of the established parametric model to the 3D scan based on the correspondence to generate the constructed 3D scan. The method also includes applying animation to the constructed 3D scan. The constructed 3D scan with the applied animation is displayed on a monitor.

[0008] Other technical features will be apparent to those skilled in the art from the following figures, description and claims.

[0009] Before proceeding with the detailed description below, it may be advantageous to define the specific words and phrases used throughout this patent document. The term “connection” and its derivatives refer to any direct or indirect communication between two or more elements, whether or not these elements are physically in contact with each other. The terms “transmit,” “receive,” and “communicate,” and their derivatives, include both direct and indirect communication. The terms “comprise” and “include,” and their derivatives, mean non-limiting inclusion. The term “or” is inclusive, meaning “and / or.” The phrase “related to” and its derivatives mean including, being included within, interconnected with, containing, being contained within, connected to or connected to, coupled to or connected to, communicable with, cooperating with, intertwined, juxtaposed, proximate, bound to or bound to, having, possessing the characteristics of, having or having a relationship with, etc. The term “controller” means any device, system, or part thereof that controls at least one operation. Such a controller may be implemented as hardware, or a combination of hardware and software and / or firmware. The functionality associated with any particular controller can be centralized or distributed, local or remote. When used with a list of items, the phrase "at least one" means that one or more different combinations of the listed items can be used, and only one item in the list may be required. For example, "at least one of A, B, and C" includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A and B and C.

[0010] Furthermore, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed by computer-readable program code and implemented in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, associated data, or portions thereof suitable for implementation in appropriate computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer-readable medium" includes any type of medium accessible by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive, optical disc (CD), digital video disc (DVD), or any other type of storage. "Non-transitory" computer-readable medium excludes wired, wireless, optical, or other communication links that temporarily transmit electrical or other signals. Non-transitory computer-readable medium includes media that can permanently store data as well as media that can store data and subsequently rewrite it, such as rewritable optical discs or erasable storage devices.

[0011] Definitions of other specific words and phrases are provided throughout this patent document. Those skilled in the art will understand that, in many (if not most) instances, such definitions apply to the prior and future use of the words and phrases thus defined. Attached Figure Description

[0012] To gain a more thorough understanding of this disclosure and its advantages, reference is now made to the accompanying drawings, in which like reference numerals denote like components:

[0013] Figure 1 An exemplary communication system according to an embodiment of the present disclosure is shown;

[0014] Figure 2 and Figure 3 Exemplary electronic devices according to embodiments of the present disclosure are shown;

[0015] Figure 4 An example diagram is shown illustrating the conversion of a 3D scan image into an animated model according to an embodiment of the present disclosure;

[0016] Figure 5A An exemplary method for constructing an animated model from a static object according to an embodiment of the present disclosure is shown;

[0017] Figure 5B and Figure 5C An established parametric body model according to an embodiment of this disclosure is shown;

[0018] Figure 6A An exemplary method for fitting according to embodiments of the present disclosure is shown;

[0019] Figure 6B and Figure 6C An example of associating a static object with an established parametric body model according to an embodiment of this disclosure is shown;

[0020] Figure 7A and Figure 7B An example is shown of deforming components of an existing parametric body model according to an embodiment of the present disclosure;

[0021] Figure 7C An exemplary method of using semantic variations according to embodiments of this disclosure is shown;

[0022] Figure 8A An exemplary method for skinning according to embodiments of the present disclosure is shown;

[0023] Figure 8B The embodiments of the present disclosure are shown in Figure 8A Weights are transferred during the skinning process;

[0024] Figure 8C The embodiments of the present disclosure are shown in Figure 8A During the skinning process, the scan map is converted into a predefined pose;

[0025] Figure 9A An exemplary method for patching is shown according to an embodiment of the present disclosure;

[0026] Figure 9B and Figure 9C 2D images of a 3D scan according to an embodiment of the present disclosure, as well as images before and after patching;

[0027] Figure 9D and Figure 9E This illustrates the process of correcting errors related to the hands and feet of a 3D scanned object according to embodiments of the present disclosure;

[0028] Figure 9F Images are shown at different stages of generating the constructed 3D scan image according to embodiments of the present disclosure;

[0029] Figure 9G The filling is shown according to an embodiment of the present disclosure; and

[0030] Figure 10 An exemplary method for object construction according to an embodiment of this disclosure is shown. Detailed Implementation

[0031] The following discussion Figures 1 to 10 The various embodiments described in this patent document to illustrate the principles of this disclosure are merely exemplary and should not be construed in any way as limiting the scope of this disclosure. Those skilled in the art will understand that the principles of this disclosure can be implemented in any suitably arranged system or apparatus.

[0032] Many different types of devices can provide immersive experiences related to augmented reality (AR) or virtual reality (VR). It should be noted that VR is a rendered version of a visual object or scene, while AR is an interactive experience of a real-world environment in which objects residing in the real-world environment are augmented by virtual objects, virtual information, or both. An exemplary device capable of providing AR and VR content is a head-mounted display (HMD). An HMD is a device that allows a user to view a VR scene and adjust the displayed content based on the user's head movements. Typically, HMDs rely on a dedicated screen integrated into the device and connected to an external computer (cable-connected), or on a device plugged into the HMD (uncable-connected), such as a smartphone. The first approach utilizes one or more lightweight screens and benefits from high computing power. Conversely, smartphone-based systems utilize greater mobility and are less expensive to produce. In both cases, the resulting video experience is the same. It should be noted that, as used herein, the term "user" can refer to a person using an electronic device or another device (such as an AI-powered electronic device).

[0033] A 3D scan of an object is a photorealistic model obtained from a set of images. A 3D scan is a 360-degree 3D image of an object (such as a person or animal). Images can come from cameras such as color cameras (e.g., RGB cameras), depth cameras, or a combination of both (e.g., RGB-D cameras). A 3D scan of an object can be created by capturing multiple images of the object at different angles (or orientations) and then stitching them together. 3D scans of objects can be used in 3D graphics (such as games, user avatars, movies, etc.), 3D printing (creating small statues of objects), and more. For example, a 3D scan can be created as an avatar of a person. A 3D scan can consist of millions of vertices.

[0034] Embodiments of this disclosure take into account that since the images used to generate 3D scan images are static (non-moving), the 3D scan images also lack the ability to move on their own. Therefore, embodiments of this disclosure provide systems and methods for automatically constructing motion from static 3D scan images. That is, constructing a 3D scan image is the ability to provide motion (animation) to a static 3D scan image. After constructing the 3D scan image, a portion of the scan image can move relative to other portions of the 3D scan image at joints. For example, if the scan image is about a person, the joints of the 3D scan image can correspond to elbows, knees, hips, wrists, fingers, toes, ankles, etc. The joints of the 3D scan image are used to provide movement that simulates movement that can be performed by a person. In other words, constructing a 3D scan image (such as a 3D scan image of a person) can animate a static scan image that can be used in AR and VR applications.

[0035] To animate 3D scans, "rigging" is applied. Rigging involves bone-based layering and appropriate skinning weights, which are defined to deform the surfaces of the 3D scan based on input actions. The skeletal structure can be based on a parametric model that includes joints at predetermined locations. That is, the skeletal structure resembles the human skeleton because it is designed to move at specific locations (such as joints) while other parts remain rigid.

[0036] According to embodiments of this disclosure, the construction comprises two parts: fitting and skinning. Fitting involves applying the skeletal structure of a pre-built parametric model to a 3D scan image. A skeleton is a joint tree used to define the skeletal structure of an object (such as a human). While the tree structure remains fixed for different scan models, joint positions can vary. Thus, body movements are defined on the skeletal tree. Each joint has three degrees of freedom (DoF) to define its rotation. Root joints have three or more DOFs to define translation. Skinning defines how surface vertices (skin) are affected by the joint movements. For a given movement of the skeleton, the surface of the scan image is deformed to perform the movement with minimal artifacts. When the movement is applied to a previously constructed 3D scan image with a skeleton and skinning is applied to it, the 3D scan image is animated to follow the movement.

[0037] The embodiments of this disclosure take into account the possibility that, for a given 3D scan image, there may be occlusions (such as holes), or that parts of the 3D scan image may be incorrectly connected (such as...). Figure 9F (As shown in image 942a). A 3D scan may exhibit self-occlusion when one body part blocks another. For example, a person's arm may obscure part of the torso. Similarly, a portion of a 3D scan may be omitted when a body part is hidden. For example, a person's hand may be in a pocket and therefore not included in the 3D scan. Therefore, embodiments of this disclosure provide systems and methods for identifying occluded or missing portions of a 3D scan and filling in the occluded or missing areas.

[0038] Embodiments of this disclosure provide systems and methods for automatically constructing 3D scan images to animate them. The automatic construction of 3D scan images can be performed on a server or on an electronic device that captures images compiled to generate 3D scan images.

[0039] To animate static 3D scan images, embodiments of this disclosure use a pre-built parametric body model (also called a parametric body model). The parametric body model is used to transfer construction information to the 3D scan image model. First, the parametric body model and the 3D scan image are aligned. Alignment is the process of optimizing the shape and pose parameters of the parametric body model to the shape and pose parameters of the 3D scan image. Semantic deformable components (SDCs) can be used to enhance the fitting results. After aligning the parametric body model and the 3D scan image, the correspondences between the parametric body model and the 3D scan image are identified. To create a complete, animable 3D scan image, the identified correspondences are used to disconnect unwanted connections in the 3D scan image, remove self-occlusion, etc. To disconnect unwanted connections, correspondences are used to separate connected but separable parts of the 3D scan image. For example, if a person's arm is attached to the torso, correspondences are used to separate the arm from the torso so that the arm can be animated. After constructing the 3D scan image, animation can be added to the avatar to perform any predefined actions.

[0040] Embodiments of this disclosure provide systems and methods for constructing 3D scan images of objects, such as people. The person can be in any pose. The person can wear various types of clothing. The 3D scan image can even include unwanted connections, self-occlusion, and missing regions.

[0041] Embodiments of this disclosure also provide systems and methods for combining 3D joint errors and chamfer distances as fitting energy, wherein the fitting energy is minimized to align the 3D scan image with a parametric body model, regardless of changes in clothing and pose. Embodiments of this disclosure also provide systems and methods for enhancing the fit between the 3D scan image and the parametric body model by performing SDC (Self-Construction Control). It should be noted that SDC can also be used to reshape the body after it has been fully constructed (in order to change the physical shape of the constructed 3D scan image). Embodiments of this disclosure also provide systems and methods for using geometric and texture patching to break unwanted connections, repair self-occlusion, fill missing regions in the 3D scan image, etc.

[0042] Figure 1 An exemplary communication system 100 according to an embodiment of the present disclosure is shown. Figure 1 The embodiments of the communication system 100 shown are for illustrative purposes only. Other embodiments of the communication system 100 may be used without departing from the scope of this disclosure.

[0043] Communication system 100 includes a network 102 that facilitates communication between the various components within communication system 100. For example, network 102 can transmit Internet Protocol (IP) packets, Frame Relay frames, Asynchronous Transfer Mode (ATM) cells, or other information between network addresses. Network 102 includes one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), all or part of a global network such as the Internet, or any other communication system located in one or more locations.

[0044] In this example, network 102 facilitates communication between server 104 and various client devices 106-116. Client devices 106-116 may be, for example, smartphones, tablets, laptops, personal computers, wearable devices, head-mounted displays (HMDs), etc. Server 104 may represent one or more servers. Each server 104 includes any suitable computing or processing device that can provide computing services to one or more client devices, such as client devices 106-116. For example, each server 104 may include one or more processing devices, one or more memories storing instructions and data, and one or more network interfaces facilitating communication through network 102. As described in more detail below, server 104 can build animable 3D models onto one or more display devices, such as client devices 106-116.

[0045] Each client device 106-116 represents any suitable computing or processing device that interacts with at least one server (such as server 104) or other computing device via network 102. In this example, client devices 106-116 include desktop computer 106, mobile phone or mobile device 108 (e.g., smartphone), PDA 110, laptop computer 112, tablet computer 114, and HMD 116. However, any other or additional client devices may be used in communication system 100. Smartphones represent a class of mobile devices 108 that are handheld devices with a mobile operating system and integrated mobile broadband cellular network connectivity for voice, short message service (SMS), and Internet data communication. HMD 116 can display a 360° scene including one or more animated scans.

[0046] In this example, some client devices 108-116 communicate indirectly with network 102. For example, mobile device 108 and PDA 110 communicate via one or more base stations 118, such as cellular base stations or eNodeBs (eNBs). Additionally, laptop computer 112, tablet computer 114, and HMD 116 communicate via one or more wireless access points 120, such as IEEE 802.11 wireless access points. Note that these are for illustrative purposes only, and each client device 106-116 may communicate directly with network 102 or indirectly with network 102 via any suitable intermediary or network. In some embodiments, server 104 or any client device 106-116 may be used to animate a received 3D scan and send the animated scan to another client device, such as any client device 106-116.

[0047] In some implementations, any of client devices 106-114 securely and efficiently transmits information to another device (e.g., server 104). Furthermore, any of client devices 106-116 can trigger information transmission between itself and server 104. Any of client devices 106-114 can function as a VR display when connected to a headset via a bracket, and function similarly to HMD 116. For example, mobile device 108 can function similarly to HMD 116 when attached to a bracket system and worn on a user's eyes. Mobile device 108 (or any other client device 106-116) can trigger information transmission between itself and server 104.

[0048] In some implementations, any of client devices 106-116 or server 104 can create 3D scan images, send 3D scan images, animate 3D scan images, receive animated 3D scan images, render animated 3D scan images, or perform a combination thereof. For example, mobile device 108 can capture an image of a model and transmit the image to server 104 for construction. As another example, mobile device 108 can receive multiple images from other client devices and transmit the images to server 104 for construction. Yet another example is that mobile device 108 (or any other client device) can perform the construction of the model.

[0049] although Figure 1 An example of a communication system 100 is shown, but it is possible to compare it with other systems. Figure 1 Various changes can be made. For example, communication system 100 can include any number of each component in any suitable arrangement. Typically, computing and communication systems have multiple configurations, and Figure 1 This disclosure is not intended to limit the scope to any particular configuration. Although Figure 1This paper illustrates an operating environment in which the various features disclosed in this patent document can be used, but these features can be used in any other suitable system.

[0050] although Figure 1 An example of a communication system 100 is shown, but it is possible to compare it with other systems. Figure 1 Various changes can be made. For example, communication system 100 can include any number of each component in any suitable arrangement. Typically, computing and communication systems have multiple configurations, and Figure 1 This disclosure is not intended to limit the scope to any particular configuration. Although Figure 1 This paper illustrates an operating environment in which the various features disclosed in this patent document can be used, but these features can be used in any other suitable system.

[0051] Figure 2 and Figure 3 An exemplary electronic device according to embodiments of the present disclosure is shown. In particular, Figure 2 An exemplary server 200 is shown, and server 200 can represent Figure 1 Server 104. Server 200 can represent one or more local servers, remote servers, cluster computers, and components that act as a single pool of seamless resources, cloud-based servers, etc. Server 200 can be... Figure 1 The client device 106-116 can access one or more of the other server.

[0052] like Figure 2 As shown, server 200 includes bus system 205 that supports communication between at least one processing device (such as processor 210), at least one storage device 215, at least one communication interface 220 and at least one input / output (I / O) unit 225.

[0053] Processor 210 executes instructions that can be stored in memory 230. Processor 210 may include any suitable number and type of processors or other devices of any suitable arrangement. Exemplary types of processor 210 include microprocessors, microcontrollers, digital signal processors, field-programmable gate arrays, application-specific integrated circuits (ASICs), and discrete circuits. In some embodiments, processor 210 may animate 3D scan images stored in storage device 215.

[0054] Memory 230 and permanent storage space 235 are examples of storage devices 215 representing any structure capable of storing and facilitating the retrieval of information (such as data, program code, or other suitable information of temporary or permanent nature). Memory 230 may represent random access memory or any other suitable volatile or non-volatile storage device. For example, instructions stored in memory 230 may include instructions for constructing a received 3D scan image and instructions for animate the constructed 3D scan image. Permanent storage space 235 may contain one or more components or devices supporting long-term storage of data, such as read-only memory, hard disk drive, flash memory, or optical disk.

[0055] Communication interface 220 supports communication with other systems or devices. For example, communication interface 220 may include a network interface card or facilitate communication via... Figure 1 The wireless transceiver 220 is used for communication with network 102. Communication interface 220 can support communication via any suitable physical or wireless communication link. For example, communication interface 220 can send a constructed 3D scan image to another device, such as one of client devices 106-116.

[0056] I / O unit 225 allows for data input and output. For example, I / O unit 225 can provide connectivity for user input via a keyboard, mouse, keypad, touchscreen, or other suitable input device. I / O unit 225 can also send output to a display, printer, or other suitable output device. However, note that I / O unit 225 can be omitted, such as when I / O interaction occurs with server 200 via a network connection.

[0057] Note that, although Figure 2 Described as a representative Figure 1 The server 104, however, can have the same or similar architecture in one or more different client devices 106-116. For example, desktop computer 106 or laptop computer 112 can have the same architecture as the server 104. Figure 2 The same or similar structures shown.

[0058] Figure 3 An exemplary electronic device 300 is shown, and the electronic device 300 can represent Figure 1 One or more of the client devices 106-116. Electronic device 300 may be a mobile communication device, a desktop computer (similar to...) Figure 1 Desktop computers 106), portable electronic devices (similar to) Figure 1 Mobile devices 108, PDA 110, laptop computer 112, tablet computer 114, or HMD 116, etc. In some embodiments, Figure 1One or more of the client devices 106-116 may include the same or similar configuration as electronic device 300. In some embodiments, electronic device 300 can be used for data transmission, building and animateting 3D models, and media presentation applications.

[0059] like Figure 3 As shown, electronic device 300 includes an antenna 305, a radio frequency (RF) transceiver 310, a transmit (TX) processing circuitry 315, a microphone 320, and a receive (RX) processing circuitry 325. The RF transceiver 310 may include, for example, an RF transceiver, a Bluetooth transceiver, a Wi-Fi transceiver, a Zigbee transceiver, an infrared transceiver, and various other wireless communication signals. Electronic device 300 also includes a speaker 330, a processor 340, an input / output (I / O) interface (IF) 345, an input section 350, a display 355, a memory 360, and a sensor 365. The memory 360 includes an operating system (OS) 361, one or more applications 362, and an image 363.

[0060] RF transceiver 310 receives incoming RF signals from an access point (such as a base station, Wi-Fi router, or Bluetooth device) or other device (such as Wi-Fi, Bluetooth, cellular, 5G, LTE, LTE-A, WiMAX, or any other type of wireless network) of network 102 from antenna 305. RF transceiver 310 down-converts the incoming RF signals to generate an intermediate frequency (IF) signal or a baseband signal. The IF signal or baseband signal is sent to RX processing circuitry 325, which generates a processed baseband signal by filtering, decoding, and / or digitizing the baseband signal or IF signal. RX processing circuitry 325 sends the processed baseband signal to speaker 330 (e.g., for voice data) or processor 340 for further processing (e.g., for web browsing data).

[0061] The TX processing circuit 315 receives analog or digital voice data from the microphone 320 or other outgoing baseband data from the processor 340. Outgoing baseband data may include web data, email, or interactive video game data. The TX processing circuit 315 encodes, multiplexes, and / or digitizes the outgoing baseband data to generate a processed baseband signal or intermediate frequency (IF) signal. The RF transceiver 310 receives the outgoing processed baseband signal or IF signal from the TX processing circuit 315 and up-converts the baseband signal or IF signal into an RF signal transmitted via the antenna 305.

[0062] Processor 340 may include one or more processors or other processing devices. Processor 340 may execute instructions stored in memory 360, such as OS 361, to control the overall operation of electronic device 300. For example, processor 340 may control RF transceiver 310, RX processing circuitry 325, and TX processing circuitry 315 to receive forward channel signals and transmit reverse channel signals according to known principles. Processor 340 may include any suitable number and type of processors or other devices of any suitable arrangement. For example, in some embodiments, processor 340 includes at least one microprocessor or microcontroller. Exemplary types of processor 340 include microprocessors, microcontrollers, digital signal processors, field-programmable gate arrays, application-specific integrated circuits (ASICs), and discrete circuits.

[0063] Processor 340 is also capable of executing other processes and programs residing in memory 360, such as operations for receiving and storing data. Processor 340 may move data into or out of memory 360 as needed during execution. In some embodiments, processor 340 is configured to execute one or more applications 362 based on OS 361 or in response to signals received from an external source or operator. For example, applications 362 may include VR or AR applications, camera applications (for still images and videos), stitching applications (for stitching multiple images together to generate a 3D model), building applications, video call applications, email clients, social media clients, SMS clients, virtual assistants, etc.

[0064] The processor 340 is also connected to an I / O interface 345, which provides the electronic device 300 with the ability to connect to other devices such as client devices 106-114. The I / O interface 345 is the communication path between these accessories and the processor 340.

[0065] Processor 340 is also coupled to input unit 350 and display 355. An operator of electronic device 300 can use input unit 350 to input data or input values ​​into electronic device 300. Input unit 350 may be a keyboard, touchscreen, mouse, trackball, voice input, or other device capable of acting as a user interface to allow user interaction with electronic device 300. For example, input unit 350 may include a voice recognition process, thereby allowing user input of voice commands. In another example, input unit 350 may include a touch panel, (digital) pen sensor, keys, or ultrasonic input device. Touch panel can recognize touch input in at least one of the following methods: capacitive, pressure-sensitive, infrared, or ultrasonic. Input unit 350 can be associated with sensor 365 and / or camera by providing additional input to processor 340. In some embodiments, sensor 365 includes one or more inertial measurement units (IMUs) (such as accelerometers, gyroscopes, and magnetometers), motion sensors, optical sensors, cameras, pressure sensors, heart rate sensors, altimeters, etc. Input unit 350 may also include control circuitry. In a capacitive solution, the input unit 350 can recognize touch or proximity.

[0066] Display 355 may be a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic LED (OLED), an active-matrix OLED (AMOLED), or other displays capable of displaying text and / or graphics such as those from websites, videos, games, images, etc. The size of display 355 may be suitable for an HMD. Display 355 may be a single display or multiple displays capable of creating stereoscopic images. In some embodiments, display 355 is a head-up display (HUD). Display 355 may display 3D objects, such as animated 3D objects.

[0067] Memory 360 is coupled to processor 340. A portion of memory 360 may include RAM, and another portion of memory 360 may include flash memory or other ROM. Memory 360 may include permanent storage space (not shown), which represents any structure capable of storing and facilitating the retrieval of information such as data, program code, and / or other suitable information. Memory 360 may contain one or more components or devices supporting long-term storage of data, such as read-only memory, hard disk drive, flash memory, or optical disk. Memory 360 may also contain media content. Media content may include various types of media, such as images, videos, 3D content, VR content, AR content, animations, and static 3D objects.

[0068] The memory 360 may also include an image 363. Image 363 may include a still image of an object (such as a person). Images may be captured from multiple angles around the user, so that a 360-degree view of the object can be generated when the images are stitched together. In some embodiments, image 363 may include a single 3D object previously generated by electronic device 300 or received from another electronic device.

[0069] Electronic device 300 also includes one or more sensors 365, which can measure physical quantities or detect the activation state of electronic device 300 and convert the measured or detected information into electrical signals. For example, sensor 365 may include one or more buttons for touch input, a camera, a gesture sensor, an IMU sensor (such as a gyroscope or gyroscope sensor, and an accelerometer), an eye-tracking sensor, a barometric pressure sensor, a magnetic sensor or magnetometer, a grip sensor, a proximity sensor, a color sensor, a biophysical sensor, a temperature / humidity sensor, an illuminance sensor, an ultraviolet (UV) sensor, an electromyography (EMG) sensor, an electroencephalography (EEG) sensor, an electrocardiography (ECG) sensor, an IR sensor, an ultrasound sensor, an iris sensor, a fingerprint sensor, a color sensor (such as a red-green-blue (RGB) sensor / camera), a depth sensor, a D-RGB sensor (depth red-green-blue sensor / camera), etc. Sensor 365 may also include control circuitry for controlling any of the included sensors.

[0070] As discussed in more detail below, one or more of these sensors 365 can be used to control the user interface (UI), detect UI input, determine orientation and orientation toward the user for 3D content display recognition, etc. Any of these sensors 365 may be located within the electronic device 300, within an auxiliary device operatively connected to the electronic device 300, within a headset configured to hold the electronic device 300, or within a single device in which the electronic device 300 includes the headset.

[0071] although Figure 2 and Figure 3 Examples of electronic devices are shown, but more can be found on... Figure 2 and Figure 3 Make various changes. For example, you can combine, further subdivide, or omit. Figure 2 and Figure 3 The various components within it can be added, and additional components can be added as needed. As a specific example, processor 340 can be divided into multiple processors, such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs). Furthermore, like computing and communication systems, electronic devices and servers can have various configurations, and Figure 2 and Figure 3This disclosure is not limited to any particular electronic device or server.

[0072] Figure 4 Example Figure 400 illustrates the conversion of a 3D scan image into an animated model according to an embodiment of the present disclosure. Figure 4 The embodiments described are for illustrative purposes only, and other embodiments may be used without departing from the scope of this disclosure.

[0073] Figure 400 includes two distinct inputs, a first input 410 and a second input 420. The first input 410 and the second input 420 are 360-degree views of an object. The object is a human figure. The first input 410 and the second input 420 can be generated from multiple images stitched together to produce a single 3D scan image. The first input 410 and the second input 420 are static, meaning that the inputs do not have associated motion.

[0074] The automated construction disclosed in embodiments of this disclosure animates static 3D scan images. For example, for a given input such as a first input 410, outputs 412a, 412b, 412c, 412d, 412e, and 412f (collectively referred to as 412) will be movable avatars based on a set of animation instructions. Similarly, for a second input 420, outputs 422a, 422b, 422c, 422d, 422e, and 422f (collectively referred to as 422) will be movable avatars based on a set of animation instructions. The animation added to the first input 410 is dancing, while the animation added to the second input 420 is fighting. As shown, the movable avatars 412 and 422 move at corresponding joints in a manner similar to a human, while areas without joints remain straight (not bent).

[0075] although Figure 4 Figure 400 is shown, but it is possible to... Figure 4 Make various changes. For example, different animations can be applied to the input. Figure 4 This disclosure is not intended to be limited to any particular system or apparatus.

[0076] Figure 5A An exemplary method 500 for constructing an animated model from a static object is shown according to an embodiment of the present disclosure. Figure 5B and Figure 5C Existing parametric body models 580 and 585 are shown according to embodiments of the present disclosure. Figures 5A to 5C The embodiments described are for illustrative purposes only, and other embodiments may be used without departing from the scope of this disclosure.

[0077] Method 500 can be used by server 104, Figure 1 One of the client devices 106-116 Figure 2 Server 200 Figure 3 The method 500 may be executed by an electronic device 300, or another suitable device. In some embodiments, the method 500 may be executed by a “cloud” of computers interconnected via one or more networks, wherein each computer is independently controlled by the cloud. Figure 1 Network 102 access utilizes a cluster of computers and components as a single pool of seamless resources in a computing system. In some implementations, a portion of the components used to process method 500 may include different devices, such as multiple servers 104 or 200, multiple client devices 106-116, or other combinations of different devices. For ease of illustration, method 500 is described as being performed by a computing system having Figure 2 The internal components of server 200 Figure 1 Server 104 executes. Note that, in addition to or as a replacement for server 104, there are other servers with... Figure 3 The internal components of the electronic device 300 Figure 1 Any of the client devices 106-116 can be used to execute method 500.

[0078] In step 510, processor 210 obtains a 3D object. The 3D object can be a scanned image of a 3D object (such as a person). It can be obtained from... Figure 3 The electronic device 300 obtains a 3D object from an image 363. The processor 210 can stitch the images 363 together to form a 3D scan representing the 3D object.

[0079] In step 520, processor 210 will assign parameters to the body model (such as...) Figure 5B The processor 210 fits the parametric body model (580) to the 3D scan map. For example, for a given 3D object in step 510, the processor 210 rasterizes the object into multiple 2D views. Then, the processor 210 fuses the identified landmarks from the different views. The landmarks are then fused together to obtain 3D joints. The processor 210 then fits the parametric body model (such as...) to the 3D scan map based on the position of the 3D joints. Figure 5B The parameters of the body model (580) are aligned with the 3D scan image. Figures 6A to 6C The parameters of the body model in step 520 (such as...) are described in more detail. Figure 5B Fitting the body model (580) with the 3D scan image.

[0080] In step 530, processor 210 performs skinning, which is used to find the parameters of the body model (such as...). Figure 5B The nearest triangle to the vertex of the parametric body model (580) is found in the 3D scan image. This is done when identifying parametric body models (such as...) Figure 5BAfter taking the triangle of the body model 580 and the vertices of the nearest 3D scan map, the processor 210 interpolates the weights used for vertex transformation of the 3D scan map. Figures 8A to 8C The skinning process in step 530 is described in more detail.

[0081] In step 540, processor 210 performs patching. For example, processor 210 identifies any holes and fills them, as well as disconnects unwanted connections in the 3D scan map. In step 540, processor 210 transfers the UV coordinates of vertices to the 2D domain to fill holes and add fill to reduce any color discontinuities that may occur during motion of the constructed 3D scan map. Figures 9A to 9G The patching in step 540 is described in more detail.

[0082] In step 550, processor 210 uses an exporter to reduce the size of the constructed 3D scan image. In some embodiments, the exporter is a GLTF exporter. In some embodiments, after using the exporter, the constructed 3D scan image can be sent to... Figure 1 Any of the client devices.

[0083] In step 560, the processor 210 may receive animation action 565. Animation action 565 may be stored in a database or... Figure 2 The information is stored in the memory 230. Animation actions can be received from another device, such as client devices 106-116. The animation actions provide instructions on which joint (or combination of joints) of the constructed 3D scan should move, as well as the magnitude and direction of the movement.

[0084] In step 570, processor 210 outputs the animated object. In some implementations, when the server (such as server 200) constructs the object, the object is transferred to one or more client devices, such as... Figure 1 The head-mounted display 116 or mobile device 108, or information storage. In some embodiments, when an object is constructed in a client device, the object can be displayed on the client device's screen.

[0085] Figure 5BThe parametric body model 580 shows a complete view of the parametric body model. The parametric body model comprises a hierarchical set of interconnected parts, such as skeletons and joints connecting the skeletons, where the skeletons allow joint movement. That is, the skeleton is a joint tree used to define the skeletal structure of the parametric body model. Body movements are defined on the skeleton tree. For example, some or all of the joints rotate in three DoFs. Joints can be located throughout the parametric body model 580, such as at the neck, shoulder, wrist, elbow, fingers, hip, knee, ankle, toes, and throughout the face and back of the parametric body model 580. The joints enable the parametric body model 580 to bend in positions similar to a human and perform similar movements at each joint simulating human motion.

[0086] Figure 5C The parameters of the body model 585 are shown. Figure 5B A close-up of the parametric body model 580. The parametric body model 585 is a mesh composed of multiple triangular shapes. As the joints move, the triangles can stretch or contract as needed to modify the outline of the parametric body model 585. Note that there are other shapes besides triangles or as alternatives to triangles.

[0087] although Figures 5A to 5C Method 500 and parameters body models 580 and 585 are shown, but it is possible to modify them. Figures 5A to 5C Make various changes. For example, although Figure 5A It is shown as a series of steps, but the various steps can overlap, occur in parallel, or occur any number of times. Figures 5A to 5C This disclosure is not intended to be limited to any particular system or apparatus.

[0088] Figure 6A An exemplary method 520a for fitting according to an embodiment of the present disclosure is shown. Method 520a is described in more detail. Figure 5A Step 520. Figure 6B and Figure 6C An example of associating a static object with an established parametric body model according to an embodiment of this disclosure is shown. Figures 6A to 6C The embodiments described are for illustrative purposes only, and other embodiments may be used without departing from the scope of this disclosure.

[0089] Method 520a can be used by server 104, Figure 1 One of the client devices 106-116 Figure 2 Server 200 Figure 3 The method 520a may be performed by an electronic device 300 or another suitable device. In some embodiments, method 520a may be performed by a “cloud” of computers interconnected via one or more networks, wherein each computer is independently controlled by the cloud. Figure 1Network 102 access utilizes a cluster of computers and components as a single pool of seamless resources in a computing system. In some implementations, a portion of the components used to process method 520a may be included in different devices, such as multiple servers 104 or 200, multiple client devices 106-116, or other combinations of different devices. For ease of illustration, method 520a is described as being performed by a computing system having Figure 2 The internal components of server 200 Figure 1 Server 104 executes. Note that, in addition to or as a replacement for server 104, there are other servers with... Figure 3 The internal components of the electronic device 300 Figure 1 Any of the client devices 106-116 can be used to execute method 520a.

[0090] Given a 3D human body scan, such as in Figure 5A The 3D object obtained in step 510 is first rasterized by processor 210 into multiple views (step 522). In some embodiments, the 3D object is rasterized into 15 different views. The views are two-dimensional and derived from different orientations of the 3D object. For example, Figure 6B Two views, view 512a and view 514a, are shown among a plurality of views.

[0091] In step 524, processor 210 identifies 2D landmarks from multiple views of the 3D object. For example, pose detection is performed by the processor to identify landmarks in each of the multiple views. Landmarks are located at joints in the 2D image. In some implementations, the algorithm OPENPOSE is used as a pose detection algorithm.

[0092] When the 3D object is a 3D scan of a person, processor 210 identifies multiple landmarks on (i) the object's body, (ii) the object's hands, (iii) the object's face, (iv), etc. In some embodiments, processor 210 identifies 24 landmarks on the body of the person's 3D scan, 42 landmarks on the hands, and 70 landmarks on the face. For example, Figure 6C Markers are shown, such as marker 516a on view 512b. Note that view 512b is different from... Figure 6B View 512a is the same view, but includes all identified landmarks. Multiple landmarks, as shown, are located throughout view 512b.

[0093] Figure 6C Views 512b and 514b are also shown. Note that view 512b is different from... Figure 6A View 512a is the same view, but includes the identified landmarks. Similarly, view 514b is... Figure 6BView 514a is the same view, but includes all identified landmarks and skeletal structures. For example, landmark 516b on view 514b corresponds to landmark 516a in view 512b. View 514b also includes skeletal structures such as bone 518. A bone is a rigid structure that can move relative to one or more identified landmarks (such as landmarks 516b or 516c). Bone 518 connects landmark 516b (corresponding to the elbow of a 3D object) to landmark 516c (corresponding to the shoulder of a 3D object), or both. As shown, bone 518 corresponds to the humerus of a human body (which extends from the shoulder to the elbow).

[0094] After performing pose detection and landmark recognition, processor 210 fuses the sets of 2D landmarks from different views into 3D coordinates in step 526. Equation (1) describes the process for each 3D landmark J. j Minimize 2D landmarks from different views.

[0095] Formula (1)

[0096]

[0097] In view i, expression w ij (confidence) is a landmark. j The confidence weight. The expression ProjectM i It is the projection matrix of i. In view i, the expression is... It is a boundary marker J j 2D coordinates.

[0098] In step 528, processor 210 formulates an optimization energy by minimizing the landmark difference. Equation (2) describes the optimization energy, which depends on combining 3D joint errors and chamfer distance to fit the established parametric body model to the 3D scan map. 3D joint errors are the differences between the 3D landmarks of the 3D scan map and the 3D joints of the established parametric body model. Chamfer distance is the distance between the 3D scan map and the established parametric body model.

[0099] Formula (2)

[0100] E(θ, β) = E data (θ,β)+E pose (θ)+E body_landmark (θ,β)+E face_landmark (θ,β)+E hand_landmark (θ, β)

[0101] Variables θ and β are sets of pose and shape parameters of the established parametric body model. Variable E dataIt is a data fitting term that measures the chamfer difference between the established parametric body model and the 3D scan image. Variable E pose This is the regularization term used to regularize the pose parameters. Variable E body_landmark E face_landmark and E hand_landmark It is a landmark-guided fitting term used to measure the differences between 3D body, face, and hand landmarks and the established parametric body model and 3D scan image. After optimization, alignment is achieved between the established parametric body model and the 3D scan image.

[0102] although Figures 6A to 6C Method 530a and its various applications are shown, but it is possible to... Figures 6A to 6C Make various changes. For example, although Figure 6A It is shown as a series of steps, but the various steps can overlap, occur in parallel, or occur any number of times. Figures 6A to 6C This disclosure is not intended to be limited to any particular system or apparatus.

[0103] Figure 7A and Figure 7B An example is shown of deforming components of an existing parametric body model according to an embodiment of this disclosure. Figure 7C An exemplary method 760 using semantic variations according to embodiments of the present disclosure is shown. Figures 7A to 7C The embodiments described are for illustrative purposes only, and other embodiments may be used without departing from the scope of this disclosure.

[0104] In some implementations, processor 210 reparameterizes the shape space of the Rigged Parametric Model into a SDC that conforms to different semantic segments of the human body. Rigged parametric body models are typically trained from datasets of finite size. Thus, the global shape blending shapes of these Rigged Parametric Body Models (typically calculated from principal component analysis of shapes) span only a very limited shape space and cannot be used to represent the human body of different age groups and body shapes.

[0105] like Figure 7A As shown, processor 210 can modify specific components of a built-in parametric body model to fit the built-in parametric body model to a given 3D scan of a human. For example, as Figure 7A As shown in the figure 700, the existing parametric body model 710 can be modified based on changing the parameters of (i) body and leg proportions 720a, (ii) waist 720b, (iii) leg length 720c, and (iv) arm length 720d.

[0106] The established parametric body models 721 and 723 represent changes in the body and leg proportions 720a relative to the established parametric body model 710. For example, when the body and leg proportions 720a of the established parametric body model 710 decrease, as shown in established parametric body model 721, the body size increases while the leg size decreases. Conversely, when the body and leg proportions 720a of the established parametric body model 710 increase, as shown in established parametric body model 723, the body size decreases while the leg size increases.

[0107] Established parametric body models 724 and 726 represent variations in waist 720b relative to established parametric body model 710. For example, as shown in established parametric body model 724, when the factor associated with waist 720b of established parametric body model 710 decreases, the waist size increases. Conversely, as shown in established parametric body model 726, when the factor associated with waist 720b of established parametric body model 710 increases, the waist size decreases.

[0108] The established parametric body models 727 and 729 represent variations in leg length 720c relative to the established parametric body model 710. For example, as shown in established parametric body model 727, when the factor associated with the leg length 720c of the established parametric body model 710 decreases, the leg size increases. Conversely, as shown in established parametric body model 729, when the factor associated with the leg length 720c of the established parametric body model 710 increases, the leg size decreases.

[0109] The established parametric body models 730 and 732 represent variations in arm length 720d relative to the established parametric body model 710. For example, when the factor associated with the arm length 720d of the established parametric body model 710 decreases, as shown in the established parametric body model 730, the arm size increases. Conversely, when the factor associated with the arm length 720d of the established parametric body model 710 increases, as shown in the established parametric body model 731, the arm size decreases.

[0110] Figure 7B Figure 750 illustrates the effect of altering a pre-built parametric body model on the constructed 3D scan. Specifically, Figure 750 shows the effect of modifying specific components of the pre-built parametric body model to modify the constructed 3D scan using the modified pre-built parametric body model. As discussed in more detail below, the constructed 3D scan is a 3D scan that incorporates the characteristics of the pre-built parametric body model.

[0111] The constructed 3D scan image 752 represents a 3D scan image constructed using the components of a pre-built parametric body model. When the pre-built parametric body model is modified, as shown in outputs 754a to 754f, the constructed 3D scan image 752 undergoes corresponding transformations. For example, by decreasing the size of the waist of the pre-built parametric body model, the waist of the constructed 3D scan image is modified as shown in output 754a. Conversely, by increasing the size of the waist of the pre-built parametric body model, the waist of the constructed 3D scan image is modified as shown in output 754b. As another example, by decreasing the size of the legs of the pre-built parametric body model, the legs of the constructed 3D scan image are modified as shown in output 754c. Conversely, by increasing the size of the legs of the pre-built parametric body model, the legs of the constructed 3D scan image are modified as shown in output 754d. As yet another example, by decreasing the size of the arms of the pre-built parametric body model, the arms of the constructed 3D scan image are modified as shown in output 754e. Conversely, by increasing the size of the arms of the existing parametric body model, the arms of the constructed 3D scan are modified as shown in output 754f. Therefore, SDC can be used both to fit an existing parametric body model to a 3D scan and to modify the appearance of the constructed 3D scan.

[0112] Method 760 in Figure 7 describes the process of reparameterizing the shape of an existing parametric body model. Method 760 can be implemented by server 104. Figure 1 One of the client devices 106-116 Figure 2 Server 200 Figure 3 The method 760 may be executed by an electronic device 300 or another suitable device. In some embodiments, the method 760 may be executed by a “cloud” of computers interconnected via one or more networks, each computer acting independently within the cloud. Figure 1 Network 102 access utilizes a cluster of computers and components as a single pool of seamless resources in a computing system. In some implementations, a portion of the components used to process method 760 may include different devices, such as multiple servers 104 or 200, multiple client devices 106-116, or other combinations of different devices. For ease of illustration, method 760 is described as being performed by a computing system having Figure 2 The internal components of server 200 Figure 1 Server 104 executes. Note that, in addition to or as a replacement for server 104, there are other servers with... Figure 3 The internal components of the electronic device 300 Figure 1 Any of the client devices 106-116 can be used to execute method 760.

[0113] In step 762, processor 210 generates multiple training samples. Samples generated using different hybrid shape coefficients of the established parametric body model are used as training samples (denoted as X in formula (3) below). The training samples are used to analyze the shape changes of different body parts. The training samples are... Where F is the number of training samples, and N is the number of vertices in the built parametric body model.

[0114] In step 764, processor 210 optimizes the SDC for each body part. This is to generate local deformation components. This allows for the existence of separate components for controlling the length and shape of certain body parts, and the processor 210 formulates SDC optimizations based on formula (3). Expression It is the weight, where N C It represents the number of deformation components. The first term is ||X-WC|| 2 This ensures that C can represent the data sample X. The second term l1 / l2 is the weighted norm of the local deformation component C, which is described in formula (4).

[0115] Formula (3)

[0116] argmin W，C ||X-WC|| 2 +S(C) st V(W)

[0117] Formula (4)

[0118]

[0119] The variable Λ is the weight of the spatial variation. The variable Λ constrains the sparsity and localization of the deformation components. For each component, the processor 210 generates Λ by first defining the deformation center of a circle containing the vertices on the body model. i A circle can be located at the arm of the existing parametric body model, different circles can be located at the leg of the existing parametric body model, and additional circles can be located around the torso and thigh of the existing parametric body model. The circles are based on semantic body measurements, such as chest, waist, hips, etc. Then, as described in Equation (5) below, the geodesic distance d of the circle is... ij The spatial variation is formed by λ. Formula (5) can achieve smooth shape changes. The constraint V(W) is determined by setting... It is necessary to prevent the weight from becoming too large.

[0120] Formula (5)

[0121] Λ ij =λmax(0,min(1,(d)) ij -d min ) / (dij -d max )))

[0122] In step 766, processor 210 performs the fitting step 520 of Figure 5, the skinning step 530 of Figure 5, and Figure 6A Method 520a, and Figure 8A Method 530a, or a combination thereof, uses SDC. Additionally, SDC can be used to semantically reshape the 3D body by increasing or decreasing the size of the legs, abdomen, etc. Deformation components are established based on body measurements (such as forearm length, calf length, waist circumference, head circumference, etc.). Using SDC, the constructed 3D scan can be modified, for example, to make the constructed 3D scan slimmer and stronger.

[0123] although Figures 7A to 7C This shows the SDC and its various applications, but it is possible to... Figures 7A to 7C Make various changes. For example, although Figure 7C It is shown as a series of steps, but the various steps can overlap, occur in parallel, or occur any number of times. Figures 7A to 7C This disclosure is not intended to be limited to any particular system or apparatus.

[0124] Figure 8A An exemplary method 530a for skinning according to an embodiment of the present disclosure is shown. Method 530a is described in more detail. Figure 5A Step 530. Figure 8B The embodiments of the present disclosure are shown in Figure 8A The weights are transferred during the skinning method. Figure 8C The embodiments of the present disclosure are shown in Figure 8A The skinning method converts the scan map into a predefined pose. Figures 8A to 8C The embodiments described are for illustrative purposes only, and other embodiments may be used without departing from the scope of this disclosure.

[0125] Method 530a can be used by server 104, Figure 1 One of the client devices 106-116 Figure 2 Server 200 Figure 3 The method 530a may be performed by an electronic device 300 or another suitable device. In some embodiments, method 530a may be performed by a “cloud” of computers interconnected via one or more networks, wherein each computer is independently controlled by the cloud. Figure 1Network 102 access utilizes a cluster of computers and components as a single pool of seamless resources in a computing system. In some implementations, a portion of the components used to process method 530a may include different devices, such as multiple servers 104 or 200, multiple client devices 106-116, or other combinations of different devices. For ease of illustration, method 530a is described as being performed by a computing system having Figure 2 The internal components of server 200 Figure 1 Server 104 executes. Note that, in addition to or as a replacement for server 104, there are other servers with... Figure 3 The internal components of the electronic device 300 Figure 1 Any of the client devices 106-116 can be used to execute method 530a.

[0126] In step 532, processor 210 identifies the correspondence between the established parametric model and the 3D scan image. To identify the correspondence, processor 210 associates regions of the established parametric model with corresponding vertices in the 3D scan image. For each vertex in the 3D scan image, processor 210 identifies the nearest triangle on the established parametric body model.

[0127] In step 534, processor 210 identifies attributes by interpolating the weights and then transfers the weights. Transferring the weights of the existing parametric model to the 3D scan map is based on correspondences generated when the constructed 3D scan map is created. Transferring the attributes of the existing parametric model to the 3D scan map is based on associating joints, skin weights, UV coordinates, and markers related to identifiable parts of the existing parametric model to the 3D scan map.

[0128] The interpolation weights correspond to the barycenter coordinates. For example, Figure 8B The image shows vertex 812 on the 3D scan map and its nearest triangle 810 on the existing parametric body model.

[0129] For example, for vertex i ( Figure 8B The vertex of triangle 812 is located at point 812, and the centroid of its nearest point on triangle 810 is w1, w2. 2i w3. Then, the binding weights to the joints can be identified as described in formula (6).

[0130] Formula (6)

[0131] W ij =w1*W j1 +w2*W j2 +w3*W j3

[0132] Note the variable W j1 W j2W j3 These are the three binding weights corresponding to the three vertices of the nearest triangle. The processor 210 can re-formulate the vertex transformations of a 3D scan map similar to the established parametric body model. That is, for each vertex i, the coordinates V of its transformation are described in equation (7). i .

[0133] Formula (7)

[0134]

[0135] In formula (7), W ij This refers to the interpolated skinning weights of the joints. The expression T... j This is the transformation of joint j. Expression These are the vertex coordinates in the T pose.

[0136] Note the expression in formula (8) below. And the expression of formula (9) below It is a hybrid shape of shape and pose defined by shape parameter β and pose parameter θ. These two hybrid shapes are defined using the same corresponding strategy and centroid interpolation as the binding weights described above. In equations (8) and (9), B S (β) k and B p (θ) k It describes the shape and pose-dependent deformation of the nearest triangle with vertex index k of a body model with established parameters.

[0137] Formula (8)

[0138]

[0139] Formula (9)

[0140]

[0141] In step 536, as Figure 8C As shown, the 3D scan image is converted into a T-pose. The T-pose represents a standing person with their feet together and their arms outstretched to the sides. The vertex coordinates in the T-pose can be calculated by inverting the transformation of formula (10).

[0142] Formula (10)

[0143]

[0144] In step 538, the processor performs linear blending. To perform linear blending, the processor 210 uses the linear blending skin of formula (7) above, deforming the 3D scan image while fixing β using varying pose parameters θ, or as... Figure 4As shown, we can adjust the body shape parameter β to create some exaggerated body styles. Note that the 3D scan is not animated by various parameters; it can be manually moved by varying different pose parameters θ, such as from... Figure 8C The T-pose moves to output 412a, output 412b, output 412c, output 412d, output 412e, or output 412f.

[0145] although Figures 8A to 8C The text illustrates skinning and its various applications, but it is possible to... (The sentence is incomplete and requires further context to be fully translated. Figures 8A to 8C Make various changes. For example, although Figure 8A It is shown as a series of steps, but the various steps can overlap, occur in parallel, or occur any number of times. Figures 8A to 8C This disclosure is not intended to be limited to any particular system or apparatus.

[0146] Figure 9A An exemplary method 540a for patching according to an embodiment of the present disclosure is shown. Method 540a is described in more detail. Figure 5A Step 540. Figure 9B and Figure 9C 2D images of 3D scans before and after repair according to embodiments of the present disclosure are shown. Figure 9D and Figure 9E This illustrates a process for correcting errors related to the hands and feet of a 3D scanned object according to an embodiment of this disclosure. Figure 9F Images are shown at different stages of generating the constructed 3D scan image according to embodiments of the present disclosure. Figure 9G The filling is shown according to an embodiment of this disclosure. Figures 9A to 9G The embodiments described are for illustrative purposes only, and other embodiments may be used without departing from the scope of this disclosure.

[0147] Patching describes the process of repairing a 3D scan image. Repairing a 3D scan image removes artifacts that degrade its quality when it is animated. For example, due to some triangle vertices being too close between different body parts, or due to self-occlusion of body parts, it is necessary to repair stretched triangles or missing information (such as geometry and color) from the original 3D scan image. Repairing a 3D scan image includes: fixing unnecessarily connected parts of the 3D scan image (such as...) Figure 9F Separate the image 942a) from the 3D scan image, including occlusions or holes (such as... Figure 9B Repairing parts of holes 909A and 909B, and repairing hands and feet (such as those in 3D scans) Figure 9D and Figure 9E (As shown in the image) Repair, etc.

[0148] Method 540a can be used by server 104, Figure 1 One of the client devices 106-116 Figure 2 Server 200 Figure 3 The method may be performed by an electronic device 300, or another suitable device. In some embodiments, method 540a may be performed by a “cloud” of computers interconnected via one or more networks, wherein each computer is being... Figure 1 Network 102 access utilizes a cluster of computers and components as a single pool of seamless resources in a computing system. In some implementations, a portion of the components used to process method 540a may be included in different devices, such as multiple servers 104 or 200, multiple client devices 106-116, or other combinations of different devices. For ease of illustration, method 540a is described as being performed by a computing system having Figure 2 The internal components of server 200 Figure 1 Server 104 executes. Note that, in addition to or as a replacement for server 104, there are other servers with... Figure 3 The internal components of the electronic device 300 Figure 1 Any of the client devices 106-116 can be used to execute method 540a.

[0149] In step 542, UV coordinate information is transferred from the constructed 3D scan image based on whether vertices cross different body parts, and certain triangles in the constructed 3D scan image are cut. To identify different body parts, server 200 identifies markers associated with different parts of the constructed parametric model. In some embodiments, groups of triangles in the constructed parametric model are marked to identify different body parts. Figure 8B The diagram shows triangles from a pre-built parametric model. Different labels can be assigned to different groups of triangles based on their positions within the pre-built parametric model. For example, labels could include head, torso, leg one, leg two, arm one, arm two, hand one, hand two, and foot one and foot two. Thus, triangles in the pre-built parametric model can include specific labels based on their positions.

[0150] After identifying different body parts of the established parametric model, the server 200 marks the vertices of the constructed 3D scan image based on the correspondence between the triangles of the established parametric model and the constructed 3D scan image.

[0151] The server automatically cuts the stretched triangles and transfers the UV coordinates of all vertices to the 2D UV domain. This is done by interpolating the UV coordinates of the nearest triangle from each vertex onto the constructed parametric model. Equation (11) below describes the identification of vertices from the constructed 3D scan map. For example, W1, W2, and W3 correspond to three points of the triangle (such as...). Figure 8B (as shown in triangle 810, w1, w2, and w3).

[0152] Formula (11)

[0153] UV = w1UV1 + w2UV2 + w3UV3

[0154] Figure 9B Images 904 and 906 show 2D images based on UV transfer. Images 904 and 906 use markers to separate different body parts from the constructed 3D scan images. For example, image 904 shows a 2D displacement image, and image 906 shows a 2D color image based on the constructed 3D scan images 902a and 902b.

[0155] Note that portions of the constructed 3D scan image 902b include holes, such as hole 909a. Hole 909a is represented as hole 909b as shown in image 904. Holes (such as holes 909a and 909b) may be formed due to occlusion. Also note that some portions of the constructed 3D scan image 902a may be poorly formed, such as hand 908. As shown, hand 908 does not depict the individual fingers.

[0156] In step 544, server 200 bakes color and geometric textures on the 2D UV domain, where the geometric textures represent the displacement of the scanned image relative to the surface of the established parametric model. That is, server 200 transforms the 3D geometry and texture inpainting problem into a 2D inpainting problem (according to step 542). Then, server 200 bakes the displacement information into image 904 and the color information into image 906 to generate, respectively. Figure 9C Images 914 and 916. Hole 909b is filled by baking a texture onto image 904, as shown in 919b. In some implementations, the server can use any patching technique.

[0157] In some implementations, server 200 adds fill areas to avoid color leakage between different graphics and to smooth along the cut seams. Fill is added around each body part in images 914 and 916. See below. Figure 9G Describe the filling process in more detail.

[0158] In some implementations, such as Figure 9D Image 920 or Figure 9B As shown in hand 908, the constructed 3D scan of the hand typically includes artifacts. The server uses a parametric model template to provide the hand mesh (in... Figure 9D (As shown in image 922) as a hand (such as) on the scanned image Figure 9CThe replacement of the hand (918). The hand mesh is used to replace the existing hand in the constructed 3D scan image. The sigmoid blending function as described in Equation (12) can be used to remove the gap between the hand and the arm.

[0159] Formula (12)

[0160]

[0161] In formula (12) above, x is the distance between an arm vertex and the nearest hand vertex. In some implementations, the maximum value of x for the blended vertices on the arm mesh is 0.015. w in formula (12) is the blending weight. The server then smooths the arm vertices using a blending function as described in formula (13). In formula (13) below, v reconstruction and v template These are vertices from the constructed 3D scan image and the template, respectively.

[0162] Formula (13)

[0163] v final =w*v reconstruction +(1-w)*v template

[0164] In some implementations, such as Figure 9E As shown in image 930, the foot in the constructed 3D scan image typically includes artifacts. The server can replace the foot in the scan image with a template such as foot 932. In some implementations, the server replaces the reconstructed "foot" mesh with the shoe mesh of the original 3D scan image.

[0165] In step 546, the server repairs any texture issues. For example, the server can use a neural network to identify and repair any color or geometric textures in the constructed 3D scan map that were not repaired prior to step 544.

[0166] In step 548, after performing patching and filling, the server is based on ( Figure 9C Modified images 914 and 916, Figure 9D The modified hand in image 922, and ( Figure 9D The modified 932 is used to regenerate the constructed 3D scan image, such as... Figure 9C As shown in the modified constructed 3D scan images 912a and 912b. That is, the modified constructed 3D scan images 912a and 912b do not include any holes and the improved hands and feet.

[0167] Figure 9FVarious images are shown at different stages of modifying the constructed 3D scan. For example, image 940 shows the original 3D scan. Image 942a shows the deformed mesh after linear blending skinning with stretched triangles. Image 942b shows the effect of cutting different body parts of the constructed 3D scan. That is, the stretched triangles shown in image 942a are cut so that the hand is no longer undesirably attached to the torso. Image 942c shows the constructed 3D scan reconstructed based on repairs. That is, image 942c is the constructed 3D scan with reconstructed geometry and color texture. Image 942d shows the fully modified constructed 3D scan with repaired hands and shoes. That is, image 942d is the fully modified constructed 3D scan based on image 940.

[0168] Figure 9G The image illustrates the filling process according to an embodiment of this disclosure. As discussed above in step 544, the server adds fill to areas of the 2D image to prevent color bleeding between different graphics and to smooth along the cut seams. Image 950 shows the constructed 3D scan without fill. As shown, color bleeding exists at locations 951a and 951b. After the fill is performed, as shown in image 952, the fill prevents color bleeding.

[0169] although Figures 9A to 9G This illustrates repair and its various applications, but it is possible to... Figures 9A to 9G Make various changes. For example, although Figure 9A It is shown as a series of steps, but the various steps can overlap, occur in parallel, or occur any number of times. Figures 9A to 9G This disclosure is not intended to be limited to any particular system or apparatus.

[0170] Figure 10 An exemplary method 1000 for object construction according to an embodiment of the present disclosure is shown. Method 1000 may be provided by server 104, Figure 1 One of the client devices 106-116 Figure 2 Server 200 Figure 3 The method 1000 may be executed by an electronic device 300 or another suitable device. In some embodiments, the method 1000 may be executed by a “cloud” of computers interconnected via one or more networks, wherein each computer is being... Figure 1 Network 102 access utilizes a cluster of computers and components as a single pool of seamless resources in a computing system. In some implementations, a portion of the components used to process method 1000 may include different devices, such as multiple servers 104 or 200, multiple client devices 106-116, or other combinations of different devices. For ease of illustration, method 1000 is described as being performed by a computing system having Figure 2The internal components of server 200 Figure 1 Server 104 executes. Note that, in addition to or as a replacement for server 104, there are other servers with... Figure 3 The internal components of the electronic device 300 Figure 1 Any of the client devices 106-116 can be used to execute method 1000.

[0171] In step 1002, server 200 receives a 3D scan of the object. In some embodiments, server 200 generates the 3D scan based on multiple images received from another device (such as one of the client devices) corresponding to different orientations of the object. For example, server 200 may be configured to stitch multiple images together to generate a 3D scan. Note that the multiple 2D images may come from one or more cameras, such as a color camera, a depth camera, etc.

[0172] In step 1004, server 200 matches the built-in parametric model to the 3D scan image. The built-in parametric model is a model similar to the 3D scan image and includes pre-identified joints. For example, if the 3D scan image is a human, then the built-in parametric model is also a human model. That is, the built-in parametric model of a human has predefined joints corresponding to the joints of the human body at specific locations on the model. For example, the built-in parametric model includes a skeletal system, such as bones and joints, enabling the built-in parametric model to mimic human movement.

[0173] To match the established parametric model with the 3D scan image, server 200 minimizes the surface distance between the established parametric model and the 3D scan image. Additionally, to match the established parametric model with the 3D scan image, server 200 minimizes the 3D joint error between the established parametric model and the 3D scan image.

[0174] To match the established parametric model with the 3D scan image, server 200 extracts different views (or orientations) of the 3D scan image. For example, server 200 presents the 3D scan image in different views (or orientations). In some implementations, server 200 presents the 3D scan image in 15 different views (or orientations), such as front, back, side, top, bottom, and various other angles in between.

[0175] After presenting the 3D scan images in different views, server 200 identifies multiple 2D landmarks in each view. These landmarks correspond to various joint positions throughout the body. Depending on the orientation of each view, server 200 identifies a predefined number of landmarks for the body, hands, and face. In some implementations, there are 25 landmarks for the entire body, 42 for the entire hand, and 70 for the entire face. Not all landmarks will be identified in every view, as some may be occluded.

[0176] Then, server 200 combines multiple 2D landmarks from different orientations of the 3D scan image to identify 3D landmarks in the 3D scan image. That is, a 3D landmark is a set of coordinates on the 3D scan image corresponding to multiple 2D landmarks identified in multiple views. Next, server 200 identifies 3D joint errors based on the differences between the 3D landmarks of the 3D scan image and the 3D joints of the built parametric model. Server 200 then minimizes the 3D joint errors and the distance between the surface of the 3D scan image and the built parametric model. For example, the built parametric model can be modified to match the shape of the 3D scan image. For example, the distance between two joints of the built parametric model can be lengthened or shortened to match the distance between corresponding joints in the 3D scan image.

[0177] In some implementations, server 200 generates modifiable local regions of the existing parametric model. Server 200 then modifies these local regions to match the corresponding shape of the 3D scan image by deforming them. For example, the ratio of the body to the legs of the existing parametric model can be changed. For instance, the height of the legs relative to the body height can be changed while keeping the overall height of the existing parametric model constant. Similarly, the waist of the existing parametric model can be increased or decreased to match the 3D scan image. Likewise, the leg length and arm length of the existing parametric model can be increased or decreased to match the 3D scan image.

[0178] In step 1006, server 200 identifies the correspondence between the established parametric model and the 3D scan image. To identify the correspondence, server 200 associates regions of the established parametric model with corresponding vertices of the 3D scan image. For example, for each vertex of the 3D scan image, server 200 identifies the nearest triangle in the established parametric model. Subsequently, server 200 identifies weights and interpolates the weights.

[0179] In step 1008, server 200 transfers the attributes of the existing parametric model to the 3D scan map based on the correspondence to generate the constructed 3D scan map. In order to transfer the attributes of the existing parametric model to the 3D scan map, server 200 associates the joints, skin weights, UV coordinates, and markers related to the identifiable parts of the existing parametric model to the 3D scan map.

[0180] In some implementations, when the 3D object represents a person, server 200 identifies the hands and feet of the 3D scan image. Server 200 can replace the hands in the constructed 3D scan image with a template mesh of the hands. Server 200 can replace the feet in the constructed 3D scan image with a scan mesh.

[0181] In some implementations, server 200 can separate unintentionally connected portions of the constructed 3D scan image. For example, as Figure 4 As shown in Example 942a of F, if a portion of the arm in the constructed 3D scan remains attached to the body.

[0182] In some implementations, server 200 can identify and fill any holes (occlusions) in the constructed 3D scan image. To fill the holes, server 200 transfers body marker information from the built parametric model to the constructed 3D scan image. Server 200 can segment the constructed 3D scan image based on the body marker information at vertices corresponding to two or more predefined body parts. After segmenting the constructed 3D scan image into multiple 3D parts, coordinates are transferred from the built parametric model to the constructed 3D scan image based on correspondences. Server 200 generates a 2D image including a 2D displacement image and a color image based on the transferred coordinates. The 2D image is... Figure 9B Images 904 and 906 are shown. Note that the 2D displacement image and the color image represent multiple 3D portions of the constructed 3D scan image indicated by body marking information, and the first displacement image of the 2D displacement image corresponds to the first color image of the color image.

[0183] To transfer coordinates, server 200 identifies the UV coordinates of the vertices of the 3D scan map based on interpolation of the first UV coordinates of the nearest triangle from the established parametric model. Then, server 200 transfers the UV coordinates of the vertices to the 2D domain to generate a 2D displacement image and a color image of the 2D image.

[0184] Then, server 200 modifies the 2D displacement image and color image of the 2D image. For example, server 200 may add padding to the boundaries of the 2D displacement image and color image to prevent color discontinuities in different regions when the 2D image is used to reconstruct the constructed 3D scan image.

[0185] Server 200 can also modify 2D displacement and color images by filling holes and adding color. For example, server 200 can identify portions of displacement and color images that include areas lacking geometry and texture (such as holes). To modify the displacement and color images, server 200 adds geometry to the holes in the displacement image based on the corresponding surfaces of the established parametric model. Server 200 also adds texture to the added geometry based on the texture of the first color image. Once the holes are filled and colored, server 200 reconstructs the constructed 3D scan image based on the modified 2D displacement and color images.

[0186] In some implementations, after generating the constructed 3D scan, the server 200 can modify the appearance of the constructed 3D scan. For example, the server 200 can deform local areas of the constructed 3D scan. For example, the body and leg proportions of the constructed 3D scan can be changed. For example, the leg height relative to the body height can be changed while keeping the overall height of the constructed 3D scan fixed. As another example, the waist of the constructed 3D scan can be increased or decreased. Adjusting the waist can represent increasing or decreasing the weight of the constructed 3D scan. For example, if a user wants their avatar represented in the constructed 3D scan to be slimmer, the waist of the constructed 3D scan can be decreased. Similarly, the leg length and arm length of the constructed 3D scan can be increased or decreased based on user input.

[0187] In step 1010, server 200 adds animation to the constructed 3D scan map. The constructed 3D scan map can be animated and moved based on received input that provides instructions describing the motion.

[0188] In some implementations, a constructed 3D scan with applied animation is displayed on a screen. For example, a constructed 3D scan with applied animation can be transmitted from a server to one of the client devices, including a display. If the constructed 3D scan is created on a client device (rather than a server), it can be displayed in response to input from a user.

[0189] although Figure 10 Method 1000 is shown, but it is possible to... Figure 10 Make various changes. For example, although Figure 10 It is shown as a series of steps, but the various steps can overlap, occur in parallel, or occur any number of times. Figure 10 This disclosure is not intended to be limited to any particular system or apparatus.

[0190] Although the accompanying drawings illustrate different examples of user devices, various changes can be made to the drawings. For example, a user device can include any number of each component in any suitable arrangement. Generally, the drawings do not limit the scope of this disclosure to any particular configuration. Furthermore, while the drawings illustrate operating environments in which the various user device features disclosed in this patent document can be used, these features can be used in any other suitable system. Nothing described in this application should be construed as implying that any particular element, step, or function is an essential element that must be included within the scope of the claims.

[0191] Although this disclosure has been described with reference to exemplary embodiments, various changes and modifications may be suggested to those skilled in the art. This disclosure is intended to include such changes and modifications that fall within the scope of the appended claims.

Claims

1. An electronic device for object construction, the electronic device comprising: The processor is configured as follows: Obtain a 3D scan of the object; The established parameter model is matched to the three-dimensional scan image by minimizing the surface distance and three-dimensional joint error between the established parameter model and the three-dimensional scan image; Identify the correspondence between the established parameter model and the 3D scan image; Based on the correspondence, the attributes of the established parameter model are transferred to the 3D scan image to generate the constructed 3D scan image; Transfer the body marker information of the established parametric model to the constructed 3D scan image; Based on the body marker information, the constructed 3D scan image is segmented at vertices corresponding to two or more predefined body parts; After dividing the constructed 3D scan image into multiple 3D parts, the UV coordinates are transferred from the established parameter model to the constructed 3D scan image based on the correspondence. A two-dimensional image is generated based on the transferred UV coordinates, including a two-dimensional displacement image and a color image, wherein the two-dimensional displacement image and the color image represent the plurality of three-dimensional portions of the constructed three-dimensional scan map indicated by the body marking information, and a first displacement image of the two-dimensional displacement image corresponds to a first color image of the color image; Modify the two-dimensional displacement image and color image of the two-dimensional image; The constructed three-dimensional scan image is reconstructed based on the modified two-dimensional displacement image and the color image; and The animation is applied to the reconstructed 3D scan image. The reconstructed 3D scan image, which incorporates the applied animation, is displayed on the screen. 2.The electronic device of claim 1, wherein, In order to match the established parameter model to the 3D scan image, the processor is configured to: Identify multiple two-dimensional landmarks in the three-dimensional scan image from different orientations; Combining the plurality of two-dimensional landmarks of different orientations from the three-dimensional scan image to identify the three-dimensional landmarks of the three-dimensional scan image, wherein the three-dimensional landmarks are a set of coordinates on the three-dimensional scan image corresponding to the set of the plurality of two-dimensional landmarks; Based on the difference between the 3D landmarks in the 3D scan image and the 3D joints in the established parametric model, the 3D joint error is identified; and Minimize the 3D joint error and minimize the distance between the surface of the 3D scan and the established parametric model. 3.The electronic device of claim 1, wherein The processor is also configured to: Generate a modifiable local region of the established parametric model; and The local region of the established parametric model is modified by deforming the local region to match the corresponding shape of the 3D scan image.

4. The electronic device according to claim 1, wherein: To identify the correspondence between the established parametric model and the 3D scan image, the processor is configured to: associate regions of the established parametric model with corresponding vertices of the 3D scan image, and In order to transfer the attributes of the established parametric model to the 3D scan image, the processor is configured to associate joints, skin weights, UV coordinates, and markers associated with identifiable portions of the established parametric model to the 3D scan image.

5. The electronic device according to claim 1, wherein: The object is a human body model; and The processor is also configured to: Identify the hands and feet of the human model in the 3D scan image; and Replace the hand in the constructed 3D scan image with a template mesh, and replace the foot in the constructed 3D scan image with a scan mesh. 6.The electronic device of claim 1, wherein In order to transfer the coordinates from the established parametric model to the constructed 3D scan image, the processor is configured to: The UV coordinates of the vertices of the 3D scan map are identified by interpolating the first UV coordinates of the nearest triangle from the established parametric model; and The UV coordinates of the vertex are transferred to the two-dimensional domain to generate a two-dimensional displacement image and a color image of the two-dimensional image. 7.The electronic device of claim 1, wherein To modify the two-dimensional displacement image and the color image, the processor is configured to add padding to the boundaries of the two-dimensional displacement image and the color image to prevent color discontinuities in regions of the reconstructed three-dimensional scan image that correspond to different parts of the predefined body parts of the built parametric model.

8. The electronic device according to claim 1, wherein: The processor is further configured to: identify portions of the first displacement image and the first color image that include holes, wherein the holes represent areas of the constructed 3D scan image lacking geometry and texture, and In order to modify the first displacement image and the first color image, the processor is configured to: Based on the corresponding surface of the established parameter model, add geometry to the holes in the first displacement image to fill the holes. Texture is added to the added geometry based on the texture of the first color image, and Add fill to the area near the boundary between the two-dimensional displacement image and the color image.

9. The electronic device according to claim 1, wherein, In order to obtain the 3D scan image, the processor is configured to stitch together multiple 2D views of the object from at least one of a camera and a depth camera.

10. A method for object construction, comprising: Obtain a 3D scan of the object; The established parameter model is matched to the three-dimensional scan image by minimizing the surface distance between the established parameter model and the three-dimensional scan image and by minimizing the three-dimensional joint error. Identify the correspondence between the established parameter model and the 3D scan image; Based on the correspondence, the attributes of the established parameter model are transferred to the 3D scan image to generate the constructed 3D scan image; Transfer the body marker information of the established parametric model to the constructed 3D scan image; Based on the body marker information, the constructed 3D scan image is segmented at vertices corresponding to two or more predefined body parts; After dividing the constructed 3D scan image into multiple 3D parts, the UV coordinates are transferred from the established parameter model to the constructed 3D scan image based on the correspondence. A two-dimensional image is generated based on the transferred UV coordinates, including a two-dimensional displacement image and a color image, wherein the two-dimensional displacement image and the color image represent the plurality of three-dimensional portions of the constructed three-dimensional scan map indicated by the body marking information, and a first displacement image of the two-dimensional displacement image corresponds to a first color image of the color image; Modify the two-dimensional displacement image and color image of the two-dimensional image; The constructed three-dimensional scan image is reconstructed based on the modified two-dimensional displacement image and the color image; and The animation is applied to the reconstructed 3D scan image. The reconstructed 3D scan image, which incorporates the applied animation motions, is displayed on a screen.

11. The method according to claim 10, wherein, Matching the established parameter model to the 3D scan image includes: Identify multiple two-dimensional landmarks in the three-dimensional scan image from different orientations; Combining the plurality of two-dimensional landmarks of different orientations from the three-dimensional scan image to identify the three-dimensional landmarks of the three-dimensional scan image, wherein the three-dimensional landmarks are a set of coordinates on the three-dimensional scan image corresponding to the set of the plurality of two-dimensional landmarks; Based on the difference between the 3D landmarks in the 3D scan image and the 3D joints in the established parametric model, the 3D joint error is identified; and Minimize the 3D joint error and minimize the distance between the surface of the 3D scan and the established parametric model.

12. The method of claim 10, further comprising: Generate a modifiable local region of the established parametric model; as well as The local region of the established parametric model is modified by deforming it to match the corresponding shape of the 3D scan image.

13. The method of claim 10, wherein: Identifying the correspondence between the established parametric model and the 3D scan image includes: associating regions of the established parametric model with corresponding vertices of the 3D scan image, and Transferring the attributes of the established parametric model to the 3D scan image includes associating joints, skin weights, UV coordinates, and markers related to identifiable parts of the established parametric model to the 3D scan image.

14. The method of claim 10, wherein, The object is a human body model, and the method further includes: Identify the hands and feet of the human model in the 3D scan image; and Replace the hand in the constructed 3D scan image with a template mesh, and replace the foot in the constructed 3D scan image with a scan mesh.

Images